Smart Home Skill Evaluation REST API Reference

Use the Smart Home Evaluation REST API to start evaluations and get evaluation results. Smart home evaluations verify that you implemented the Alexa smart home interfaces in your skill correctly.

If you want to run the Smart Home tests in the Alexa developer console, see Use the Smart Home Test Tool.

API endpoint

The endpoint of the Smart Home Evaluation API is https://api.amazonalexa.com.

Authentication

Each API request must have an authorization header whose value is the access token retrieved from Login with Amazon (LWA). For details, see Get an Access Token for SMAPI.

Operations

The Smart Home Evaluation API includes the following operations.

Operation HTTP Method and URI

Create smart home capability evaluation

POST /v1/skills/{skillId}/smartHome/testing/capabilityEvaluations

Get smart home capability evaluation status

GET /v1/skills/{skillId}/smartHome/testing/capabilityEvaluations/{evaluationId}

Get smart home capability evaluation results

GET /v1/skills/{skillId}/smartHome/testing/capabilityEvaluations/{evaluationId}/results?maxResults={maxResults}&offset={offset}

List smart home capability evaluations

GET /v1/skills/{skillId}/smartHome/testing/capabilityEvaluations?stage={stage}&startTimestampFrom={startTimestampFrom}&startTimestampTo={startTimestampTo}&maxResults={maxResults}&nextToken={nextToken}

List smart home capability test plans

GET /v1/skills/{skillId}/smartHome/testing/capabilityTestPlans?maxResults={maxResults}&nextToken={nextToken}

Create smart home capability evaluation

Create and start a capability evaluation against the specified smart home skill.

Before you create an evaluation, make sure that you do the following test setup:

  • Enable your skill in the Alexa app.
  • Your skill configures account linking and you link your device account with your Alexa account successfully.
  • Alexa discovered your test device successfully and you can view your device in the Alexa app.

Request

To create an evaluation, you make a POST request to the smartHome/testing/capabilityEvaluations resource.

Request path and header example

Copied to clipboard.

POST /v1/skills/{skillId}/smartHome/testing/capabilityEvaluations
Host: api.amazonalexa.com
Content-Type: application/json
Accept: application/json
Authorization: Bearer {access token}
X-Amzn-RequestId: {request ID}

Request path and header parameters

Parameter Located in Description Type Required

skillId

Path

Identifies the skill.
Valid values: 1 – 255 characters.

String

Yes

access token

Header

LWA token.

String

Yes

request ID

Header

Unique identifier for the request, preferably a version 4 UUID.

String

Yes

Request body example

Copied to clipboard.

{
    "endpoint": {
        "endpointId": "endpoint-001"
    },
    "capabilityTestPlan": {
        "id": "test-plan-001"
    },
    "stage": "development"
}

Request body properties

Property Description Type Required

endpoint

Represents a connected device or component associated with a user's device cloud account.

Object

Yes

endpoint.endpointId

Unique identifier for the endpoint.
Valid value: Up to 256 characters. Can include letters, numbers, spaces, and the following special characters: _ - = # ; : ? @ &

String

Yes

capabilityTestPlan

Represents the test plan to evaluate the skill against.

Object

Yes

capabilityTestPlan.id

Identifies the test plan.

String

Yes

stage

Indicates the stage of the skill.
Valid values: development, live.

String

Yes

Response

A successful response returns HTTP 200 OK, along with the identifier of the evaluation to indicate that the evaluation started successfully. On error, the response returns the appropriate HTTP status code and includes a response body with an error code and human readable message.

Response body example

{
    "id": "testRunId12312",
    "endpoint": {
        "endpointId": "endpoint-001"
    },
    "capabilityTestPlan": {
        "id": "test-plan-001"
    },
    "stage": "development"
}

Response body properties

Property Description Type Required

id

Identifies the evaluation.

String

Yes

endpoint

Represents a connected device or component associated with a user's device cloud account.

Object

Yes

endpoint.endpointId

Unique identifier for the endpoint.
Valid value: Up to 256 characters. Can include letters, numbers, spaces, and the following special characters: _ - = # ; : ? @ &

String

Yes

capabilityTestPlan

Represents the test plan to evaluate the skill against.

Object

Yes

capabilityTestPlan.id

Identifies the test plan.

String

Yes

stage

Indicates the stage of the skill.
Valid values: development, live.

String

Yes

HTTP status codes

Status Description

200 OK

Response body contains the evaluation identifier indicating that the evaluation started.

400 Bad Request

Indicates that one or more properties in the request body are invalid.
The following example shows the response body with the error and message.

{
     "message": "The property is outside the allowed range.",
     "code": "INVALID_STRING_LENGTH"
}

401 Unauthorized

Request didn't include the authorization token or the token is invalid or expired. Or, the client doesn't have access to the resource.

403 Forbidden

Indicates that the authorization token is valid, but the requested operation isn't allowed.

404 Not Found

Requested resource not found.

429 Too Many Requests

Skill has exceeded the permitted rate limit (specified number of requests per unit of time). The skill can retry the request by using exponential back-off.

500 Server Error

Error occurred on the server. The skill can retry the request by using exponential back-off.

503 Service Unavailable

Server is down for maintenance, overloaded, or otherwise unavailable to handle the incoming request.

Get smart home capability evaluation status

Get the status of an evaluation.

Request

To get the evaluation status, you make a GET request to the smartHome/testing/capabilityEvaluations resource.

Request path and header example

Copied to clipboard.


GET /v1/skills/{skillId}/smartHome/testing/capabilityEvaluations/{evaluationId}
Host: api.amazonalexa.com
Content-Type: application/json
Authorization: Bearer {access token}
Accept: application/json
X-Amzn-RequestId: {request ID}

Request path and header parameters

Parameter Located in Description Type Required

skillId

Path

Identifies the skill.
Valid values: 1 – 255 characters.

String

Yes

evaluationId

Path

Identifies the evaluation.
Use the ID returned from Create smart home capability evaluation.

String

Yes

access token

Header

LWA token.

String

Yes

request ID

Header

Unique identifier for the request, preferably a version 4 UUID.

String

Yes

Request body example

The request has no body.

Request body properties

The request has no body.

Response

A successful response returns HTTP 200 OK, along with the evaluation status and metrics. On error, the response returns the appropriate HTTP status code and includes a response body with an error code and human readable message.

Response body example

The following example shows a response.

{
    "startTimestamp": "2019-07-17T06:45:40.689Z",
    "endTimestamp": "2019-07-17T06:55:40.689Z",
    "status": "PASSED",
    "metrics": {
        "PowerController": {
            "totalTestCases": 2,
            "passedTestCases": 2,
            "failedTestCases": 0,
            "errorTestCases": 0
        }
    },
    "input": {
        "endpoint": {
            "endpointId": "endpoint-001"
        },
        "capabilityTestPlan": {
            "id": "test-plan-001"
        },
        "stage": "live"
    }
}

Response body properties

Property Description Type

startTimestamp

Start time of the evaluation.
Defined in ISO 8601 format, YYYY-MM-DDThh:mm:ssZ.

String

endTimestamp

End time of the evaluation.
Defined in ISO 8601 format, YYYY-MM-DDThh:mm:ssZ.

String

status

Status of the evaluation.
Valid values: PASSED, FAILED, IN_PROGRESS, ERROR.

String

error

(Optional) Included when an error occurs retrieving the evaluation status.

Object

error.code

Error code.
Valid values: CAPABILITY_NOT_SUPPORTED, DISCOVERY_FAILED, MULTIPLE_MATCHED_ENDPOINTS, MULTIPLE_MATCHED_TEST_PLANS, TEST_CASE_TIME_OUT, NO_SUCH_ENDPOINT, NO_SUCH_SKILL_STAGE, NO_SUCH_TEST_PLAN.

String

error.message

Human-readable description of the error.

String

metrics

Describes the number of test cases in the evaluation.

Object

metrics.<name>

Name of the test plan.

Object

metrics.<name>.totalTestCases

Total number of test cases in the test plan.

Integer

metrics.<name>.passedTestCases

Number of test cases that passed.

Integer

metrics.<name>.failedTestCases

Number of test cases that failed.

Integer

metrics.<name>.errorTestCases

Number of test cases that didn't complete due to an error condition, such as unplugged device.

Integer

input

Evaluation details given in Create smart home capability evaluation.

Type

input.endpoint

Represents a connected device or component associated with a user's device cloud account.

Object

input.endpoint.endpointId

Unique identifier for the endpoint.

String

input.capabilityTestPlan

Represents the test plan to evaluate the skill against.

Object

input.capabilityTestPlan.id

Identifies the test plan.

String

input.stage

Indicates the stage of the skill.

String

HTTP status codes

Status Description

200 OK

Response body contains an array of the customer lists. Default lists appear first in the array.

400 Bad Request

Indicates that one or more properties in the request body are invalid.
The following example shows the response body with the error and message.

{
     "message": "The property is outside the allowed range.",
     "code": "INVALID_STRING_LENGTH"
}

401 Unauthorized

Request didn't include the authorization token or the token is invalid or expired. Or, the client doesn't have access to the resource.

403 Forbidden

Indicates that the authorization token is valid, but the requested operation isn't allowed.

404 Not Found

Requested resource not found.

429 Too Many Requests

Skill has exceeded the permitted rate limit (specified number of requests per unit of time). The skill can retry the request by using exponential back-off.

500 Server Error

Error occurred on the server. The skill can retry the request by using exponential back-off.

503 Service Unavailable

Server is down for maintenance, overloaded, or otherwise unavailable to handle the incoming request.

Get smart home capability evaluation results

Get the results of the specified evaluation.

Request

To get the results of an evaluation, you make a GET request to the smartHome/testing/capabilityEvaluations resource.

Request header example

Copied to clipboard.

GET /v1/skills/{skillId}/smartHome/testing/capabilityEvaluations/{evaluationId}/results?maxResults=2&offset=1
Host: api.amazonalexa.com
Content-Type: application/json
Authorization: Bearer {access token}
Accept: application/json

Request path and header parameters

Parameter Located in Description Type Required

skillId

Path

Identifies the skill.
Valid values: 1 – 255 characters.

String

Yes

evaluationId

Path

Identifies the evaluation.
Use the ID returned from Create smart home capability evaluation.

String

Yes

maxResults

Query

Maximum number of items to return in the response. The response might contain fewer items than the value you specify, but never contains more. Default: 20
Valid values: 1 – 1000.

Integer

No

offset

Query

Offset of the result page to return.

Integer

No

access token

Header

LWA token.

String

Yes

access token

Header

Access token used for the customer. Set to apiAccessToken property of the System object from the LaunchRequest or IntentRequest.

String

Yes

Request body example

The request has no body.

Request body properties

The request has no body.

Response

A successful response returns HTTP 200 OK, along with the evaluation results. On error, the response returns the appropriate HTTP status code and includes a response body with an error code and human readable message.

Response body example

The following example shows a response.

{
    "paginationContext": {
        "offset": "1",
        "totalCount": 2
    },
    "testCaseResults": [{
        "name": "LightingTestCase1",
        "status": "PASSED",
        "capability": "PowerController",
        "expectedCapabilityStates": [{
                "namespace": "Alexa.PowerController",
                "name": "powerState",
                "value": "ON"
            },
            {
                "namespace": "Alexa.BrightnessController",
                "name": "brightness",
                "value": 20
            }
        ],
        "directive": {
            "header": {
                "namespace": "Alexa.BrightnessController",
                "name": "setBrightness"
            },
            "payload": {
                "brightness": 20
            }
        },
        "actualCapabilityStates": [{
                "namespace": "Alexa.PowerController",
                "name": "powerState",
                "value": "ON"
            },
            {
                "namespace": "Alexa.BrightnessController",
                "name": "brightness",
                "value": 20
            }
        ]
    }]
}

Response body properties

Parameter Description Type

paginationContext

Included when there are more items that match the request.

Object

paginationContext.offset

Offset in the result page.

Integer

paginationContext.totalCount

Total number of items that satisfy the request. This number might be higher than the number of items returned in this response.

Integer

testCaseResults

List of test plans that satisfy the request.

Array of objects

testCaseResults.name

Human-readable test case name.

String

testCaseResults.status

Status of the test case.
Valid values: PASSED, FAILED, ERROR.

String

testCaseResults.expectedCapabilityStates

Smart Home API interfaces and expected property values.

Array of objects

testCaseResults.expectedCapabilityStates.namespace

Interface name, such as Alexa.PowerController.

String

testCaseResults.expectedCapabilityStates.name

Property name, such as brightness.

String

testCaseResults.expectedCapabilityStates.comparison

Operation used to compare the expected property value with the actual property value.
Valid values: EQUAL_TO, GREATER_THAN, LESS_THAN.

String

testCaseResults.expectedCapabilityStates.value

Expected property value results.

String

testCaseResults.directive

Directive sent to the skill.

Object

testCaseResults.directive.header

Header information.

Object

testCaseResults.directive.header.namespace

Name of the interface.

String

testCaseResults.directive.header.name

Name of the directive.

String

testCaseResults.directive.payload

JSON payload included in the directive.

Object

testCaseResults.actualCapabilityStates

Actual property value results returned from the skill.

Array of objects

testCaseResults.actualCapabilityStates.namespace

Interface name, such as Alexa.PowerController.

String

testCaseResults.actualCapabilityStates.name

Property name, such as brightness.

String

testCaseResults.actualCapabilityStates.comparison

Operation used to compare the expected property value with the actual property value.
Valid values: EQUAL_TO, GREATER_THAN, LESS_THAN.

String

testCaseResults.actualCapabilityStates.value

Expected property value.

String

testCaseResults.expectedResponse

Expected response to the directive.

String

testCaseResults.actualResponse

Actual response to the directive.

String

error

(Optional) Included when an error occurs.

Object

error.code

Error code.
Valid values: CAPABILITY_NOT_SUPPORTED, DISCOVERY_FAILED, MULTIPLE_MATCHED_ENDPOINTS, MULTIPLE_MATCHED_TEST_PLANS, TEST_CASE_TIME_OUT, NO_SUCH_ENDPOINT, NO_SUCH_SKILL_STAGE, NO_SUCH_TEST_PLAN.

String

error.message

Human-readable description of the error.

String

HTTP status codes

Status Description

200 OK

Response body contains an array of the customer lists. Default lists appear first in the array.

400 Bad Request

Indicates that one or more properties in the request body are invalid.
The following example shows the response body with the error and message.

{
     "message": "The property is outside the allowed range.",
     "code": "INVALID_STRING_LENGTH"
}

401 Unauthorized

Request didn't include the authorization token or the token is invalid or expired. Or, the client doesn't have access to the resource.

403 Forbidden

Indicates that the authorization token is valid, but the requested operation isn't allowed.

404 Not Found

Requested resource not found.

429 Too Many Requests

Skill has exceeded the permitted rate limit (specified number of requests per unit of time). The skill can retry the request by using exponential back-off.

500 Server Error

Error occurred on the server. The skill can retry the request by using exponential back-off.

503 Service Unavailable

Server is down for maintenance, overloaded, or otherwise unavailable to handle the incoming request.

List smart home capability evaluations

List the evaluations for a specified skill.

Request

To list the evaluations for a skill, you make a GET request to the smartHome/testing/capabilityEvaluations resource.

Request header example

Copied to clipboard.

GET /v1/skills/{skillId}/smartHome/testing/capabilityEvaluations?stage=development&startTimestampFrom=2022-07-12T09:00:00.000Z&startTimestampTo=2022-07-15T11:00:00.000Z&maxResults=10&nextToken=someToken
Host: api.amazonalexa.com
Content-Type: application/json
Authorization: Bearer {access token}
Accept: application/json

Request path and header parameters

Parameter Located in Description Type Required

skillId

Path

Identifies the skill.
Valid values: 1 – 255 characters

String

Yes

stage

Query

Filter based on the stage of the skill.
Valid values: development, live.

String

Yes

startTimestampFrom

Query

Beginning of the start time to query evaluation results.
Defined in ISO 8601 format, YYYY-MM-DDThh:mm:ssZ.

String

No

startTimestampTo

Query

End of the start time to query evaluation results.
Defined in ISO 8601 format, YYYY-MM-DDThh:mm:ssZ.

String

No

maxResults

Query

Maximum number of items to return in the response. The response might contain fewer items than the value you specify, but never contains more. Default: 20
Valid values: 1 – 1000.

Integer

No

nextToken

Query

Results might paginate. Include the value of nextToken from a previous List smart home capability evaluations response to get the next page of results. If not included, the response returns the first page of matching results.

String

No

access token

Header

LWA token.

String

Yes

Request body example

The request has no body.

Request body properties

The request has no body.

Response

A successful response returns HTTP 200 OK, along with a list of XYZ. On error, the response returns the appropriate HTTP status code and includes a response body with an error code and human readable message.

Response body example

The following example shows a response.

{
    "paginationContext": {
        "nextToken": "someToken"
    },
    "evaluations": [{
        "id": "213123",
        "startTimestamp": "2019-07-12T21:52:10.329Z",
        "endTimestamp": "2019-07-12T21:55:10.329Z",
        "status": "PASSED",
        "endpointId": "endpoint-001",
        "testPlanId": "test-plan-1234567",
        "sourceTestPlanIds": ["test-plan-1", "test-plan-2"]
    }]
}

Response body properties

Parameter Description Type

paginationContext

Included when there are more items that match the request.

Object

paginationContext.nextToken

Use the nextToken value in your next List smart home capability test plans request to retrieve the remaining test plans.

String

evaluations

List of evaluations that match the query parameters.

Array of objects

evaluations.id

Identifies the evaluation.
Use the ID returned from Create smart home capability evaluation.

String

evaluations.startTimestamp

Start time of the evaluation.
Defined in ISO 8601 format, YYYY-MM-DDThh:mm:ssZ.

String

evaluations.endTimestamp

End time of the evaluation.
Defined in ISO 8601 format, YYYY-MM-DDThh:mm:ssZ.

String

evaluations.status

Status of the evaluation.
Valid values: PASSED, FAILED, IN_PROGRESS, ERROR.

String

evaluations.endpointId

Unique identifier for the endpoint.

String

evaluations.testPlanId

Identifies the test plan.

String

evaluations.sourceTestPlanIds

List of all test plan identifiers included in the evaluation.

Array of strings

HTTP status codes

Status Description

200 OK

Response body contains an array of the customer lists. Default lists appear first in the array.

400 Bad Request

Indicates that one or more properties in the request body are invalid.
The following example shows the response body with the error and message.

{
     "message": "The property is outside the allowed range.",
     "code": "INVALID_STRING_LENGTH"
}

401 Unauthorized

Request didn't include the authorization token or the token is invalid or expired. Or, the client doesn't have access to the resource.

403 Forbidden

Indicates that the authorization token is valid, but the requested operation isn't allowed.

404 Not Found

Requested resource not found.

429 Too Many Requests

Skill has exceeded the permitted rate limit (specified number of requests per unit of time). The skill can retry the request by using exponential back-off.

500 Server Error

Error occurred on the server. The skill can retry the request by using exponential back-off.

503 Service Unavailable

Server is down for maintenance, overloaded, or otherwise unavailable to handle the incoming request.

List smart home capability test plans

Request available smart home capability test plans. To view the available test plans on GitHub, see alexa/alexa-smarthome/capability_evaluations/test_plans/.

Request

To list available test plans, you make a GET request to the smartHome/testing/capabilityEvaluations resource.

Request header example

Copied to clipboard.

GET /v1/skills/{skillId}/smartHome/testing/capabilityTestPlans?maxResults=1&nextToken=someToken
Host: api.amazonalexa.com
Content-Type: application/json
Authorization: Bearer {access token}
Accept: application/json
X-Amzn-RequestId: {request ID}

Request path and header parameters

Parameter Located in Description Type Required

skillId

Path

Identifies the skill.
Valid values: 1 – 255 characters.

String

Yes

maxResults

Query

Maximum number of items to return in the response. The response might contain fewer items than the value you specify, but never contains more. Default: 20
Valid values: 1 – 1000.

Integer

No

nextToken

Query

Results might paginate. Include the value of nextToken from a previous List smart home capability test plans response to get the next page of results. If not included, the response returns the first page of matching results.

String

No

access token

Header

LWA token.

String

Yes

request ID

Header

Unique identifier for the request, preferably a version 4 UUID.

String

Yes

Request body example

The request has no body.

Request body properties

The request has no body.

Response

A successful response returns HTTP 200 OK, along with a list of test plans. On error, the response returns the appropriate HTTP status code and includes a response body with an error code and human readable message.

Response body example

The following example shows a response.

{
    "paginationContext": {
        "nextToken": "someToken",
        "totalCount": 2
    },
    "testPlans": [{
        "id": "1231231",
        "name": "LightingTestPlan"
    }]
}

Response body properties

Property Description Type

paginationContext

Included when there are more items that match the request.

Object

paginationContext.nextToken

Use the nextToken value in your next List smart home capability test plans request to retrieve the remaining test plans.

String

paginationContext.totalCount

Total number of items that satisfy the request. This number might be higher than the number of items returned in this response.

Integer

testPlans

List of test plans that satisfy the request.

Array of objects

testPlans[].id

Identifies the test plan.

String

testPlans[].name

Human-readable test plan name.

String

HTTP status codes

Status Description

200 OK

Response body contains an array of test plans.

400 Bad Request

Indicates that one or more properties in the request body are invalid.
The following example shows the response body with the error and message.

{
     "message": "The property is outside the allowed range.",
     "code": "INVALID_STRING_LENGTH"
}

401 Unauthorized

Request didn't include the authorization token or the token is invalid or expired. Or, the client doesn't have access to the resource.

403 Forbidden

Indicates that the authorization token is valid, but the requested operation isn't allowed.

404 Not Found

Requested resource not found.

429 Too Many Requests

Skill has exceeded the permitted rate limit (specified number of requests per unit of time). The skill can retry the request by using exponential back-off.

500 Server Error

Error occurred on the server. The skill can retry the request by using exponential back-off.

503 Service Unavailable

Server is down for maintenance, overloaded, or otherwise unavailable to handle the incoming request.