Smart Home Skill Evaluation REST API Reference

Note: Submit your skill and see it come to life.

Use the Smart Home Evaluation REST API to start evaluations and get evaluation results. Smart home evaluations verify that you implemented the Alexa smart home interfaces in your skill correctly.

If you want to run the Smart Home tests in the Alexa developer console, see Use the Smart Home Test Tool.

API endpoint

The endpoint of the Smart Home Evaluation API is https://api.amazonalexa.com.

Authentication

Each API request must have an authorization header whose value is the access token retrieved from Login with Amazon (LWA). For details, see Get an Access Token for SMAPI.

Operations

The Smart Home Evaluation API includes the following operations.

Operation	HTTP method and URI
Create smart home capability evaluation	`POST /v1/skills/{skillId}/smartHome/testing/capabilityEvaluations`
Get smart home capability evaluation status	`GET /v1/skills/{skillId}/smartHome/testing/capabilityEvaluations/{evaluationId}`
Get smart home capability evaluation results	`GET /v1/skills/{skillId}/smartHome/testing/capabilityEvaluations/{evaluationId}/results?maxResults={maxResults}&offset={offset}`
List smart home capability evaluations	`GET /v1/skills/{skillId}/smartHome/testing/capabilityEvaluations?stage={stage}&startTimestampFrom={startTimestampFrom}&startTimestampTo={startTimestampTo}&maxResults={maxResults}&nextToken={nextToken}`
List smart home capability test plans	`GET /v1/skills/{skillId}/smartHome/testing/capabilityTestPlans?maxResults={maxResults}&nextToken={nextToken}`

Create smart home capability evaluation

Create and start a capability evaluation against the specified smart home skill.

Before you create an evaluation, make sure that you do the following test setup:

Enable your skill in the Alexa app.
Your skill configures account linking and you link your device account with your Alexa account successfully.
Alexa discovered your test device successfully and you can view your device in the Alexa app.

Request

To create an evaluation, you make a POST request to the smartHome/testing/capabilityEvaluations resource.

Request path and header example

Copied to clipboard.

POST /v1/skills/{skillId}/smartHome/testing/capabilityEvaluations
Host: api.amazonalexa.com
Content-Type: application/json
Accept: application/json
Authorization: Bearer {access token}
X-Amzn-RequestId: {request ID}

Request path and header parameters

Parameter	Located in	Description	Type	Required
`skillId`	Path	Identifies the skill. Valid values: 1 – 255 characters.	String	Yes
`access token`	Header	LWA token.	String	Yes
`request ID`	Header	Unique identifier for the request, preferably a version 4 UUID.	String	Yes

Request body example

Copied to clipboard.

{
    "endpoint": {
        "endpointId": "endpoint-001"
    },
    "capabilityTestPlan": {
        "id": "test-plan-001"
    },
    "stage": "development"
}

Request body properties

Property	Description	Type	Required
`endpoint`	Represents a connected device or component associated with a user's device cloud account.	Object	Yes
`endpoint.endpointId`	Unique identifier for the endpoint. Valid value: Up to 256 characters. Can include letters, numbers, spaces, and the following special characters: `_ - = # ; : ? @ &`	String	Yes
`capabilityTestPlan`	Represents the test plan to evaluate the skill against.	Object	Yes
`capabilityTestPlan.id`	Identifies the test plan.	String	Yes
`stage`	Indicates the stage of the skill. Valid values: `development`, `live`.	String	Yes

Response

A successful response returns HTTP 200 OK, along with the identifier of the evaluation to indicate that the evaluation started successfully. On error, the response returns the appropriate HTTP status code and includes a response body with an error code and human readable message.

Response body example

{
    "id": "testRunId12312",
    "endpoint": {
        "endpointId": "endpoint-001"
    },
    "capabilityTestPlan": {
        "id": "test-plan-001"
    },
    "stage": "development"
}

Response body properties

Property	Description	Type	Required
`id`	Identifies the evaluation.	String	Yes
`endpoint`	Represents a connected device or component associated with a user's device cloud account.	Object	Yes
`endpoint.endpointId`	Unique identifier for the endpoint. Valid value: Up to 256 characters. Can include letters, numbers, spaces, and the following special characters: `_ - = # ; : ? @ &`	String	Yes
`capabilityTestPlan`	Represents the test plan to evaluate the skill against.	Object	Yes
`capabilityTestPlan.id`	Identifies the test plan.	String	Yes
`stage`	Indicates the stage of the skill. Valid values: `development`, `live`.	String	Yes

HTTP status codes

Status	Description
`200 OK`	Response body contains the evaluation identifier indicating that the evaluation started.
`400 Bad Request`	Indicates that one or more properties in the request body are invalid. The following example shows the response body with the error and message. `{ "message": "The property is outside the allowed range.", "code": "INVALID_STRING_LENGTH" }`
`401 Unauthorized`	Request didn't include the authorization token or the token is invalid or expired. Or, the client doesn't have access to the resource.
`403 Forbidden`	Indicates that the authorization token is valid, but the requested operation isn't allowed.
`404 Not Found`	Requested resource not found.
`429 Too Many Requests`	Skill has exceeded the permitted rate limit (specified number of requests per unit of time). The skill can retry the request by using exponential back-off.
`500 Server Error`	Error occurred on the server. The skill can retry the request by using exponential back-off.
`503 Service Unavailable`	Server is down for maintenance, overloaded, or otherwise unavailable to handle the incoming request.

Get smart home capability evaluation status

Get the status of an evaluation.

Request

To get the evaluation status, you make a GET request to the smartHome/testing/capabilityEvaluations resource.

Request path and header example

Copied to clipboard.

GET /v1/skills/{skillId}/smartHome/testing/capabilityEvaluations/{evaluationId}
Host: api.amazonalexa.com
Content-Type: application/json
Authorization: Bearer {access token}
Accept: application/json
X-Amzn-RequestId: {request ID}

Request path and header parameters

Parameter	Located in	Description	Type	Required
`skillId`	Path	Identifies the skill. Valid values: 1 – 255 characters.	String	Yes
`evaluationId`	Path	Identifies the evaluation. Use the ID returned from Create smart home capability evaluation.	String	Yes
`access token`	Header	LWA token.	String	Yes
`request ID`	Header	Unique identifier for the request, preferably a version 4 UUID.	String	Yes

Request body example

The request has no body.

Request body properties

The request has no body.

Response

A successful response returns HTTP 200 OK, along with the evaluation status and metrics. On error, the response returns the appropriate HTTP status code and includes a response body with an error code and human readable message.

Response body example

The following example shows a response.

{
    "startTimestamp": "2019-07-17T06:45:40.689Z",
    "endTimestamp": "2019-07-17T06:55:40.689Z",
    "status": "PASSED",
    "metrics": {
        "PowerController": {
            "totalTestCases": 2,
            "passedTestCases": 2,
            "failedTestCases": 0,
            "errorTestCases": 0
        }
    },
    "input": {
        "endpoint": {
            "endpointId": "endpoint-001"
        },
        "capabilityTestPlan": {
            "id": "test-plan-001"
        },
        "stage": "live"
    }
}

Response body properties

Property	Description	Type
`startTimestamp`	Start time of the evaluation. Defined in ISO 8601 format, `YYYY-MM-DDThh:mm:ssZ`.	String
`endTimestamp`	End time of the evaluation. Defined in ISO 8601 format, `YYYY-MM-DDThh:mm:ssZ`.	String
`status`	Status of the evaluation. Valid values: `PASSED`, `FAILED`, `IN_PROGRESS`, `ERROR`.	String
`error`	(Optional) Included when an error occurs retrieving the evaluation status.	Object
`error.code`	Error code. Valid values: `CAPABILITY_NOT_SUPPORTED`, `DISCOVERY_FAILED`, `MULTIPLE_MATCHED_ENDPOINTS`, `MULTIPLE_MATCHED_TEST_PLANS`, `TEST_CASE_TIME_OUT`, `NO_SUCH_ENDPOINT`, `NO_SUCH_SKILL_STAGE`, `NO_SUCH_TEST_PLAN`.	String
`error.message`	Human-readable description of the error.	String
`metrics`	Describes the number of test cases in the evaluation.	Object
`metrics.<name>`	Name of the test plan.	Object
`metrics.<name>.totalTestCases`	Total number of test cases in the test plan.	Integer
`metrics.<name>.passedTestCases`	Number of test cases that passed.	Integer
`metrics.<name>.failedTestCases`	Number of test cases that failed.	Integer
`metrics.<name>.errorTestCases`	Number of test cases that didn't complete due to an error condition, such as unplugged device.	Integer
`input`	Evaluation details given in Create smart home capability evaluation.	Type
`input.endpoint`	Represents a connected device or component associated with a user's device cloud account.	Object
`input.endpoint.endpointId`	Unique identifier for the endpoint.	String
`input.capabilityTestPlan`	Represents the test plan to evaluate the skill against.	Object
`input.capabilityTestPlan.id`	Identifies the test plan.	String
`input.stage`	Indicates the stage of the skill.	String

HTTP status codes

Status	Description
`200 OK`	Response body contains an array of the customer lists. Default lists appear first in the array.
`400 Bad Request`	Indicates that one or more properties in the request body are invalid. The following example shows the response body with the error and message. `{ "message": "The property is outside the allowed range.", "code": "INVALID_STRING_LENGTH" }`
`401 Unauthorized`	Request didn't include the authorization token or the token is invalid or expired. Or, the client doesn't have access to the resource.
`403 Forbidden`	Indicates that the authorization token is valid, but the requested operation isn't allowed.
`404 Not Found`	Requested resource not found.
`429 Too Many Requests`	Skill has exceeded the permitted rate limit (specified number of requests per unit of time). The skill can retry the request by using exponential back-off.
`500 Server Error`	Error occurred on the server. The skill can retry the request by using exponential back-off.
`503 Service Unavailable`	Server is down for maintenance, overloaded, or otherwise unavailable to handle the incoming request.

Get smart home capability evaluation results

Get the results of the specified evaluation.

Request

To get the results of an evaluation, you make a GET request to the smartHome/testing/capabilityEvaluations resource.

Request path and header example

Copied to clipboard.

GET /v1/skills/{skillId}/smartHome/testing/capabilityEvaluations/{evaluationId}/results?maxResults=2&offset=1
Host: api.amazonalexa.com
Content-Type: application/json
Authorization: Bearer {access token}
Accept: application/json

Request path and header parameters

Parameter	Located in	Description	Type	Required
`skillId`	Path	Identifies the skill. Valid values: 1 – 255 characters.	String	Yes
`evaluationId`	Path	Identifies the evaluation. Use the ID returned from Create smart home capability evaluation.	String	Yes
`maxResults`	Query	Maximum number of items to return in the response. The response might contain fewer items than the value you specify, but never contains more. Default: 20 Valid values: 1 – 1000.	Integer	No
`offset`	Query	Offset of the result page to return.	Integer	No
`access token`	Header	LWA token.	String	Yes
`access token`	Header	Access token used for the customer. Set to `apiAccessToken` property of the `System` object from the `LaunchRequest` or `IntentRequest`.	String	Yes

Request body example

The request has no body.

Request body properties

The request has no body.

Response

A successful response returns HTTP 200 OK, along with the evaluation results. On error, the response returns the appropriate HTTP status code and includes a response body with an error code and human readable message.

Response body example

The following example shows a response.

{
    "paginationContext": {
        "offset": "1",
        "totalCount": 2
    },
    "testCaseResults": [{
        "name": "LightingTestCase1",
        "status": "PASSED",
        "capability": "PowerController",
        "expectedCapabilityStates": [{
                "namespace": "Alexa.PowerController",
                "name": "powerState",
                "value": "ON"
            },
            {
                "namespace": "Alexa.BrightnessController",
                "name": "brightness",
                "value": 20
            }
        ],
        "directive": {
            "header": {
                "namespace": "Alexa.BrightnessController",
                "name": "setBrightness"
            },
            "payload": {
                "brightness": 20
            }
        },
        "actualCapabilityStates": [{
                "namespace": "Alexa.PowerController",
                "name": "powerState",
                "value": "ON"
            },
            {
                "namespace": "Alexa.BrightnessController",
                "name": "brightness",
                "value": 20
            }
        ]
    }]
}

Response body properties

Property	Description	Type
`paginationContext`	Included when there are more items that match the request.	Object
`paginationContext.offset`	Offset in the result page.	Integer
`paginationContext.totalCount`	Total number of items that satisfy the request. This number might be higher than the number of items returned in this response.	Integer
`testCaseResults`	List of test plans that satisfy the request.	Array of objects
`testCaseResults.name`	Human-readable test case name.	String
`testCaseResults.status`	Status of the test case. Valid values: `PASSED`, `FAILED`, `ERROR`.	String
`testCaseResults.expectedCapabilityStates`	Smart Home API interfaces and expected property values.	Array of objects
`testCaseResults.expectedCapabilityStates.namespace`	Interface name, such as `Alexa.PowerController`.	String
`testCaseResults.expectedCapabilityStates.name`	Property name, such as `brightness`.	String
`testCaseResults.expectedCapabilityStates.comparison`	Operation used to compare the expected property value with the actual property value. Valid values: `EQUAL_TO`, `GREATER_THAN`, `LESS_THAN`.	String
`testCaseResults.expectedCapabilityStates.value`	Expected property value results.	String
`testCaseResults.directive`	Directive sent to the skill.	Object
`testCaseResults.directive.header`	Header information.	Object
`testCaseResults.directive.header.namespace`	Name of the interface.	String
`testCaseResults.directive.header.name`	Name of the directive.	String
`testCaseResults.directive.payload`	JSON payload included in the directive.	Object
`testCaseResults.actualCapabilityStates`	Actual property value results returned from the skill.	Array of objects
`testCaseResults.actualCapabilityStates.namespace`	Interface name, such as `Alexa.PowerController`.	String
`testCaseResults.actualCapabilityStates.name`	Property name, such as `brightness`.	String
`testCaseResults.actualCapabilityStates.comparison`	Operation used to compare the expected property value with the actual property value. Valid values: `EQUAL_TO`, `GREATER_THAN`, `LESS_THAN`.	String
`testCaseResults.actualCapabilityStates.value`	Expected property value.	String
`testCaseResults.expectedResponse`	Expected response to the directive.	String
`testCaseResults.actualResponse`	Actual response to the directive.	String
`error`	(Optional) Included when an error occurs.	Object
`error.code`	Error code. Valid values: `CAPABILITY_NOT_SUPPORTED`, `DISCOVERY_FAILED`, `MULTIPLE_MATCHED_ENDPOINTS`, `MULTIPLE_MATCHED_TEST_PLANS`, `TEST_CASE_TIME_OUT`, `NO_SUCH_ENDPOINT`, `NO_SUCH_SKILL_STAGE`, `NO_SUCH_TEST_PLAN`.	String
`error.message`	Human-readable description of the error.	String

HTTP status codes

Status	Description
`200 OK`	Response body contains an array of the customer lists. Default lists appear first in the array.
`400 Bad Request`	Indicates that one or more properties in the request body are invalid. The following example shows the response body with the error and message. `{ "message": "The property is outside the allowed range.", "code": "INVALID_STRING_LENGTH" }`
`401 Unauthorized`	Request didn't include the authorization token or the token is invalid or expired. Or, the client doesn't have access to the resource.
`403 Forbidden`	Indicates that the authorization token is valid, but the requested operation isn't allowed.
`404 Not Found`	Requested resource not found.
`429 Too Many Requests`	Skill has exceeded the permitted rate limit (specified number of requests per unit of time). The skill can retry the request by using exponential back-off.
`500 Server Error`	Error occurred on the server. The skill can retry the request by using exponential back-off.
`503 Service Unavailable`	Server is down for maintenance, overloaded, or otherwise unavailable to handle the incoming request.

List smart home capability evaluations

List the evaluations for a specified skill.

Request

To list the evaluations for a skill, you make a GET request to the smartHome/testing/capabilityEvaluations resource.

Request path and header example

Copied to clipboard.

GET /v1/skills/{skillId}/smartHome/testing/capabilityEvaluations?stage=development&startTimestampFrom=2022-07-12T09:00:00.000Z&startTimestampTo=2022-07-15T11:00:00.000Z&maxResults=10&nextToken=someToken
Host: api.amazonalexa.com
Content-Type: application/json
Authorization: Bearer {access token}
Accept: application/json

Request path and header parameters

Parameter	Located in	Description	Type	Required
`skillId`	Path	Identifies the skill. Valid values: 1 – 255 characters	String	Yes
`stage`	Query	Filter based on the stage of the skill. Valid values: `development`, `live`.	String	Yes
`startTimestampFrom`	Query	Beginning of the start time to query evaluation results. Defined in ISO 8601 format, `YYYY-MM-DDThh:mm:ssZ`.	String	No
`startTimestampTo`	Query	End of the start time to query evaluation results. Defined in ISO 8601 format, `YYYY-MM-DDThh:mm:ssZ`.	String	No
`maxResults`	Query	Maximum number of items to return in the response. The response might contain fewer items than the value you specify, but never contains more. Default: 20 Valid values: 1 – 1000.	Integer	No
`nextToken`	Query	Results might paginate. Include the value of `nextToken` from a previous `List smart home capability evaluations` response to get the next page of results. If not included, the response returns the first page of matching results.	String	No
`access token`	Header	LWA token.	String	Yes

Request body example

The request has no body.

Request body properties

The request has no body.

Response

A successful response returns HTTP 200 OK, along with a list of XYZ. On error, the response returns the appropriate HTTP status code and includes a response body with an error code and human readable message.

Response body example

The following example shows a response.

{
    "paginationContext": {
        "nextToken": "someToken"
    },
    "evaluations": [{
        "id": "213123",
        "startTimestamp": "2019-07-12T21:52:10.329Z",
        "endTimestamp": "2019-07-12T21:55:10.329Z",
        "status": "PASSED",
        "endpointId": "endpoint-001",
        "testPlanId": "test-plan-1234567",
        "sourceTestPlanIds": ["test-plan-1", "test-plan-2"]
    }]
}

Response body properties

Property	Description	Type
`paginationContext`	Included when there are more items that match the request.	Object
`paginationContext.nextToken`	Use the `nextToken` value in your next `List smart home capability test plans` request to retrieve the remaining test plans.	String
`evaluations`	List of evaluations that match the query parameters.	Array of objects
`evaluations.id`	Identifies the evaluation. Use the ID returned from Create smart home capability evaluation.	String
`evaluations.startTimestamp`	Start time of the evaluation. Defined in ISO 8601 format, `YYYY-MM-DDThh:mm:ssZ`.	String
`evaluations.endTimestamp`	End time of the evaluation. Defined in ISO 8601 format, `YYYY-MM-DDThh:mm:ssZ`.	String
`evaluations.status`	Status of the evaluation. Valid values: `PASSED`, `FAILED`, `IN_PROGRESS`, `ERROR`.	String
`evaluations.endpointId`	Unique identifier for the endpoint.	String
`evaluations.testPlanId`	Identifies the test plan.	String
`evaluations.sourceTestPlanIds`	List of all test plan identifiers included in the evaluation.	Array of strings

HTTP status codes

Status	Description
`200 OK`	Response body contains an array of the customer lists. Default lists appear first in the array.
`400 Bad Request`	Indicates that one or more properties in the request body are invalid. The following example shows the response body with the error and message. `{ "message": "The property is outside the allowed range.", "code": "INVALID_STRING_LENGTH" }`
`401 Unauthorized`	Request didn't include the authorization token or the token is invalid or expired. Or, the client doesn't have access to the resource.
`403 Forbidden`	Indicates that the authorization token is valid, but the requested operation isn't allowed.
`404 Not Found`	Requested resource not found.
`429 Too Many Requests`	Skill has exceeded the permitted rate limit (specified number of requests per unit of time). The skill can retry the request by using exponential back-off.
`500 Server Error`	Error occurred on the server. The skill can retry the request by using exponential back-off.
`503 Service Unavailable`	Server is down for maintenance, overloaded, or otherwise unavailable to handle the incoming request.

List smart home capability test plans

Request available smart home capability test plans. To view the available test plans on GitHub, see alexa/alexa-smarthome/capability_evaluations/test_plans/.

Request

To list available test plans, you make a GET request to the smartHome/testing/capabilityEvaluations resource.

Request path and header example

Copied to clipboard.

GET /v1/skills/{skillId}/smartHome/testing/capabilityTestPlans?maxResults=1&nextToken=someToken
Host: api.amazonalexa.com
Content-Type: application/json
Authorization: Bearer {access token}
Accept: application/json
X-Amzn-RequestId: {request ID}

Request path and header parameters

Parameter	Located in	Description	Type	Required
`skillId`	Path	Identifies the skill. Valid values: 1 – 255 characters.	String	Yes
`maxResults`	Query	Maximum number of items to return in the response. The response might contain fewer items than the value you specify, but never contains more. Default: 20 Valid values: 1 – 1000.	Integer	No
`nextToken`	Query	Results might paginate. Include the value of `nextToken` from a previous `List smart home capability test plans` response to get the next page of results. If not included, the response returns the first page of matching results.	String	No
`access token`	Header	LWA token.	String	Yes
`request ID`	Header	Unique identifier for the request, preferably a version 4 UUID.	String	Yes

Request body example

The request has no body.

Request body properties

The request has no body.

Response

A successful response returns HTTP 200 OK, along with a list of test plans. On error, the response returns the appropriate HTTP status code and includes a response body with an error code and human readable message.

Response body example

The following example shows a response.

{
    "paginationContext": {
        "nextToken": "someToken",
        "totalCount": 2
    },
    "testPlans": [{
        "id": "1231231",
        "name": "LightingTestPlan"
    }]
}

Response body properties

Property	Description	Type
`paginationContext`	Included when there are more items that match the request.	Object
`paginationContext.nextToken`	Use the `nextToken` value in your next `List smart home capability test plans` request to retrieve the remaining test plans.	String
`paginationContext.totalCount`	Total number of items that satisfy the request. This number might be higher than the number of items returned in this response.	Integer
`testPlans`	List of test plans that satisfy the request.	Array of objects
`testPlans[].id`	Identifies the test plan.	String
`testPlans[].name`	Human-readable test plan name.	String

HTTP status codes

Status	Description
`200 OK`	Response body contains an array of test plans.
`400 Bad Request`	Indicates that one or more properties in the request body are invalid. The following example shows the response body with the error and message. `{ "message": "The property is outside the allowed range.", "code": "INVALID_STRING_LENGTH" }`
`401 Unauthorized`	Request didn't include the authorization token or the token is invalid or expired. Or, the client doesn't have access to the resource.
`403 Forbidden`	Indicates that the authorization token is valid, but the requested operation isn't allowed.
`404 Not Found`	Requested resource not found.
`429 Too Many Requests`	Skill has exceeded the permitted rate limit (specified number of requests per unit of time). The skill can retry the request by using exponential back-off.
`500 Server Error`	Error occurred on the server. The skill can retry the request by using exponential back-off.
`503 Service Unavailable`	Server is down for maintenance, overloaded, or otherwise unavailable to handle the incoming request.

Smart Home Skill Evaluation REST API Reference

API endpoint

Authentication

Operations

Create smart home capability evaluation

Request

Request path and header example

Request path and header parameters

Request body example

Request body properties

Response

Response body example

Response body properties

HTTP status codes

Get smart home capability evaluation status

Request

Request path and header example

Request path and header parameters

Request body example

Request body properties

Response

Response body example

Response body properties

HTTP status codes

Get smart home capability evaluation results

Request

Request path and header example

Request path and header parameters

Request body example

Request body properties

Response

Response body example

Response body properties

HTTP status codes

List smart home capability evaluations

Request

Request path and header example

Request path and header parameters

Request body example

Request body properties

Response

Response body example

Response body properties

HTTP status codes

List smart home capability test plans

Request

Request path and header example

Request path and header parameters

Request body example

Request body properties

Response

Response body example

Response body properties

HTTP status codes

Related topics