Smart Home Evaluation API
The Smart Home Evaluation API lets you run a suite of automated tests against your skill for Lighting and Thermostat capabilities. This will help you:
- Shorten the overall skill development time.
- Cut costs by reducing manual testing efforts.
- Increase the likelihood of passing "Works with Alexa" certification.
- Improve your skill quality.
Note: The Smart Home Evaluation API is in beta and is subject to change at any time without notice.
- Prerequisites
- List Smart Home Capability test plans
- Create Smart Home Capability evaluation
- List Smart Home Capability evaluations
- Get Smart Home Capability evaluation
- Get Smart Home Capability Evaluation results
Prerequisites
- Your Alexa account needs to be account linked.
- Your test device should be discovered and visible in your Alexa account.
List Smart Home Capability test plans
This operation returns all of the test plan information. You can see the all the available test plans here: Test plans
Request
The API endpoint is https://api.amazonalexa.com. Each API request must have an Authorization header whose value is the access token retrieved from Login with Amazon.
HTTP method and URI path
Get /v1/skills/{skillId}/smartHome/testing/capabilityTestPlans?maxResults=<integer>&nextToken=<string>
Request parameters
| Parameter Type | Description | Parameter Type | Required? |
|---|---|---|---|
skillId |
The unique identifier of the skill. | Path | Yes |
maxResults |
The maximum number of items to return in the response. The default maximum is 20 if you don't include this parameter. When you include this parameter, the response might contain fewer items than the value you specify, but never contains more. Don't specify a value higher than 1000. | Query | No |
nextToken |
A token provided to continue returning results from a previous request. NextToken will only available when available results are equal to or more than requested results. | Query | No |
Request headers
Content-Type: application/json
Accept: application/json
X-Amzn-RequiestId: string
Response
A successful response returns HTTP 200 OK.
Response headers
Content-Type: application/json
X-Amzn-RequiestId: string
Response body structure
{
"paginationContext": {
"nextToken": string
"totalCount": integer
},
"testPlans": [
{
"id": string,
"description": string
}
]
}
Response body fields
paginationContext object
| Field | Description |
|---|---|
nextToken |
A token provided to continue returning results from a previous request. NextToken will only available when available results are equal to or more than requested results. |
totalCount |
The total number of items that satisfy the request. This number might be higher than the number of items returned in this response. |
testPlans object
| Field | Description |
|---|---|
id |
The unique identifier of the test plan. |
description |
The test plan description. |
Non-200 Response body
{
"message": string
}
| Field | Description |
|---|---|
message |
A human readable description of the error. |
Examples
The following examples show a sample request and response.
Sample request
GET /v1/skills/amzn1.ask.skill.12345/smartHome/testing/capabilityTestPlans?maxResults=1
Content-Type: application/json
Accept: application/json
X-Amzn-RequestId: 123-1234567-123
Sample response
HTTP 200 Ok
Content-Type: application/json
X-Amzn-RequestId: 123-1234567-123
Response body
{
"paginationContext": {
"nextToken": "879731421==",
"totalCount": 2
},
"testPlans": [
{
"id": "1231231",
"description": "LightingTestPlan description"
}
]
}
Create Smart Home Capability evaluation
Use this operation to create and start a capability evaluation against a Smart Home skill.
Request
The API endpoint is https://api.amazonalexa.com. Each API request must have an Authorization header whose value is the access token retrieved from Login with Amazon.
HTTP method and URI path
Post /v1/skills/{skillId}/smartHome/testing/capabilityEvaluations
Request parameters
| Parameter | Description | Parameter Type | Required? |
|---|---|---|---|
skillId |
The unique identifier of the skill. | Path | Yes |
Request headers
Content-Type: application/json
Accept: application/json
X-Amzn-RequiestId: string
Request body structure
{
"endpoint": {
"endpointId": string
},
"capabilityTestPlan": {
"id": string
},
"stage": string
}
Request body elements
| Parameter | Description | Required? |
|---|---|---|
stage |
The stage indicates the stage of the skill. Potential values are development and live. |
Yes |
endpoint object
| Parameter | Description | Required? |
|---|---|---|
endpointId |
A unique ID of the endpoint. | Yes |
capabilityTestPlan object
| Parameter | Description | Required? |
|---|---|---|
id |
A unique ID to identify each test plan. | Yes |
Response
A successful request returns HTTP 200 OK.
Response headers
Content-Type: application/json
X-Amzn-RequestId: string
Location: string
Response body structure
{
"id": string,
"endpoint": {
"endpointId": string
},
"capabilityTestPlan": {
"id": string
},
"stage": string
}
Response body elements
The response body includes the "request object".
| Parameter | Description |
|---|---|
id |
A unique id to identify each capability evaluation. |
Non-200 Response body
{
"message": string
}
| Parameter | Description |
|---|---|
message |
A human readable description of an error. |
Examples
The following examples show a sample request and response.
Sample request
POST /v1/skills/amzn1.ask.skill.12345/smartHome/testing/capabilityEvaluations
Content-Type: application/json
Accept: application/json
X-Amzn-RequestId: 123-1234567-123
{
"endpoint": {
"endpointId": "endpoint-001"
},
"capabilityTestPlan": {
"id": "test-plan-001"
},
"stage": "live"
}
Sample response
HTTP 200 Ok
Content-Type: application/json
X-Amzn-RequestId: 123-1234567-123
Location: /v1/skills/amzn1.ask.skill.12345/smartHome/testing/capabilityEvaluations/testRunId12312
{
"id": "testRunId12312",
"endpoint": {
"endpointId": "endpoint-001"
},
"capabilityTestPlan": {
"id": " test-plan-001"
},
"stage": "live"
}
List Smart Home Capability evaluations
Returns all of the capability evaluation information for a given skill id.
Request
The API endpoint is https://api.amazonalexa.com. Each API request must have an Authorization header whose value is the access token retrieved from Login with Amazon.
HTTP method and URI path
Get /v1/skills/{skillId}/smartHome/testing/capabilityEvaluations?stage=<string>&startTimestampFrom=<string>&startTimestampTo=<string>&maxResults=<integer>&nextToken=<string>
Request parameters
| Parameter | Description | Parameter Type | Required? |
|---|---|---|---|
skillId |
The unique identifier of the skill. | Path | Yes |
stage |
The stage indicates the stage of the skill. Potential values are development and live. |
Query | Yes |
startTimestampFrom |
The left limit of the start time to query evaluation history. The timestamp format is: yyyy-MM-dd'T'HH:mm:ss.SSS'Z. An example of the format is: 2019-10-10T23:37:37.982Z. | Query | No |
startTimestampTo |
The right limit of the start time to query evaluation history. The timestamp format is: yyyy-MM-dd'T'HH:mm:ss.SSS'Z. An example of the format is: 2019-10-10T23:37:37.982Z. | Query | No |
maxResults |
The maximum number of items to return in the response. The default maximum is 20 if you don't include this parameter. When you include this parameter, the response might contain fewer items than the value you specify, but never contains more. Don't specify a value higher than 1000. | Query | No |
nextToken |
A token provided to continue returning results from a previous request. NextToken will only available when available results are equal to or more than requested results. | Query | No |
Request headers
Content-Type: application/json
Accept: application/json
X-Amzn-RequestId: string
Response
A successful request returns HTTP 200 Ok.
Response headers
Content-Type: application/json
X-Amzn-RequestId: string
HTTP 200 Ok
{
"paginationContext": {
"nextToken": string,
"totalCount": integer
},
"evaluations": [
{
"id": string,
"status": string,
"startTimestamp": string,
"endTimestamp": string,
}
]
}
paginationContext object
| Parameter | Description |
|---|---|
nextToken |
A token provided to continue returning results from a previous request. NextToken will only available when available results are equal to or more than requested results. |
totalCount |
The total number of items that satisfy the request. This number might be higher than the number of items returned in this response. |
Evaluation object
| Parameter | Description |
|---|---|
id |
A unique id to identify each capability evaluation. |
status |
The status of the evaluation. The potential values are: PASSED, FAILED, IN_PROGRESS, or ERROR. |
startTimestamp |
The start time of the evaluation. An example of the format is: yyyy-MM-dd'T'HH:mm:ss.SSS'Z . |
endTimestamp |
The end time of the evaluation. An example of the format is: yyyy-MM-dd'T'HH:mm:ss.SSS'Z. |
Non-200 Response body
{
"message": string
}
| Parameter | Description |
|---|---|
message |
A human readable description of an error. |
Examples
The following examples show a sample request and response.
Sample request
GET /v1/skills/amzn1.ask.skill.12345/smartHome/testing/capabilityEvaluations?maxResults=1
Content-Type: application/json
Accept: application/json
X-Amzn-RequestId: 123-1234567-123
Sample response
HTTP 200 Ok
Content-Type: application/json
X-Amzn-RequestId: 123-1234567-123
{
"paginationContext": {
"nextToken": "879731421==",
"totalCount": 2
},
"evaluations": [
{
"startTimestamp": "2019-07-12T21:52:10.329Z",
"endTimestamp": "2019-07-12T21:55:10.329Z",
"status": "PASSED",
"id": "213123"
}
]
}
Get Smart Home Capability evaluation
Get top-level information and status of a Smart Home capability evaluation.
Request
The API endpoint is https://api.amazonalexa.com. Each API request must have an Authorization header whose value is the access token retrieved from Login with Amazon.
HTTP method and URI path
GET /v1/skills/{skillId}/smartHome/testing/capabilityEvaluations/{evaluationId}
Request parameters
| Parameter | Description | Parameter Type | Required? |
|---|---|---|---|
skillId |
The unique identifier of the skill. | Path | Yes |
evaluationId |
A unique id to identify each capability evaluation. | Path | Yes |
Request headers
Content-Type: application/json
Accept: application/json
X-Amzn-RequestId: string
Response
A successful request returns HTTP 200 Ok.
Response headers
Content-Type: application/json
X-Amzn-RequestId: string
Response body structure
{
"startTimestamp": string,
"endTimestamp": string,
"status": string,
"error": {
"message": string
},
"metrics": {
"totalTestCases": integer,
"passedTestCases": integer,
"failedTestCases": integer,
},
"input": {
"endpoint": {
"endpointId": string
},
"capabilityTestPlan": {
"id": string
},
"stage": string
}
}
Response body elements
| Parameter | Description |
|---|---|
startTimestamp |
The start time of the evaluation. An example of the format is: yyyy-MM-dd'T'HH:mm:ss.SSS'Z . |
endTimestamp |
The end time of the evaluation. An example of the format is: yyyy-MM-dd'T'HH:mm:ss.SSS'Z. |
status |
The status of the evaluation. The potential values are: PASSED, FAILED, IN_PROGRESS, or ERROR. |
Error object format
| Parameter | Description |
|---|---|
message |
A human readable description of the error. |
Metrics object
| Parameter | Description |
|---|---|
totalTestCases |
The total number of the test cases. |
passedTestCases |
The number of test cases that passed. |
failedTestCases |
The number of test cases that failed. |
Input object
The request body in Create Smart Home Capability Evaluation Operation.
Non-200 Response body
{
"message": string
}
| Parameter | Description |
|---|---|
message |
A human readable description of an error. |
Examples
The following examples show a sample request and response.
Sample request
GET /v1/skills/amzn1.ask.skill.12345/smartHome/testing/capabilityEvaluations/2131231
Content-Type: application/json
Accept: application/json
X-Amzn-RequestId: 123-1234567-123
Sample response
Content-Type: application/json
X-Amzn-RequestId: 123-1234567-123
{
"startTimestamp": "2019-07-17T06:45:40.689Z",
"endTimestamp": "2019-07-17T06:55:40.689Z",
"status": "PASSED",
"metrics": {
"totalTestCases": 2,
"passedTestCases": 2,
"failedTestCases": 0
},
"input": {
"endpoint": {
"endpointId": "endpoint-001"
},
"capabilityTestPlan": {
"id": "test-plan-001"
},
"stage": "live"
}
}
Get Smart Home Capability Evaluation results
Get test case results for an evaluation run.
Request
The API endpoint is https://api.amazonalexa.com. Each API request must have an Authorization header whose value is the access token retrieved from Login with Amazon.
HTTP method and URI path
Get /v1/skills/{skillId}/smartHome/testing/capabilityEvaluations/{evaluationId}/results?maxResults=<integer>&offset=<integer>
Request parameters
| Parameter | Description | Parameter Type | Required? |
|---|---|---|---|
skillId |
The unique identifier of the skill. | Path | Yes |
evaluationId |
A unique id to identify each capability evaluation. | Path | Yes |
maxResults |
The maximum number of items to return in the response. The default maximum is 20 if you don't include this parameter. When you include this parameter, the response might contain fewer items than the value you specify, but never contains more. Don't specify a value higher than 1000. | Query | No |
offSet |
The offset of the result page to be returned. | Query | No |
Request headers
Content-Type: application/json
Accept: application/json
X-Amzn-RequestId: string
Response
A successful request returns HTTP 200 Ok.
Response headers
Content-Type: application/json
X-Amzn-RequestId: string
Response body structure
{
"paginationContext": {
"offset": integer,
"totalCount": integer
},
"testCaseResults": [
{
"name": string,
"status": string,
"expectedCapabilityStates": [
{
"namespace": string,
"name": string,
"comparison": string,
"value": object
}
],
"directive": {
"header": {
"namespace": string,
"name": string,
},
"payload": object
},
"actualCapabilityStates": [
{
"namespace": string,
"name": string,
"value": object
}
],
"error": {
"message": string
}
}
]
}
paginationContext object
| Parameter | Description |
|---|---|
offset |
The offset of the result page to be returned. |
totalCount |
The total number of items that satisfy the request. This number might be higher than the number of items returned in this response. |
testCaseResult object
| Parameter | Description |
|---|---|
name |
The test case name. |
status |
The status of the evaluation. The potential values are: PASSED, FAILED, or ERROR. |
expectedCapabilityState object
| Parameter | Description |
|---|---|
namespace |
A string that specifies the category of the capability. For example: Alexa.PowerController. |
name |
The property name such as color or brightness. |
comparison |
The operation value for comparing the capability state. The example values are: EQUAL_TO, GREATER_THAN, and LESS_THAN. |
value |
The value of the property. |
directive object
| Parameter | Description |
|---|---|
namespace |
A string that specifies the category of the capability. For example: Alexa.PowerController. |
name |
The name of the directive such as TurnOn or TurnOff. |
actualCapabilityState object
| Parameter | Description |
|---|---|
namespace |
A string that specifies the category of the capability. For example: Alexa.PowerController. |
name |
The property name such as color or brightness |
comparison |
The operation value for comparing the capability state. The example values are: EQUAL_TO, GREATER_THAN, and LESS_THAN. |
value |
The property name such as color or brightness. |
Error object
| Parameter | Description |
|---|---|
message |
A human readable description of the error. |
Non-200 Response body
{
"message": string
}
| Parameter | Description |
|---|---|
message |
A human readable description of the error. |
Examples
The following examples show a sample request and response.
Sample request
GET /v1/skills/amzn1.ask.skill.12345/smartHome/testing/capabilityEvaluations/213123/results?maxResults=1
Content-Type: application/json
Accept: application/json
X-Amzn-RequestId: 123-1234567-123
Sample response
HTTP 200 Ok
Content-Type: application/json
X-Amzn-RequestId: 123-1234567-123
{
"paginationContext": {
"offset": "1",
"totalCount": 2
},
"testCaseResults": [
{
"name": "LightingTestCase1",
"status": "PASSED",
"expectedCapabilityStates": [
{
"namespace": "Alexa.PowerController",
"name": "powerState",
"value": "ON"
},
{
"namespace": "Alexa.BrightnessController",
"name": "brightness",
"value": 20
}
],
"directive": {
"header": {
"namespace": "Alexa.BrightnessController",
"name": "setBrightness"
},
"payload": {
"brightness": 20
}
},
"actualCapabilityStates": [
{
"namespace": "Alexa.PowerController",
"name": "powerState",
"value": "ON"
},
{
"namespace": "Alexa.BrightnessController",
"name": "brightness",
"value": 20
}
]
}
]
}
Errors
An unsuccessful request returns one of the following:
Non-200 Response exception
| Status code | Description |
|---|---|
HTTP 400 |
There's a validation error. |
HTTP 401 |
The authorization token is invalid or expired or doesn't have access to the resource. |
HTTP 403 |
The operation being requested isn't allowed. |
HTTP 404 |
The resource you requested is not found. |
HTTP 429 |
You have exceeded the permitted request limit. Throttling criteria includes total requests, per API, ClientId, and CustomerId. |
HTTP 500 |
There's a internal server error. |