Automatic Speech Recognition Evaluation REST API Reference
Use the Automatic Speech Recognition (ASR) Evaluation REST API to start evaluations, get results, and delete previous evaluations. ASR evaluations assess the accuracy of your skill against an annotation set. For ASR Annotation Set APIs, see Automatic Speech Recognition (ASR) Annotation Set API Reference.
If you want to use the ASR Evaluation tool in the Alexa developer console, see Automatic Speech Recognition (ASR) Evaluation tool.
API endpoint
The endpoint of the ASR Evaluation API is https://api.amazonalexa.com
.
Authentication
Each API request must have an authorization header whose value is a bearer token retrieved from Login with Amazon (LWA). For details, see Get an Access Token for SMAPI.
Operations
The ASR Evaluation API includes the following operations.
Operation | HTTP method and URI |
---|---|
| |
| |
| |
| |
|
Delete ASR evaluation
Delete the specified ASR evaluation, including in-progress evaluations.
Request
To delete an ASR evaluation, you make a DELETE
request to the asrEvaluations
resource.
Request path and header example
`DELETE /v1/skills/{skillId}/asrEvaluations/{evaluationId} HTTP/1.1`
Host: api.amazonalexa.com
Content-Type: application/json
Authorization: Bearer {access token}
Accept: application/json
Request path and header parameters
Parameter | Located in | Description | Type | Required |
---|---|---|---|---|
|
Path |
Identifies the skill. |
String |
Yes |
|
Path |
Identifies the ASR evaluation. |
String |
Yes |
|
Header |
String |
Yes |
Request body example
The request has no body.
Request body properties
The request has no body.
Response
A successful response returns HTTP 204 No Content
.
On error, the response returns the appropriate HTTP status code and includes a response body with an error code and human readable message.
Response body example
The response has no body.
Response body properties
The response has no body.
HTTP status codes
Status | Description |
---|---|
|
ASR evaluation deleted successfully. |
|
Server can't process the request due to a client error. For example, the payload is malformed, the annotation set is malformed or empty.
|
|
Authorization token is invalid, expired, or doesn't have access to the resource. |
|
The requested operation isn't allowed. |
|
The requested resource isn't found. For example, the
|
|
Request to run an evaluation for a given skill conflicts with an evaluation request that's currently in progress. |
|
User has exceeded the permitted rate limit. |
|
Server has encountered an error. |
|
Server is down for maintenance, overloaded, or otherwise unavailable to handle the incoming request. |
Get ASR evaluation results
Get the details results of the specified ASR evaluation. If you want the status of the evaluation only, use Get ASR evaluation status.
Request
To get the results of an evaluation, you make a GET
request to the asrEvaluations
resource.
Request path and header example
GET /v1/skills/{skillId}/asrEvaluations/{evaluationId}/results?status={status}&nextToken={nextToken}&maxResults={maxResults} HTTP/1.1
Host: api.amazonalexa.com
Content-Type: application/json
Authorization: Bearer {access token}
Accept: application/json
Request path and header parameters
Parameter | Located in | Description | Type | Required |
---|---|---|---|---|
|
Path |
Identifies the skill. |
String |
Yes |
|
Path |
Identifies the ASR evaluation. |
String |
Yes |
|
Query |
Identifies the results status. |
String |
No |
|
Query |
Token that you use to get the next set of results, after you receive a response with truncated results. Set this parameter to the value of |
String |
No |
|
Query |
Maximum number of results to return. |
Number |
No |
|
Header |
String |
Yes |
Request body example
The request has no body.
Request body properties
The request has no body.
Response
A successful response returns HTTP 200 OK
, along with the evaluation results.
On error, the response returns the appropriate HTTP status code and includes a response body with an error code and human readable message.
Response body example
{
"results": [{
"status": "PASSED",
"annotation": {
"evaluationWeight": 1,
"expectedTranscription": "hello world",
"filePathInUpload": "hello world.wav",
"uploadId": "upload.1",
"audioAsset": {
"downloadUrl": "https://presign.s3.aws.amazon/123412341234-hello.wav",
"expiryTime": "2018-10-25T08:25:04.679Z"
}
},
"output": {
"transcription": "hello world"
}
},
{
"status": "FAILED",
"annotation": {
"evaluationWeight": 8,
"expectedTranscription": "alexa what is the weather now",
"filePathInUpload": "weather.wav",
"uploadId": "upload.2",
"audioAsset": {
"downloadUrl": "https://presign.s3.aws.amazon/123412341234-weather.wav",
"expiryTime": "2018-10-25T08:25:04.679Z"
}
},
"error": {
"message": "An unexpected error occurred",
"code": "INTERNAL_SERVER_ERROR"
}
}
]
}
Response body properties
Property | Description | Type |
---|---|---|
|
Result of the ASR evaluation. Each item in the array represents a test case. |
Array of objects |
|
Identifies the status of the test case. |
String |
|
Information about the specific annotation used in this evaluation. |
Object |
|
Path to the audio file in the uploaded zip file. |
String |
|
Identifies the uploaded audio content. The Catalog API returns the identifier each time you create an upload. |
String |
|
Weight of the test case in an evaluation. Used to calculate metrics, such as overall error rate. |
Integer |
|
Expected transcription text for the input audio. |
String |
|
Specifies how to download the audio file. |
Object |
|
Identifies the presigned URL at which to download the audio file. |
String |
|
Expiration date for the download URL. |
String |
|
(Optional) Output of the evaluation. |
Object |
|
Actual transcription returned from ASR for the audio. |
String |
|
(Optional) Identifies the reason for the failed evaluation. |
Object |
|
Describes why the evaluation wasn't executed successfully. |
String |
|
Evaluation error code. |
String |
|
Indicates there are more matching results. If present, the response contains a subset of the results. When there are no more matching results, |
Object |
|
Identifies the next set of evaluation results to return. The token expires in 24 hours. |
String |
HTTP status codes
Status | Description |
---|---|
|
Response body contains a list of evaluation results. |
|
Server can't process the request due to a client error. For example, the payload is malformed, the annotation set is malformed or empty.
|
|
Authorization token is invalid, expired, or doesn't have access to the resource. |
|
The requested operation isn't allowed. |
|
The requested resource isn't found. For example, the
|
|
Request to run an evaluation for a given skill conflicts with an evaluation request that's currently in progress. |
|
User has exceeded the permitted rate limit. |
|
Server has encountered an error. |
|
Server is down for maintenance, overloaded, or otherwise unavailable to handle the incoming request. |
Get ASR evaluation status
Get the status of the specified ASR evaluation. If you want the detailed results of the evaluation, use Get ASR evaluation results.
Request
To get the status of an evaluation, you make a GET
request to the asrEvaluations
resource.
Request path and header example
GET /v1/skills/{skillId}/asrEvaluations/{evaluationId}/status HTTP/1.1
Host: api.amazonalexa.com
Content-Type: application/json
Authorization: Bearer {access token}
Accept: application/json
Request path and header parameters
Parameter | Located in | Description | Type | Required |
---|---|---|---|---|
|
Path |
Identifies the skill. |
String |
Yes |
|
Path |
Identifies the ASR evaluation. |
String |
Yes |
|
Header |
String |
Yes |
Request body example
The request has no body.
Request body properties
The request has no body.
Response
A successful response returns HTTP 200 OK
, along with the evaluation status.
On error, the response returns the appropriate HTTP status code and includes a response body with an error code and human readable message.
Response body example
{
"id": "evaluation.1",
"status": "COMPLETED",
"totalEvaluationCount": 2,
"completedEvaluation": 2,
"startTimestamp": "2018-10-25T08:25:04.679Z",
"request": {
"skill": {
"stage": "development",
"locale": "en-US"
},
"annotationSetId": "1234-1234-12341234"
},
"result": {
"status": "PASSED",
"metrics": {
"overallErrorRate": 0.5
}
}
}
Response body properties
Property | Description | Type |
---|---|---|
|
Indicates the status of the evaluation. A failure status indicates that the evaluation didn't start. |
String |
|
Request payload that triggered this evaluation. |
Object |
|
Information about the skill used in the evaluation. |
Object |
|
Indicates stage of the skill. |
String |
|
Country and language where the skill is offered. |
String |
|
Identifies the annotation set that's used in the evaluation. |
String |
|
Total number of evaluations requested to be run. |
Integer |
|
Number of completed evaluations. |
Integer |
|
Start time of the evaluations. |
String |
|
(Optional) Results of the evaluation. |
Object |
|
Metrics generated from the evaluation. |
Object |
|
Generated overall error rate of the evaluation. |
Double |
|
Evaluation result status. |
String |
|
(Optional) Indicates that the evaluation wasn't started. |
Object |
|
Describes why the evaluation wasn't started. |
String |
|
Evaluation error code. |
String |
HTTP status codes
Status | Description |
---|---|
|
Response body contains the evaluation status. |
|
Server can't process the request due to a client error. For example, the payload is malformed, the annotation set is malformed or empty.
|
|
Authorization token is invalid, expired, or doesn't have access to the resource. |
|
The requested operation isn't allowed. |
|
The requested resource isn't found. For example, the
|
|
Request to run an evaluation for a given skill conflicts with an evaluation request that's currently in progress. |
|
User has exceeded the permitted rate limit. |
|
Server has encountered an error. |
|
Server is down for maintenance, overloaded, or otherwise unavailable to handle the incoming request. |
List ASR evaluations
Get all completed annotation evaluations for the specified skill. You can add optional filters to the request.
Request
To list ASR evaluations, you make a GET
request to the asrEvaluations
resource.
Request path and header example
GET /v1/skills/{skillId}/asrEvaluations?locale={locale}&annotationSetId={annotationSetId}&stage={stage}&nextToken={nextToken}&maxResults={maxResults} HTTP/1.1
Host: api.amazonalexa.com
Content-Type: application/json
Authorization: Bearer {access token}
Accept: application/json
Request path and header parameters
Parameter | Located in | Description | Type | Required |
---|---|---|---|---|
|
Path |
Identifies the skill. |
String |
Yes |
|
Query |
Filter on the country and language where the skill is offered. If omitted, all locales are included in the response. |
String |
No |
|
Query |
Filters results to the specified annotation set. If omitted, all annotation sets are included in the response. |
String |
No |
|
Query |
Filter on the specified skill stage. If omitted, all stages are included in the response. |
String |
No |
|
Query |
Token that you use to get the next set of results, after you receive a response with truncated results. Set this parameter to the value of |
String |
No |
|
Query |
Maximum number of results to return. |
Number |
No |
|
Header |
String |
Yes |
Request body example
The request has no body.
Request body properties
The request has no body.
Response
A successful response returns HTTP 200 OK
, along with a list of evaluations.
On error, the response returns the appropriate HTTP status code and includes a response body with an error code and human readable message.
Response body example
{
"evaluations": [{
"id": "evaluation.1",
"status": "COMPLETED",
"startTimestamp": "2018-10-25T08:25:04.679Z",
"totalEvaluationCount": 10,
"completedEvaluationCount": 10,
"request": {
"skill": {
"stage": "development",
"locale": "en-US"
},
"annotationSetId": "annotation.set.1"
},
"result": {
"status": "PASSED",
"metrics": {
"overallErrorRate": 0.5
}
}
},
{
"id": "evaluation.2",
"status": "COMPLETED",
"totalEvaluationCount": 10,
"completedEvaluationCount": 10,
"startTimestamp": "2018-10-24T08:25:04.679Z",
"request": {
"skill": {
"stage": "live",
"locale": "en-GB"
},
"annotationSetId": "annotation.set.1"
},
"result": {
"status": "FAILED",
"metrics": {
"overallErrorRate": 0.5
}
}
},
{
"id": "evaluation.3",
"status": "FAILED",
"totalEvaluationCount": 10,
"completedEvaluationCount": 10,
"startTimestamp": "2018-10-24T08:25:04.679Z",
"request": {
"skill": {
"stage": "development",
"locale": "en-GB"
},
"annotationSetId": "annotation.set.1"
},
"error": {
"message": "An unexpected error occurred",
"code": "INTERNAL_SERVER_ERROR"
}
}
],
"paginationContext": {
"nextToken": "token.1"
}
}
Response body properties
Property | Description | Type |
---|---|---|
|
Array of evaluation runs based on the filter criteria. |
Array of objects |
|
Identifies the evaluation run. |
String |
|
Indicates the status of the evaluation. A failure status indicates that the evaluation didn't start. |
String |
|
Request payload that triggered this evaluation. |
Object |
|
Information about the skill used in the evaluation. |
Object |
|
Indicates stage of the skill. |
String |
|
Country and language where the skill is offered. |
String |
|
Identifies the annotation set that's used in the evaluation. |
String |
|
Total number of evaluations requested to be run. |
Integer |
|
Number of completed evaluations. |
Integer |
|
Start time of the evaluations. |
String |
|
(Optional) Results of the evaluation. |
Object |
|
Metrics generated from the evaluation. |
Object |
|
Generated overall error rate of the evaluation. |
Double |
|
Evaluation result status. |
String |
|
(Optional) Indicates that the evaluation wasn't started. |
Object |
|
Describes why the evaluation wasn't started. |
String |
|
Evaluation error code. |
String |
|
Indicates there are more matching results. If present, the response contains a subset of the results. When there are no more matching results, |
Object |
|
Identifies the next set of evaluation results to return. The token expires in 24 hours. |
String |
HTTP status codes
Status | Description |
---|---|
|
Response body contains a list of evaluation runs. |
|
Server can't process the request due to a client error. For example, the payload is malformed, the annotation set is malformed or empty.
|
|
Authorization token is invalid, expired, or doesn't have access to the resource. |
|
The requested operation isn't allowed. |
|
The requested resource isn't found. For example, the
|
|
Request to run an evaluation for a given skill conflicts with an evaluation request that's currently in progress. |
|
User has exceeded the permitted rate limit. |
|
Server has encountered an error. |
|
Server is down for maintenance, overloaded, or otherwise unavailable to handle the incoming request. |
Run ASR evaluation
Start an ASR evaluation against an existing annotation set. Concurrent evaluation requests for a given skill id aren't supported.
Request
To run an evaluation, you make a POST
request to the asrEvaluations
resource.
Request path and header example
POST /v1/skills/{skillId}/asrEvaluations HTTP/1.1
Host: api.amazonalexa.com
Content-Type: application/json
Authorization: Bearer {access token}
Accept: application/json
Request path and header parameters
Parameter | Located in | Description | Type | Required |
---|---|---|---|---|
|
Path |
Identifies the skill. |
String |
Yes |
|
Header |
String |
Yes |
Request body example
{
"skill": {
"stage": "live",
"locale": "en-US"
},
"annotationSetId": "annotation.set.1"
}
Request body properties
Property | Description | Type | Required |
---|---|---|---|
|
Information about the skill used in the evaluation. |
Object |
Yes |
|
Indicates stage of the skill. |
String |
Yes |
|
Country and language where the skill is offered. |
String |
Yes |
|
Identifies the annotation set to be used in the evaluation. |
String |
Yes |
Response
A successful response returns HTTP 200 OK
with a Location
parameter in the request header. The response body contains the evaluation ID. Use the ID to get the status and results of the evaluation.
On error, the response returns the appropriate HTTP status code and includes a response body with an error code and human readable message.
Response body example
{
"id": "evaluation.1"
}
Response body properties
Property | Description | Type |
---|---|---|
|
Identifies the evaluation run. |
String |
HTTP status codes
Status | Description |
---|---|
|
Empty annotation set created successfully. The response header contains a
|
|
Server can't process the request due to a client error. For example, the payload is malformed, the annotation set is malformed or empty.
|
|
Authorization token is invalid, expired, or doesn't have access to the resource. |
|
The requested operation isn't allowed. |
|
The requested resource isn't found. For example, the
|
|
Request to run an evaluation for a given skill conflicts with an evaluation request that's currently in progress. |
|
User has exceeded the permitted rate limit. |
|
Server has encountered an error. |
|
Server is down for maintenance, overloaded, or otherwise unavailable to handle the incoming request. |