Automatic Speech Recognition Evaluation REST API Reference

Use the Automatic Speech Recognition (ASR) Evaluation REST API to start evaluations, get results, and delete previous evaluations. ASR evaluations assess the accuracy of your skill against an annotation set. For ASR Annotation Set APIs, see Automatic Speech Recognition (ASR) Annotation Set API Reference.

If you want to use the ASR Evaluation tool in the Alexa developer console, see Automatic Speech Recognition (ASR) Evaluation tool.

API endpoint

The endpoint of the ASR Evaluation API is https://api.amazonalexa.com.

Authentication

Each API request must have an authorization header whose value is a bearer token retrieved from Login with Amazon (LWA). For details, see Get an Access Token for SMAPI.

Operations

The ASR Evaluation API includes the following operations.

Operation HTTP method and URI

Delete ASR evaluation

DELETE /v1/skills/{skillId}/asrEvaluations/{evaluationId}

Get ASR evaluation results

GET /v1/skills/{skillId}/asrEvaluations/{evaluationId}/results?status={status}&nextToken={nextToken}&maxResults={maxResults}

Get ASR evaluation status

GET /v1/skills/{skillId}/asrEvaluations/{evaluationId}/status

List ASR evaluations

GET /v1/skills/{skillId}/asrEvaluations?locale={locale}&annotationSetId={annotationSetId}&stage={stage}&nextToken={nextToken}&maxResults={maxResults}

Run ASR evaluation

POST /v1/skills/{skillId}/asrEvaluations

Delete ASR evaluation

Delete the specified ASR evaluation, including in-progress evaluations.

Request

To delete an ASR evaluation, you make a DELETE request to the asrEvaluations resource.

Request path and header example

Copied to clipboard.

`DELETE /v1/skills/{skillId}/asrEvaluations/{evaluationId} HTTP/1.1`
Host: api.amazonalexa.com
Content-Type: application/json
Authorization: Bearer {access token}
Accept: application/json

Request path and header parameters

Parameter Located in Description Type Required

skillId

Path

Identifies the skill.
Valid values: 1 - 255 characters.

String

Yes

evaluationId

Path

Identifies the ASR evaluation.

String

Yes

access token

Header

LWA token.

String

Yes

Request body example

The request has no body.

Request body properties

The request has no body.

Response

A successful response returns HTTP 204 No Content. On error, the response returns the appropriate HTTP status code and includes a response body with an error code and human readable message.

Response body example

The response has no body.

Response body properties

The response has no body.

HTTP status codes

Status Description

204 No Content

ASR evaluation deleted successfully.

400 Bad Request

Server can't process the request due to a client error. For example, the payload is malformed, the annotation set is malformed or empty.

The response body contains a violations array with the errors. The following example shows the response body with a validation error.

{
    "message": "Payload validations failed",
    "violations": [{
        "message": "The \"name\" property at property path \"$.name\" is outside the allowed range.",
        "code": "INVALID_STRING_LENGTH"
    }]
}

401 Unauthorized

Authorization token is invalid, expired, or doesn't have access to the resource.

403 Forbidden

The requested operation isn't allowed.
The response body contains a violations array with the errors.

404 Not Found

The requested resource isn't found. For example, the skillId isn't found.

The following example shows the response body with the error and message.

{
     "message": "The skill cannot be found.",
     "code": "RESOURCE_NOT_FOUND"
}

409 Conflict

Request to run an evaluation for a given skill conflicts with an evaluation request that's currently in progress.

429 Too Many Requests

User has exceeded the permitted rate limit.

500 Internal Server Error

Server has encountered an error.

503 Service Unavailable

Server is down for maintenance, overloaded, or otherwise unavailable to handle the incoming request.

Get ASR evaluation results

Get the details results of the specified ASR evaluation. If you want the status of the evaluation only, use Get ASR evaluation status.

Request

To get the results of an evaluation, you make a GET request to the asrEvaluations resource.

Request path and header example

Copied to clipboard.

GET /v1/skills/{skillId}/asrEvaluations/{evaluationId}/results?status={status}&nextToken={nextToken}&maxResults={maxResults} HTTP/1.1
Host: api.amazonalexa.com
Content-Type: application/json
Authorization: Bearer {access token}
Accept: application/json

Request path and header parameters

Parameter Located in Description Type Required

skillId

Path

Identifies the skill.
Valid values: 1 – 255 characters.

String

Yes

evaluationId

Path

Identifies the ASR evaluation.

String

Yes

status

Query

Identifies the results status.
Valid values: PASSED, FAILED.

String

No

nextToken

Query

Token that you use to get the next set of results, after you receive a response with truncated results. Set this parameter to the value of nextToken from the truncated response you just received.

String

No

maxResults

Query

Maximum number of results to return.
Default value: 1000.

Number

No

access token

Header

LWA token.

String

Yes

Request body example

The request has no body.

Request body properties

The request has no body.

Response

A successful response returns HTTP 200 OK, along with the evaluation results. On error, the response returns the appropriate HTTP status code and includes a response body with an error code and human readable message.

Response body example

{
    "results": [{
            "status": "PASSED",
            "annotation": {
                "evaluationWeight": 1,
                "expectedTranscription": "hello world",
                "filePathInUpload": "hello world.wav",
                "uploadId": "upload.1",
                "audioAsset": {
                    "downloadUrl": "https://presign.s3.aws.amazon/123412341234-hello.wav",
                    "expiryTime": "2018-10-25T08:25:04.679Z"
                }
            },
            "output": {
                "transcription": "hello world"
            }
        },
        {
            "status": "FAILED",
            "annotation": {
                "evaluationWeight": 8,
                "expectedTranscription": "alexa what is the weather now",
                "filePathInUpload": "weather.wav",
                "uploadId": "upload.2",
                "audioAsset": {
                    "downloadUrl": "https://presign.s3.aws.amazon/123412341234-weather.wav",
                    "expiryTime": "2018-10-25T08:25:04.679Z"
                }
            },
            "error": {
                "message": "An unexpected error occurred",
                "code": "INTERNAL_SERVER_ERROR"
            }
        }
    ]
}

Response body properties

Property Description Type

results

Result of the ASR evaluation. Each item in the array represents a test case.
Maximum size: 1000 elements.

Array of objects

results[].status

Identifies the status of the test case.
Valid values: PASSED, FAILED.

String

results[].annotation

Information about the specific annotation used in this evaluation.

Object

results[].annotation.filePathInUpload

Path to the audio file in the uploaded zip file.
For example, for a zip file that contains a folder, named folder with an audio file, named audio.mp3, the path is folder/audio.mp3.

Included when expectedTranscription is missing. When filePathInUpload is present, uploadId is present also.

String

results[].annotation.uploadId

Identifies the uploaded audio content. The Catalog API returns the identifier each time you create an upload.
Included when expectedTranscription is missing. When uploadId is present, filePathInUpload is present also.

String

results[].annotation.evaluationWeight

Weight of the test case in an evaluation. Used to calculate metrics, such as overall error rate.
Valid values: 1 – 1000, from least to most significant test case.

Integer

results[].annotation.expectedTranscription

Expected transcription text for the input audio.
Included when uploadId and filePathInUpload are missing.
Valid values: 1 – 500 Unicode characters.

String

results[].annotation.audioAsset

Specifies how to download the audio file.

Object

results[].annotation.audioAsset.audioAssetDownloadUrl

Identifies the presigned URL at which to download the audio file.

String

results[].annotation.audioAsset.audioAssetDownloadUrlExpiryTime

Expiration date for the download URL.
Defined in ISO 8601 format, YYYY-MM-DDThh:mm:ssZ.

String

results[].output

(Optional) Output of the evaluation.
Included when the evaluation status is PASSED.

Object

results[].output.transcription

Actual transcription returned from ASR for the audio.

String

results[].error

(Optional) Identifies the reason for the failed evaluation.
Included when evaluation status is FAILED.

Object

results[].error.message

Describes why the evaluation wasn't executed successfully.

String

results[].error.code

Evaluation error code.

String

paginationContext

Indicates there are more matching results. If present, the response contains a subset of the results. When there are no more matching results, paginationContext isn't included in the response.

Object

paginationContext.nextToken

Identifies the next set of evaluation results to return. The token expires in 24 hours.
Use this value for the nextToken parameter in a subsequent Get ASR evaluation results request.

String

HTTP status codes

Status Description

200 OK

Response body contains a list of evaluation results.

400 Bad Request

Server can't process the request due to a client error. For example, the payload is malformed, the annotation set is malformed or empty.

The response body contains a violations array with the errors. The following example shows the response body with a validation error.

{
    "message": "Payload validations failed",
    "violations": [{
        "message": "The \"name\" property at property path \"$.name\" is outside the allowed range.",
        "code": "INVALID_STRING_LENGTH"
    }]
}

401 Unauthorized

Authorization token is invalid, expired, or doesn't have access to the resource.

403 Forbidden

The requested operation isn't allowed.
The response body contains a violations array with the errors.

404 Not Found

The requested resource isn't found. For example, the skillId isn't found.

The following example shows the response body with the error and message.

{
     "message": "The skill cannot be found.",
     "code": "RESOURCE_NOT_FOUND"
}

409 Conflict

Request to run an evaluation for a given skill conflicts with an evaluation request that's currently in progress.

429 Too Many Requests

User has exceeded the permitted rate limit.

500 Internal Server Error

Server has encountered an error.

503 Service Unavailable

Server is down for maintenance, overloaded, or otherwise unavailable to handle the incoming request.

Get ASR evaluation status

Get the status of the specified ASR evaluation. If you want the detailed results of the evaluation, use Get ASR evaluation results.

Request

To get the status of an evaluation, you make a GET request to the asrEvaluations resource.

Request path and header example

Copied to clipboard.

GET /v1/skills/{skillId}/asrEvaluations/{evaluationId}/status HTTP/1.1
Host: api.amazonalexa.com
Content-Type: application/json
Authorization: Bearer {access token}
Accept: application/json

Request path and header parameters

Parameter Located in Description Type Required

skillId

Path

Identifies the skill.
Valid values: 1 – 255 characters.

String

Yes

evaluationId

Path

Identifies the ASR evaluation.

String

Yes

access token

Header

LWA token.

String

Yes

Request body example

The request has no body.

Request body properties

The request has no body.

Response

A successful response returns HTTP 200 OK, along with the evaluation status. On error, the response returns the appropriate HTTP status code and includes a response body with an error code and human readable message.

Response body example

{
    "id": "evaluation.1",
    "status": "COMPLETED",
    "totalEvaluationCount": 2,
    "completedEvaluation": 2,
    "startTimestamp": "2018-10-25T08:25:04.679Z",
    "request": {
        "skill": {
            "stage": "development",
            "locale": "en-US"
        },
        "annotationSetId": "1234-1234-12341234"
    },
    "result": {
        "status": "PASSED",
        "metrics": {
            "overallErrorRate": 0.5
        }
    }
}

Response body properties

Property Description Type

status

Indicates the status of the evaluation. A failure status indicates that the evaluation didn't start.
Valid values: COMPLETED, FAILED, IN_PROGRESS.

String

request

Request payload that triggered this evaluation.

Object

request.skill

Information about the skill used in the evaluation.

Object

request.skill.stage

Indicates stage of the skill.
Valid values: development, live.

String

request.skill.locale

Country and language where the skill is offered.
Valid values: ar-SA, de-DE, en-AU, en-CA, en-GB, en-IN, en-US, es-ES, es-MX, es-US, fr-CA, fr-FR, hi-IN, it-IT, ja-JP, and pt-BR.

String

request.annotationSetId

Identifies the annotation set that's used in the evaluation.

String

totalEvaluationCount

Total number of evaluations requested to be run.

Integer

completedEvaluationCount

Number of completed evaluations.

Integer

startTimestamp

Start time of the evaluations.
Defined in ISO 8601 format, YYYY-MM-DDThh:mm:ssZ.

String

result

(Optional) Results of the evaluation.
Included when the evaluation status is COMPLETED.

Object

result.metrics

Metrics generated from the evaluation.

Object

result.metrics.overallErrorRate

Generated overall error rate of the evaluation.

Double

result.status

Evaluation result status.
Valid values: PASSED, FAILED.

String

error

(Optional) Indicates that the evaluation wasn't started.
Included when the evaluation status is FAILED.

Object

error.message

Describes why the evaluation wasn't started.

String

error.code

Evaluation error code.

String

HTTP status codes

Status Description

200 OK

Response body contains the evaluation status.

400 Bad Request

Server can't process the request due to a client error. For example, the payload is malformed, the annotation set is malformed or empty.

The response body contains a violations array with the errors. The following example shows the response body with a validation error.

{
    "message": "Payload validations failed",
    "violations": [{
        "message": "The \"name\" property at property path \"$.name\" is outside the allowed range.",
        "code": "INVALID_STRING_LENGTH"
    }]
}

401 Unauthorized

Authorization token is invalid, expired, or doesn't have access to the resource.

403 Forbidden

The requested operation isn't allowed.
The response body contains a violations array with the errors.

404 Not Found

The requested resource isn't found. For example, the skillId isn't found.

The following example shows the response body with the error and message.

{
     "message": "The skill cannot be found.",
     "code": "RESOURCE_NOT_FOUND"
}

409 Conflict

Request to run an evaluation for a given skill conflicts with an evaluation request that's currently in progress.

429 Too Many Requests

User has exceeded the permitted rate limit.

500 Internal Server Error

Server has encountered an error.

503 Service Unavailable

Server is down for maintenance, overloaded, or otherwise unavailable to handle the incoming request.

List ASR evaluations

Get all completed annotation evaluations for the specified skill. You can add optional filters to the request.

Request

To list ASR evaluations, you make a GET request to the asrEvaluations resource.

Request path and header example

Copied to clipboard.

GET /v1/skills/{skillId}/asrEvaluations?locale={locale}&annotationSetId={annotationSetId}&stage={stage}&nextToken={nextToken}&maxResults={maxResults} HTTP/1.1
Host: api.amazonalexa.com
Content-Type: application/json
Authorization: Bearer {access token}
Accept: application/json

Request path and header parameters

Parameter Located in Description Type Required

skillId

Path

Identifies the skill.
Valid values: 1 – 255 characters.

String

Yes

locale

Query

Filter on the country and language where the skill is offered. If omitted, all locales are included in the response.
Valid values: ar-SA, de-DE, en-AU, en-CA, en-GB, en-IN, en-US, es-ES, es-MX, es-US, fr-CA, fr-FR, hi-IN, it-IT, ja-JP, and pt-BR.

String

No

annotationSetId

Query

Filters results to the specified annotation set. If omitted, all annotation sets are included in the response.

String

No

stage

Query

Filter on the specified skill stage. If omitted, all stages are included in the response.
Valid values: development, live.

String

No

nextToken

Query

Token that you use to get the next set of results, after you receive a response with truncated results. Set this parameter to the value of nextToken from the truncated response you just received.

String

No

maxResults

Query

Maximum number of results to return.
Default value: 1000.

Number

No

access token

Header

LWA token.

String

Yes

Request body example

The request has no body.

Request body properties

The request has no body.

Response

A successful response returns HTTP 200 OK, along with a list of evaluations. On error, the response returns the appropriate HTTP status code and includes a response body with an error code and human readable message.

Response body example

{
    "evaluations": [{
            "id": "evaluation.1",
            "status": "COMPLETED",
            "startTimestamp": "2018-10-25T08:25:04.679Z",
            "totalEvaluationCount": 10,
            "completedEvaluationCount": 10,
            "request": {
                "skill": {
                    "stage": "development",
                    "locale": "en-US"
                },
                "annotationSetId": "annotation.set.1"
            },
            "result": {
                "status": "PASSED",
                "metrics": {
                    "overallErrorRate": 0.5
                }
            }
        },
        {
            "id": "evaluation.2",
            "status": "COMPLETED",
            "totalEvaluationCount": 10,
            "completedEvaluationCount": 10,
            "startTimestamp": "2018-10-24T08:25:04.679Z",
            "request": {
                "skill": {
                    "stage": "live",
                    "locale": "en-GB"
                },
                "annotationSetId": "annotation.set.1"
            },
            "result": {
                "status": "FAILED",
                "metrics": {
                    "overallErrorRate": 0.5
                }
            }
        },
        {
            "id": "evaluation.3",
            "status": "FAILED",
            "totalEvaluationCount": 10,
            "completedEvaluationCount": 10,
            "startTimestamp": "2018-10-24T08:25:04.679Z",
            "request": {
                "skill": {
                    "stage": "development",
                    "locale": "en-GB"
                },
                "annotationSetId": "annotation.set.1"
            },
            "error": {
                "message": "An unexpected error occurred",
                "code": "INTERNAL_SERVER_ERROR"
            }
        }
    ],
    "paginationContext": {
        "nextToken": "token.1"
    }
}

Response body properties

Property Description Type

evaluations

Array of evaluation runs based on the filter criteria.

Array of objects

evaluations[].id

Identifies the evaluation run.

String

evaluations[].status

Indicates the status of the evaluation. A failure status indicates that the evaluation didn't start.
Valid values: COMPLETED, FAILED, IN_PROGRESS.

String

evaluations[].request

Request payload that triggered this evaluation.

Object

evaluations[].request.skill

Information about the skill used in the evaluation.

Object

evaluations[].request.skill.stage

Indicates stage of the skill.
Valid values: development, live.

String

evaluations[].request.skill.locale

Country and language where the skill is offered.
Valid values: ar-SA, de-DE, en-AU, en-CA, en-GB, en-IN, en-US, es-ES, es-MX, es-US, fr-CA, fr-FR, hi-IN, it-IT, ja-JP, and pt-BR.

String

evaluations[].request.annotationSetId

Identifies the annotation set that's used in the evaluation.

String

evaluations[].totalEvaluationCount

Total number of evaluations requested to be run.

Integer

evaluations[].completedEvaluationCount

Number of completed evaluations.

Integer

evaluations[].startTimestamp

Start time of the evaluations.
Defined in ISO 8601 format, YYYY-MM-DDThh:mm:ssZ.

String

evaluations[].result

(Optional) Results of the evaluation.
Included when the evaluation status is COMPLETED.

Object

evaluations[].result.metrics

Metrics generated from the evaluation.

Object

evaluations[].result.metrics.overallErrorRate

Generated overall error rate of the evaluation.

Double

evaluations[].result.status

Evaluation result status.
Valid values: PASSED, FAILED.

String

evaluations[].error

(Optional) Indicates that the evaluation wasn't started.
Included when the evaluation status is FAILED.

Object

evaluations[].error.message

Describes why the evaluation wasn't started.

String

evaluations[].error.code

Evaluation error code.

String

paginationContext

Indicates there are more matching results. If present, the response contains a subset of the results. When there are no more matching results, paginationContext isn't included in the response.

Object

paginationContext.nextToken

Identifies the next set of evaluation results to return. The token expires in 24 hours.
Use this value for the nextToken parameter in a subsequent Get ASR evaluation results request.

String

HTTP status codes

Status Description

200 OK

Response body contains a list of evaluation runs.

400 Bad Request

Server can't process the request due to a client error. For example, the payload is malformed, the annotation set is malformed or empty.

The response body contains a violations array with the errors. The following example shows the response body with a validation error.

{
    "message": "Payload validations failed",
    "violations": [{
        "message": "The \"name\" property at property path \"$.name\" is outside the allowed range.",
        "code": "INVALID_STRING_LENGTH"
    }]
}

401 Unauthorized

Authorization token is invalid, expired, or doesn't have access to the resource.

403 Forbidden

The requested operation isn't allowed.
The response body contains a violations array with the errors.

404 Not Found

The requested resource isn't found. For example, the skillId isn't found.

The following example shows the response body with the error and message.

{
     "message": "The skill cannot be found.",
     "code": "RESOURCE_NOT_FOUND"
}

409 Conflict

Request to run an evaluation for a given skill conflicts with an evaluation request that's currently in progress.

429 Too Many Requests

User has exceeded the permitted rate limit.

500 Internal Server Error

Server has encountered an error.

503 Service Unavailable

Server is down for maintenance, overloaded, or otherwise unavailable to handle the incoming request.

Run ASR evaluation

Start an ASR evaluation against an existing annotation set. Concurrent evaluation requests for a given skill id aren't supported.

Request

To run an evaluation, you make a POST request to the asrEvaluations resource.

Request path and header example

Copied to clipboard.

POST /v1/skills/{skillId}/asrEvaluations HTTP/1.1
Host: api.amazonalexa.com
Content-Type: application/json
Authorization: Bearer {access token}
Accept: application/json

Request path and header parameters

Parameter Located in Description Type Required

skillId

Path

Identifies the skill.
Valid values: 1 – 255 characters.

String

Yes

access token

Header

LWA token.

String

Yes

Request body example

Copied to clipboard.

{
    "skill": {
        "stage": "live",
        "locale": "en-US"
    },
    "annotationSetId": "annotation.set.1"
}

Request body properties

Property Description Type Required

skill

Information about the skill used in the evaluation.

Object

Yes

skill.stage

Indicates stage of the skill.
Valid values: development, live.

String

Yes

skill.locale

Country and language where the skill is offered.
Valid values: ar-SA, de-DE, en-AU, en-CA, en-GB, en-IN, en-US, es-ES, es-MX, es-US, fr-CA, fr-FR, hi-IN, it-IT, ja-JP, and pt-BR.

String

Yes

annotationSetId

Identifies the annotation set to be used in the evaluation.

String

Yes

Response

A successful response returns HTTP 200 OK with a Location parameter in the request header. The response body contains the evaluation ID. Use the ID to get the status and results of the evaluation.

On error, the response returns the appropriate HTTP status code and includes a response body with an error code and human readable message.

Response body example

{
    "id": "evaluation.1"
}

Response body properties

Property Description Type

id

Identifies the evaluation run.

String

HTTP status codes

Status Description

200 OK

Empty annotation set created successfully. The response header contains a Location parameter that includes the skillId and annotationSetId. The annotationSetId is the same as the id in the response body. For example:

HTTP 200 OK
Content-Type: application/json
Location: v1/skills/{skillId}/asrEvaluations/{evaluationId}/status

400 Bad Request

Server can't process the request due to a client error. For example, the payload is malformed, the annotation set is malformed or empty.

The response body contains a violations array with the errors. The following example shows the response body with a validation error.

{
    "message": "Payload validations failed",
    "violations": [{
        "message": "The \"name\" property at property path \"$.name\" is outside the allowed range.",
        "code": "INVALID_STRING_LENGTH"
    }]
}

401 Unauthorized

Authorization token is invalid, expired, or doesn't have access to the resource.

403 Forbidden

The requested operation isn't allowed.
The response body contains a violations array with the errors.

404 Not Found

The requested resource isn't found. For example, the skillId isn't found.

The following example shows the response body with the error and message.

{
     "message": "The skill cannot be found.",
     "code": "RESOURCE_NOT_FOUND"
}

409 Conflict

Request to run an evaluation for a given skill conflicts with an evaluation request that's currently in progress.

429 Too Many Requests

User has exceeded the permitted rate limit.

500 Internal Server Error

Server has encountered an error.

503 Service Unavailable

Server is down for maintenance, overloaded, or otherwise unavailable to handle the incoming request.