Natural Language Understanding Evaluation REST API Reference

Use the Natural Language Understanding (NLU) Evaluation REST API to test the accuracy of the interaction model that you defined with your skill. NLU evaluations assess the accuracy of your skill against an annotation set. To create an annotation set, see NLU Annotation Set REST API Reference. After your annotation set passes the evaluation, rerun the evaluation when you make changes to your interaction model to make sure that your changes don't degrade the accuracy.

If you want to use the NLU Evaluation tool in the Alexa developer console, see Batch Test Your NLU Model.

API endpoint

The endpoint of the NLU Evaluation API is https://api.amazonalexa.com.

Authentication

Each API request must have an authorization header whose value is a bearer token retrieved from Login with Amazon (LWA). For details, see Get an Access Token for SMAPI.

Operations

The NLU Evaluation API includes the following operations.

Operation HTTP method and URI

Get evaluation results

GET /v1/skills/{skillId}/nluEvaluations/{evaluationId}?locale={locale}&sort.field={sortField}&testCaseStatus={testCaseStatus}&actualIntentName={actualIntentName}&expectedIntentName={expectedIntentName}&nextToken={nextToken}&maxResults={maxResults}

Get evaluation status

GET /v1/skills/{skillId}/nluEvaluations/{evaluationId}/status

List evaluations

GET /v1/skills/{skillId}/nluEvaluations?locale={locale}&annotationId={annotationId}&stage={stage}&nextToken={nextToken}&maxResults={maxResults}

Run NLU evaluation

POST /v1/skills/{skillId}/nluEvaluations

Get evaluation results

Get the details results of the specified NLU evaluation. If you want the status of the evaluation only, use Get NLU evaluation status.

Request

To get the results of an evaluation, you make a GET request to the nluEvaluations resource.

Request path and header example

Copied to clipboard.

GET /v1/skills/{skillId}/nluEvaluations/{evaluationId}?locale={locale}&sort.field={sortField}&testCaseStatus={testCaseStatus}&actualIntentName={actualIntentName}&expectedIntentName={expectedIntentName}&nextToken={nextToken}&maxResults={maxResults}
Host: api.amazonalexa.com
Content-Type: application/json
Authorization: Bearer {access token}
Accept: application/json

Request path and header parameters

Parameter Located in Description Type Required

skillId

Path

Identifies the skill.
Valid values: 1 – 255 characters.

String

Yes

evaluationId

Path

Identifies the NLU evaluation.

String

Yes

sortField

Query

Sort the result on the specified field in alphabetical order.
Valid values: STATUS, ACTUAL_INTENT, EXPECTED_INTENT.

String

No

testCaseStatus

Query

Filter on the test case status.
Valid values: PASSED, FAILED.

String

No

actualIntentName

Query

Filter results that resolve to the specified intent name.

String

No

expectedIntentName

Query

Filter results on the expected intent name.

String

No

nextToken

Query

Token that you use to get the next set of results, after you receive a response with truncated results. Set this parameter to the value of nextToken from the truncated response you just received.

String

No

maxResults

Query

Maximum number of results to return.
Default: 1000.

Number

No

access token

Header

LWA token.

String

Yes

Request body example

The request has no body.

Request body properties

The request has no body.

Response

A successful response returns HTTP 200 OK, along with the evaluation results. On error, the response returns the appropriate HTTP status code and includes a response body with an error code and human readable message.

Response body example

{

    "totalFailed": 0,
    "testCases": [{
        "status": "PASSED",
        "inputs": {
            "utterance": "i want to go to chicago on monday",
            "referenceTimestamp": "2019-08-12T18:46:31.674Z"
        },
        "actual": {
            "domain": "string",
            "intent": {
                "name": "PlanMyTripIntent",
                "confirmationStatus": "NONE",
                "slots": {
                    "toCity": {
                        "name": "toCitySlot",
                        "value": "chicago",
                        "confirmationStatus": "NONE",
                        "resolutions": {
                            "resolutionsPerAuthority": [{
                                "resolution1": {
                                    "authority": "string",
                                    "status": {
                                        "code": "ER_SUCCESS_MATCH"
                                    },
                                    "values": [{
                                        "name": "string",
                                        "id": "string"
                                    }]
                                }
                            }]
                        }
                    }
                }
            }
        },
        "expected": [{
            "domain": "string",
            "intent": {
                "name": "PlanMyTripIntent",
                "slots": {
                    "toCity": {
                        "value": "chicago"
                    }
                }
            }
        }]
    }],
    "paginationContext": {
        "nextToken": "someToken.1",
        "totalCount": "1"
    },
    "_links": {
        "self": {
            "href": "/v1/skills/amzn1.ask.skill.1/nluEvaluations"
        },
        "next": {
            "href": "/v1/skills/amzn1.ask.skill.1/nluEvaluations?nextToken=token.1&maxResults=100"
        }
    }
}

Response body properties

Property Description Type

totalFailed

Number of test cases that failed. A test fails when the utterance doesn't resolve to the expected intent and expected slots.

Integer

testCases

Array of test cases runs based on the filter criteria.

Array of objects

testCases[].status

Indicates the status of the test case.
Valid values: PASSED, FAILED.

String

testCases[].inputs

Inputs to the test case.

Object

testCases[].inputs.utterance

Utterance to test.

String

testCases[].inputs.referenceTimestamp

Date and time to use as the basis for relative date and time values. Include when the utterance tests. Use this when the utterance tests AMAZON.DATE or AMAZON.TIME slots with words that represent relative dates and times, such as "today," "tomorrow," and "now."
Defined in ISO 8601 format, YYYY-MM-DDThh:mm:ssZ.

String

actual

Actual results of the test case.

Object

actual.domain

Resolved domain name.

String

actual.intent

Resolved intent.

Object

actual.intent.name

Name of the intent.

String

actual.intent.confirmationStatus

Indicates whether the user confirmed or denied the intent.
Valid values: NONE, CONFIRMED, DENIED.

String

actual.intent.slots

List of expected slots for the utterance.

Array of objects

actual.intent.slots.<slotname>

Name of the slot. There might be one or more slots.

Object

actual.intent.slots.<slotname>.slotValue.name

Slot name.

String

actual.intent.slots.<slotname>.slotValue.value

Slot value.

String

actual.intent.slots.<slotname>.confirmationStatus

Indicates whether the user confirmed or denied the slot.
Valid values: NONE, CONFIRMED, DENIED.

String

actual.intent.slots.<slotname>.resolutions

Represents the results of resolving the words captured from the utterance.

Object

actual.intent.slots.<slotname>.resolutions.resolutionsPerAuthority

Identifies the authority for the slot values.

Object

actual.intent.slots.<slotname>.resolutions.resolutionsPerAuthority.authority

Name of the authority for slot values. For custom slot types, the label incorporates the skill ID and slot type name.

String

actual.intent.slots.<slotname>.resolutions.resolutionsPerAuthority.status

Indicates whether the spoken value matched a value or synonym defined in the slot type.

Object

actual.intent.slots.<slotname>.resolutions.resolutionsPerAuthority.status.code

Status of the slot resolution.
Valid values: ER_SUCCESS_MATCH, ER_SUCCESS_NO_MATCH, ER_ERROR_TIMEOUT, ER_ERROR_EXCEPTION.

Object

actual.intent.slots.<slotname>.resolutions.resolutionsPerAuthority.values

List of resolved values for the slot.

Array of objects

actual.intent.slots.<slotname>.resolutions.resolutionsPerAuthority.values[].name

Resolved slot value.

String

actual.intent.slots.<slotname>.resolutions.resolutionsPerAuthority.values[].id

Unique ID for the resolved slot value, based on the IDs in the slot type definition.

String

expected

Expected results of the test case.

Object

expected.domain

Expected domain name.

String

expected.intent.name

Intent that the utterance should trigger.

String

expected.intent.slots

List of expected slots for the utterance.

Array of objects

expected.intent.slots.<slotname>

Name of the slot. There might be one or more slots.

Object

actual.intent.slots.<slotname>.slotValue.value

Slot value.

String

paginationContext

Indicates there are more matching results. If present, the response contains a subset of the results. When there are no more matching results, paginationContext isn't included in the response.

Object

paginationContext.nextToken

Identifies the next set of evaluation results to return. The token expires in 24 hours.
Use this value for the nextToken parameter in a subsequent Get NLU evaluation results request.

String

_links

Links for result navigation.
Defined in JSON Hypertext Application Language specification.

_links object

HTTP status codes

Status Description

200 OK

Response body contains a list of evaluation results.

400 Bad Request

Server can't process the request due to a client error. For example, the payload is malformed, the annotation set is malformed or empty.

The response body contains a violations array with the errors. The following example shows the response body with a validation error.

{
    "message": "Payload validations failed",
    "violations": [{
        "message": "The \"name\" property at property path \"$.name\" is outside the allowed range.",
        "code": "INVALID_STRING_LENGTH"
    }]
}

401 Unauthorized

Authorization token is invalid, expired, or doesn't have access to the resource.

403 Forbidden

The requested operation isn't allowed.
The response body contains a violations array with the errors.

404 Not Found

The requested resource isn't found. For example, the skillId isn't found.

The following example shows the response body with the error and message.

{
     "message": "The skill cannot be found.",
     "code": "RESOURCE_NOT_FOUND"
}

409 Conflict

Request to run an evaluation for a given skill conflicts with an evaluation request that's currently in progress.

429 Too Many Requests

User has exceeded the permitted rate limit.

500 Internal Server Error

Server has encountered an error.

503 Service Unavailable

Server is down for maintenance, overloaded, or otherwise unavailable to handle the incoming request.

Get evaluation status

Get the status of the specified NLU evaluation. If you want the detailed results of the evaluation, use Get evaluation results.

Request

To get the status of an evaluation, you make a GET request to the nluEvaluations resource.

Request path and header example

Copied to clipboard.

GET /v1/skills/{skillId}/nluEvaluations/{evaluationId}/status
Host: api.amazonalexa.com
Content-Type: application/json
Authorization: Bearer {access token}
Accept: application/json

Request path and header parameters

Parameter Located in Description Type Required

skillId

Path

Identifies the skill.
Valid values: 1 – 255 characters.

String

Yes

evaluationId

Path

Identifies the NLU evaluation.

String

Yes

access token

Header

LWA token.

String

Yes

Request body example

The request has no body.

Request body properties

The request has no body.

Response

A successful response returns HTTP 200 OK, along with the evaluation status. On error, the response returns the appropriate HTTP status code and includes a response body with an error code and human readable message.

Response body example

{
    "startTimestamp": "2019-08-12T18:41:09.684Z",
    "endTimestamp": "2019-08-12T18:41:09.684Z",
    "status": "string",
    "errorMessage": "string",
    "inputs": {
        "locale": "en-US",
        "stage": "development",
        "source": {
            "annotationId": "annotation.set.1"
        }
    },
    "_links": {
        "results": {
            "href": "/v1/skills/amzn1.ask.skill.1/nluEvaluations"
        }
    }
}

Response body properties

Property Description Type

status

Indicates the status of the evaluation. A failure status indicates that the evaluation didn't start.
Valid values: PASSED, FAILED, IN_PROGRESS, ERROR.

String

startTimestamp

Start time of the evaluations.
Defined in ISO 8601 format, YYYY-MM-DDThh:mm:ssZ.

String

endTimestamp

Time the evaluation completed.
Defined in ISO 8601 format, YYYY-MM-DDThh:mm:ssZ.

String

errorMessage

(Optional) Human-readable error message included when evaluation fails.

String

inputs

Request payload that triggered this evaluation.

Object

inputs.locale

Country and language where you offer your skill.

String

inputs.stage

Indicates stage of the skill.
Valid values: development, live.

String

inputs.source

Identifies the annotation set to use in the evaluation.

Object

inputs.source.annotationId

Identifies the annotation set.

String

_links

Links for result navigation.
Defined in JSON Hypertext Application Language specification.

_links object

HTTP status codes

Status Description

200 OK

Response body contains the evaluation status.

400 Bad Request

Server can't process the request due to a client error. For example, the payload is malformed, the annotation set is malformed or empty.

The response body contains a violations array with the errors. The following example shows the response body with a validation error.

{
    "message": "Payload validations failed",
    "violations": [{
        "message": "The \"name\" property at property path \"$.name\" is outside the allowed range.",
        "code": "INVALID_STRING_LENGTH"
    }]
}

401 Unauthorized

Authorization token is invalid, expired, or doesn't have access to the resource.

403 Forbidden

The requested operation isn't allowed.
The response body contains a violations array with the errors.

404 Not Found

The requested resource isn't found. For example, the skillId isn't found.

The following example shows the response body with the error and message.

{
     "message": "The skill cannot be found.",
     "code": "RESOURCE_NOT_FOUND"
}

409 Conflict

Request to run an evaluation for a given skill conflicts with an evaluation request that's currently in progress.

429 Too Many Requests

User has exceeded the permitted rate limit.

500 Internal Server Error

Server has encountered an error.

503 Service Unavailable

Server is down for maintenance, overloaded, or otherwise unavailable to handle the incoming request.

List evaluations

Get all completed NLU evaluations for the specified skill. You can add optional filters to the request.

Request

To list NLU evaluations, you make a GET request to the nluEvaluations resource.

Request path and header example

Copied to clipboard.

GET /v1/skills/{skillId}/nluEvaluations?locale={locale}&annotationId={annotationId}&stage={stage}&nextToken={nextToken}&maxResults={maxResults}
Host: api.amazonalexa.com
Content-Type: application/json
Authorization: Bearer {access token}
Accept: application/json

Request path and header parameters

Parameter Located in Description Type Required

skillId

Path

Identifies the skill.
Valid values: 1 – 255 characters.

String

Yes

locale

Query

Filter on the country and language where you offer the skill. If omitted, the response includes all locales.

String

No

annotationId

Query

Filters results to the specified annotation set. If omitted, the response includes all annotation sets.

String

No

stage

Query

Filter on the specified skill stage. If omitted, the response includes all stages.
Valid values: development, live.

String

No

nextToken

Query

Token that you use to get the next set of results, after you receive a response with truncated results. Set this parameter to the value of nextToken from the truncated response you received previously.

String

No

maxResults

Query

Maximum number of results to return.
Valid values: 1 – 100. Default: 10.

Number

No

access token

Header

LWA token.

String

Yes

Request body example

The request has no body.

Request body properties

The request has no body.

Response

A successful response returns HTTP 200 OK, along with a list of evaluations. On error, the response returns the appropriate HTTP status code and includes a response body with an error code and human readable message.

Response body example

{
    "evaluations": [{
        "startTimestamp": "2019-08-12T18:41:09.684Z",
        "endTimestamp": "2019-08-12T18:41:09.684Z",
        "status": "string",
        "errorMessage": "string",
        "inputs": {
            "locale": "en-US",
            "stage": "development",
            "source": {
                "annotationId": "annotation.set.1"
            }
        },
        "id": "evaluation.1"
    }],
    "paginationContext": {
        "nextToken": "someToken.1"
    },
    "_links": {
        "self": {
            "href": "/v1/skills/amzn1.ask.skill.1/nluEvaluations"
        },
        "next": {
            "href": "/v1/skills/amzn1.ask.skill.1/nluEvaluations?nextToken=token.1&maxResults=100"
        }
    }
}

Response body properties

Property Description Type

evaluations

Array of evaluation runs based on the filter criteria.

Array of objects

evaluations[].id

Identifies the evaluation run.

String

evaluations[].status

Indicates the status of the evaluation. A failure status indicates that the evaluation didn't start.
Valid values: PASSED, FAILED, IN_PROGRESS, ERROR.

String

evaluations[].startTimestamp

Start time of the evaluations.
Defined in ISO 8601 format, YYYY-MM-DDThh:mm:ssZ.

String

evaluations[].endTimestamp

Time the evaluation completed.
Defined in ISO 8601 format, YYYY-MM-DDThh:mm:ssZ.

String

evaluations[].errorMessage

(Optional) Human-readable error message included when evaluation fails.

String

evaluations[].inputs

Request payload that triggered this evaluation.

Object

evaluations[].inputs.locale

Country and language where the skill is offered.

String

evaluations[].inputs.stage

Indicates stage of the skill.
Valid values: development, live.

String

evaluations[].inputs.source

Identifies the annotation set to be used in the evaluation.

Object

evaluations[].inputs.source.annotationId

Identifies the annotation set.

String

paginationContext

Indicates there are more matching results. If present, the response contains a subset of the results. When there are no more matching results, paginationContext isn't included in the response.

Object

paginationContext.nextToken

Identifies the next set of evaluation results to return. The token expires in 24 hours.
Use this value for the nextToken parameter in a subsequent Get NLU evaluation results request.

String

paginationContext.totalCount

Number of test cases that matched the request parameters.

String

_links

Links for result navigation.
Defined in JSON Hypertext Application Language specification.

_links object

HTTP status codes

Status Description

200 OK

Response body contains a list of evaluation runs.

400 Bad Request

Server can't process the request due to a client error. For example, the payload is malformed, the annotation set is malformed or empty.

The response body contains a violations array with the errors. The following example shows the response body with a validation error.

{
    "message": "Payload validations failed",
    "violations": [{
        "message": "The \"name\" property at property path \"$.name\" is outside the allowed range.",
        "code": "INVALID_STRING_LENGTH"
    }]
}

401 Unauthorized

Authorization token is invalid, expired, or doesn't have access to the resource.

403 Forbidden

The requested operation isn't allowed.
The response body contains a violations array with the errors.

404 Not Found

The requested resource isn't found. For example, the skillId isn't found.

The following example shows the response body with the error and message.

{
     "message": "The skill cannot be found.",
     "code": "RESOURCE_NOT_FOUND"
}

409 Conflict

Request to run an evaluation for a given skill conflicts with an evaluation request that's currently in progress.

429 Too Many Requests

User has exceeded the permitted rate limit.

500 Internal Server Error

Server has encountered an error.

503 Service Unavailable

Server is down for maintenance, overloaded, or otherwise unavailable to handle the incoming request.

Run NLU evaluation

Start an NLU evaluation against an existing annotation set. Concurrent evaluation requests for a given skill id aren't supported.

Request

To run an evaluation, you make a POST request to the nluEvaluations resource.

Request path and header example

Copied to clipboard.

POST /v1/skills/{skillId}/nluEvaluations
Host: api.amazonalexa.com
Content-Type: application/json
Authorization: Bearer {access token}
Accept: application/json

Request path and header parameters

Parameter Located in Description Type Required

skillId

Path

Identifies the skill.
Valid values: 1 – 255 characters.

String

Yes

access token

Header

LWA token.

String

Yes

Request body example

Copied to clipboard.

{
    "stage": "live",
    "locale": "en-US",
    "source": {
        "annotationId": "annotation.set.1"
    }
}

Request body properties

Property Description Type Required

stage

Indicates stage of the skill.
Valid values: development, live.

String

Yes

locale

Country and language where the skill is offered.
For details, seeLocale.

String

Yes

source

Identifies the annotation set to be used in the evaluation.

Object

Yes

source.annotationId

Identifies the annotation set.
Use Create annotation set or List annotation sets to get the ID.

String

Yes

Response

A successful response returns HTTP 200 OK with a Location parameter in the response header. The response body contains the evaluation ID. Use the ID to get the status and results of the evaluation.

On error, the response returns the appropriate HTTP status code and includes a response body with an error code and human readable message.

Response body example

{
    "id": "evaluation.1"
}

Response body properties

Property Description Type

id

Identifies the evaluation run.

String

HTTP status codes

Status Description

200 OK

Empty annotation set created successfully. The response header contains a Location parameter that includes the skillId and annotationId. The annotationId is the same as the id in the response body. For example:

HTTP 200 OK
Content-Type: application/json
Location: v1/skills/{skillId}/nluEvaluations/{evaluationId}/status

400 Bad Request

Server can't process the request due to a client error. For example, the payload is malformed, the annotation set is malformed or empty.

The response body contains a violations array with the errors. The following example shows the response body with a validation error.

{
    "message": "Payload validations failed",
    "violations": [{
        "message": "The \"name\" property at property path \"$.name\" is outside the allowed range.",
        "code": "INVALID_STRING_LENGTH"
    }]
}

401 Unauthorized

Authorization token is invalid, expired, or doesn't have access to the resource.

403 Forbidden

The requested operation isn't allowed.
The response body contains a violations array with the errors.

404 Not Found

The requested resource isn't found. For example, the skillId isn't found.

The following example shows the response body with the error and message.

{
     "message": "The skill cannot be found.",
     "code": "RESOURCE_NOT_FOUND"
}

409 Conflict

Request to run an evaluation for a given skill conflicts with an evaluation request that's currently in progress.

429 Too Many Requests

User has exceeded the permitted rate limit.

500 Internal Server Error

Server has encountered an error.

503 Service Unavailable

Server is down for maintenance, overloaded, or otherwise unavailable to handle the incoming request.

Object definitions

The NLU Evaluation API defines the following objects.

Locale values

The following table shows valid values for the locale property.

Locale code Language

ar-SA

Arabic (SA)

de-DE

German (DE)

en-AU

English (AU)

en-CA

English (CA)

en-GB

English (UK)

en-IN

English (IN)

en-US

English (US)

es-ES

Spanish (ES)

es-MX

Spanish (MX)

es-US

Spanish (US)

fr-CA

French (CA)

fr-FR

French (FR)

hi-IN

Hindi (IN)

it-IT

Italian (IT)

ja-JP

Japanese (JP)

pt-BR

Portuguese (BR)