NLU Evaluation Tool API
Use the NLU evaluation tool API to programmatically batch test the natural language understanding (NLU) accuracy of your Alexa skill.
To use the API, you create a data file that contains utterances mapped to the intents and slots you expect to be sent to your skill. This is called an annotation set. Then you start an NLU evaluation using a particular annotation set to determine how well your skill's model performs against your expectations. The tool can help you measure the accuracy of your skill, and run regression testing to help make sure that changes to your interaction model don't degrade the customer experience.
You can use the NLU evaluation tool with skill models for all locales. You can use the tool through the Alexa Skills Kit Command Line Interface (ASK CLI) or the Skill Management API (SMAPI). You can also use the tool within the developer console.
Annotation sets
To use the NLU evaluation tool API, you must provide a data file that contains utterances mapped to their expected intent and slots. This is called an annotation set. Each utterance with its expected intent and slots is called an annotation. You can format this data file as JSON or CSV.
JSON format
To provide an annotation set in JSON format, use the following structure:
{
"data": [
{
"inputs": {
"utterance": "string",
"referenceTimestamp": "string (date/time in UTC format)"
},
"expected": [
{
"intent": {
"name": "string",
"slots": {
"SlotName1": {
"slotValue": {
"value": "string",
"type": "Simple"
}
},
"SlotName2": {
"slotValue": {
"type": "List",
"values": [
{
"value": "string",
"type": "Simple"
},
{
"value": "string",
"type": "Simple"
}
]
}
}
}
}
}
]
}
]
}
Field | Description |
---|---|
inputs.utterance |
Required. The utterance to test.
|
inputs.referenceTimestamp |
Optional A time and date in UTC format to use as the basis for relative date and time values (for example, 2019-08-21T00:00:00.000Z). Use this when the utterance tests Specify the date and time in UTC format: For more information about testing relative date and time values, see Create an annotation with relative dates or times. |
expected.intent.name |
Required. The name of the expected intent for the utterance. |
expected.intent.slots.SlotName |
Optional. The name of the expected slot for the utterance. You can provide more than one expected slot for an utterance. |
expected.intent.slots.SlotName.slotValue |
An object that represents the expected value for the slot. The slotValue object has either a value or values property, but never both.
For a single value slot, slotValue has a type of Simple and a value property with the string value of the slot. For a multiple-value slot, slotValue has a type of List and a values property containing an array of slot values. |
expected.intent.slots.SlotName.slotValue.type |
Specifies the type of slot value. Set to Simple for a single value slot. Set to List for a multiple-value slot. |
expected.intent.slots.SlotName.slotValue.value |
For a single-value slot, contains the expected value of the slot as a string. Omit for a multiple-value slot. |
expected.intent.slots.SlotName.slotValue.values |
For a multiple-value slot, contains an array of objects representing the expected values for the slot. Each object in the array has a value property for the string value and a type property. Specify the values in the same order as they are written in the utterance . Omit the values array for a single-value slot. |
expected.intent.slots.SlotName.slotValue.values[].value |
Contains the expected slot value as a string. |
expected.intent.slots.SlotName.slotValue.values[].type |
Specifies the type of the slot value. Set to Simple . |
JSON annotation set example
The following example shows the JSON format for an annotation set. In this example, the activity
slot can accept multiple values.
{
"data": [
{
"inputs": {
"utterance": "plan a trip"
},
"expected": [
{
"intent": {
"name": "PlanMyTripIntent"
}
}
]
},
{
"inputs": {
"utterance": "i want to go to chicago on monday",
"referenceTimestamp": "2020-12-11T12:00:00.000Z"
},
"expected": [
{
"intent": {
"name": "PlanMyTripIntent",
"slots": {
"toCity": {
"slotValue": {
"value": "Chicago",
"type": "Simple"
}
},
"travelDate": {
"slotValue": {
"value": "2020-12-14",
"type": "Simple"
}
}
}
}
}
]
},
{
"inputs": {
"utterance": "plan a trip from seattle to boston"
},
"expected": [
{
"intent": {
"name": "PlanMyTripIntent",
"slots": {
"toCity": {
"slotValue": {
"value": "Boston",
"type": "Simple"
}
},
"fromCity": {
"slotValue": {
"value": "Seattle",
"type": "Simple"
}
}
}
}
}
]
},
{
"inputs": {
"utterance": "plan a trip from chicago to denver on thursday",
"referenceTimestamp": "2020-12-11T12:00:00.000Z"
},
"expected": [
{
"intent": {
"name": "PlanMyTripIntent",
"slots": {
"toCity": {
"slotValue": {
"value": "Denver",
"type": "Simple"
}
},
"fromCity": {
"slotValue": {
"value": "Chicago",
"type": "Simple"
}
},
"travelDate": {
"slotValue": {
"value": "2020-12-17",
"type": "Simple"
}
}
}
}
}
]
},
{
"inputs": {
"utterance": "plan a trip to missoula to go camping, hiking, and fishing"
},
"expected": [
{
"intent": {
"name": "PlanMyTripIntent",
"slots": {
"toCity": {
"slotValue": {
"value": "Missoula",
"type": "Simple"
}
},
"activity": {
"slotValue": {
"type": "List",
"values": [
{
"value": "camping",
"type": "Simple"
},
{
"value": "hiking",
"type": "Simple"
},
{
"value": "fishing",
"type": "Simple"
}
]
}
}
}
}
}
]
},
{
"inputs": {
"utterance": "plan a trip to portland to go kayaking"
},
"expected": [
{
"intent": {
"name": "PlanMyTripIntent",
"slots": {
"toCity": {
"slotValue": {
"value": "Portland",
"type": "Simple"
}
},
"activity": {
"slotValue": {
"value": "kayaking",
"type": "Simple"
}
}
}
}
}
]
}
]
}
CSV format
To provide an annotation set in CSV (UTF-8) format, use the following structure:
utterance, referenceTimestamp, intent, slot[SlotName1], slot[SlotName2]
"Utterance to test", 2019-08-21T00:00:00.000Z, ExpectedIntent, "slot value 1", "slot value 2"
For multiple-value slots, duplicate the slot[SlotName]
syntax and include an index number in brackets.
In the following example, "SlotName1" is a multiple value slot. For a given utterance, the first value for this slot is under the column slot[SlotName1][0]
and the second value is under slot[SlotName1][1]
. If the utterance includes only a single value for this slot, the expected value goes under the column with no index number: slot[SlotName1]
.
utterance, referenceTimestamp, intent, slot[SlotName1][0],slot[SlotName1][1],slot[SlotName1],slot[SlotName2]
For descriptions of these fields, see the JSON format.
The following example shows the CSV format for an annotation set. In this example, the activity
slot can accept multiple values. The examples shows an utterance with three values for the slot, and an utterance with one value.
utterance, referenceTimestamp, intent, slot[toCity],slot[fromCity],slot[travelDate],slot[activity][0],slot[activity][1],slot[activity][2],slot[activity]
plan a trip,,PlanMyTripIntent,,,
i want to go to chicago on monday,2019-08-29T00:00:00.000Z,PlanMyTripIntent,chicago,,2019-08-30
plan a trip from seattle to boston,,PlanMyTripIntent,boston,seattle,
plan a trip from chicago to denver on Monday,2019-08-29T00:00:00.000Z,PlanMyTripIntent,denver,chicago,2019-09-02
plan a trip to missoula to go camping hiking and fishing,,PlanMyTripIntent,Missoula,,,camping,hiking,fishing
plan a trip to portland to go kayaking,,PlanMyTripIntent,Portland,,,,,,kayaking
The CSV file must be encoded with UTF-8.
Use SMAPI
After you create an annotation set, you can use SMAPI to create and update annotation sets, then execute NLU evaluations.
SMAPI endpoint
The API's endpoint is https://api.amazonalexa.com
. Every API request must have an Authorization
header whose value is the access token retrieved from Login with Amazon.
SMAPI reference
You can use SMAPI to manage annotation sets, and to run NLU evaluations. The following list contains all of the available operations; click each one to see more information.
Create and manage annotation sets
- Create a new annotation set for a skill
- List a skill's annotation sets
- Upload or update an annotation set
- Get an annotation set
- Update an annotation set's properties
- Get an annotation set's properties
- Delete an annotation set
Run NLU evaluations
- Start an evaluation
- List your evaluations
- Get the status of an evaluation
- Get the results of an evaluation
Create a new annotation set
Creates a new annotation set for the specified skill. This operation returns the annotation ID that you use to upload or update an annotation set.
Request
POST /v1/skills/{skillId}/nluAnnotationSets
Request body
{
"locale": "string",
"name": "string"
}
Request parameters
Field | Description | Type |
---|---|---|
skillId |
Required. Part of the URI path. The identifier of the skill for the annotation set. | String |
locale |
Required. Part of the request body. The intended interaction model locale for the annotation set. For example: en-US , en-GB , de-DE , and so on. |
String |
name |
Required. Part of the request body. A name for the annotation set. Must consist of alphanumeric characters. | String |
Response
HTTP 201 CREATED
Response headers
Header | Description |
---|---|
Location |
The location of the annotation set. Use this URL to get an annotation set. |
Response body
{
"id": "string"
}
Response fields
Field | Description | Type |
---|---|---|
id |
The identifier for the annotation set. | String |
Error responses
HTTP 400 Bad Request
HTTP 401 Unauthorized
HTTP 403 Forbidden
HTTP 404 Not Found
HTTP 429 Too Many Requests
HTTP 500 Internal Server Error
HTTP 503 Service Unavailable
List a skill's annotation sets
Gets a list of annotation sets for the specified skill.
Request
GET /v1/skills/{skillId}/nluAnnotationSets
Request parameters
Field | Description | Type |
---|---|---|
skillId |
Required. Part of the URI path. The ID of the skill for which to list annotation sets. | String |
locale |
Optional. Part of the query string. A locale for filtering the results returned in the response. For example: en-US , en-GB , de-DE , and so on. |
String |
nextToken |
Optional. Part of the query string. Use this parameter to get more annotation sets, after you receive a response with incomplete results. Set it to the value of nextToken from the incomplete response you just received. |
String |
maxResults |
Optional. Part of the query string. The maximum number of items to return in the response. When you don't include this parameter, the default maximum is 10. When you include this parameter, the response might contain fewer items than the value you specify, but will never contain more. You cannot specify a value higher than 100. | Integer |
Response
HTTP 200 OK
Response body
{
"annotationSets": [
{
"locale": "string",
"name": "string",
"numberOfEntries": 0,
"updatedTimestamp": "2019-08-12T18:46:31.674Z",
"annotationId": "string"
}
],
"paginationContext": {
"nextToken": "string"
},
"_links": {
"self": {
"href": "string"
},
"next": {
"href": "string"
}
}
}
Response fields
Field | Description | Type |
---|---|---|
annotationSets |
A list of annotation set objects, including each annotation set's locale, name, and identifier. | Object |
nextToken |
When this fields exists, it means the response contains incomplete results. This fields contains the value to use for the nextToken parameter in a subsequent request. |
String |
_links |
Links for API navigation. The structure of this field is defined in the JSON Hypertext Application Language specification. | _links object (see specification) |
Error response
HTTP 400 Bad Request
HTTP 401 Unauthorized
HTTP 403 Forbidden
HTTP 404 Not Found
HTTP 429 Too Many Requests
HTTP 500 Internal Server Error
Upload or update an annotation set
Uploads a new annotation set, or, when an annotation set exists for the specified identifier, updates it with the annotation set in the request.
You can create your annotation set in JSON or CSV format. For more information, see Annotation sets.
Request
POST /v1/skills/{skillId}/nluAnnotationSets/{annotationId}/annotations
Request body
For the request body, send a JSON or CSV payload that contains an annotation set. For more information about creating an annotation set, see Annotation sets.
Request parameters
Field | Description | Type |
---|---|---|
skillId |
Required. Part of the URI path. The identifier of the skill for this annotation set. | String |
annotationId |
Required. Part of the URI path. The identifier of the annotation set to upload or update. | String |
Content-Type |
Required. Standard HTTP header. Set the value of this header to application/json or text/csv , whichever corresponds to the type of payload in the request. |
String |
Response
HTTP 200 OK
Error responses
HTTP 400 Bad Request
HTTP 401 Unauthorized
HTTP 403 Forbidden
HTTP 404 Not Found
HTTP 429 Too Many Requests
HTTP 500 Internal Server Error
Get an annotation set
Gets the specified annotation set for the specified skill.
Request
GET /v1/skills/{skillId}/nluAnnotationSets/{annotationId}/annotations
Request parameters
Field | Description | Type |
---|---|---|
skillId |
Required. Part of the URI path. The identifier of the skill for the annotation set. | String |
annotationId |
Required. Part of the URI path. The identifier of the annotation set to get. | String |
Accept |
Required. Standard HTTP header. Set the value of this header to application/json to get the annotation set in JSON format, or text/csv to get it in CSV format. |
String |
Response
HTTP 200 OK
Response body
The response body is a JSON or CSV payload that contains an annotation set. The format depends on the value of the Accept
header in the request. For more information about the structure of an annotation set, see Annotation sets.
Error response
HTTP 400 Bad Request
HTTP 401 Unauthorized
HTTP 403 Forbidden
HTTP 404 Not Found
HTTP 429 Too Many Requests
HTTP 500 Internal Server Error
Update an annotation set's properties
Updates the properties of the specified annotation set. Currently, the only property you can update is the name.
Request
PUT /v1/skills/{skillId}/nluAnnotationSets/{annotationId}/properties
Request body
{
"name": "string"
}
Request parameters
Field | Description | Type |
---|---|---|
skillId |
Required. Part of the URI path. The identifier of the skill for the annotation set. | String |
annotationId |
Required. Part of the URI path. The identifier of the annotation set whose properties to update. | String |
name |
Required. Part of the request body. A name for the annotation set. | String |
Response
HTTP 201 CREATED
Error responses
HTTP 400 Bad Request
HTTP 401 Unauthorized
HTTP 403 Forbidden
HTTP 404 Not Found
HTTP 429 Too Many Requests
HTTP 500 Internal Server Error
Get an annotation set's properties
Gets the properties of the specified annotation set, including the name, locale, number of annotation entries, and the timestamp when the annotation set was last updated.
Request
GET /v1/skills/{skillId}/nluAnnotationSets/{annotationId}/properties
Request parameters
Field | Description | Type |
---|---|---|
skillId |
Required. Part of the URI path. The identifier of the skill for the annotation set. | String |
annotationId |
Required. Part of the URI path. The identifier of the annotation set whose properties to get. | String |
Response
HTTP 200 OK
Response body
{
"locale": "string",
"name": "string",
"numberOfEntries": 0,
"updatedTimestamp": "2019-08-12T18:46:31.674Z"
}
Response fields
Field | Description | Type |
---|---|---|
locale |
The locale of the skill's interaction model for the annotation set. For example: en-US , en-GB , de-DE , and so on. |
String |
name |
The name of the annotation set. | String |
numberOfEntries |
The number of annotation entries in the annotation set. | Integer |
updatedTimestamp |
The date and time when the annotation set was last updated. | String in ISO 8601 date and time format |
Error response
HTTP 400 Bad Request
HTTP 401 Unauthorized
HTTP 403 Forbidden
HTTP 404 Not Found
HTTP 429 Too Many Requests
HTTP 500 Internal Server Error
Delete an annotation set
Deletes an annotation set.
Request
DELETE /v1/skills/{skillId}/nluAnnotationSets/{annotationId}
Request parameters
Field | Description | Type |
---|---|---|
skillId |
Required. Part of the URI path. The identifier of the skill for the annotation set. | String |
annotationId |
Required. Part of the URI path. The identifier of the annotation set to delete. | String |
Response
HTTP 204 NO CONTENT
Error responses
HTTP 400 Bad Request
HTTP 401 Unauthorized
HTTP 403 Forbidden
HTTP 404 Not Found
HTTP 429 Too Many Requests
HTTP 500 Internal Server Error
Start an evaluation
Start an evaluation of the natural language understanding (NLU) model derived from the skill's interaction model, using the specified annotation set.
Request
POST /v1/skills/{skillId}/nluEvaluations
Request body
{
"stage": "string",
"locale": "string",
"source": {
"annotationId": "string"
}
}
Request parameters
Field | Description | Type |
---|---|---|
skillId |
Required. Part of the URI path. The ID of the skill whose model to evaluate against. | String |
stage |
Required. Part of the request body. The stage of the skill whose model to evaluate against. Valid values are development and live . |
String |
locale |
Required. Part of the request body. The locale of the interaction model to evaluate against. For example: en-US , en-GB , de-DE , and so on. |
String |
source.annotationId |
Required. Part of the request body. The identifier of the annotation set to use in the evaluation. You get this identifier when you create a new annotation set or list the annotation sets for a skill. | Object |
Response
HTTP 200 OK
Response headers
Header | Description |
---|---|
Location |
The location of the evaluation. Use this URL to get the status of the evaluation. |
Response body
{
"id": "string"
}
Response fields
Field | Description | Type |
---|---|---|
id |
An identifier for the evaluation. | String |
Error responses
HTTP 400 Bad Request
HTTP 401 Unauthorized
HTTP 403 Forbidden
HTTP 404 Not Found
HTTP 429 Too Many Requests
HTTP 500 Internal Server Error
List evaluations
Get a list of evaluations. You can optionally page and filter the results.
Request
GET /v1/skills/{skillId}/nluEvaluations
Request parameters
Field | Description | Type |
---|---|---|
skillId |
Required. Part of the URI path. The ID of the skill for which to list evaluations. | String |
locale |
Optional. Part of the query string. A locale for filtering the results returned in the response. For example: en-US , en-GB , de-DE , and so on. |
String |
stage |
Optional. Part of the query string. A stage for filtering the results returned in the response. Valid values are development and live . |
String |
annotationId |
Optional. Part of the query string. An identifier for filtering the results returned in the response to only those evaluations started from the specified annotation set. | String |
nextToken |
Optional. Part of the query string. Use this parameter to get more evaluations, after you receive a response with incomplete results. Set it to the value of nextToken from the incomplete response you just received. |
String |
maxResults |
Optional. Part of the query string. The maximum number of items to return in the response. When you don't include this parameter, the default maximum is 10. When you include this parameter, the response might contain fewer items than the value you specify, but will never contain more. You cannot specify a value higher than 100. | Integer |
Response
HTTP 200 OK
Response body
{
"paginationContext": {
"nextToken": "string"
},
"_links": {
"self": {
"href": "string"
},
"next": {
"href": "string"
}
},
"evaluations": [
{
"startTimestamp": "2019-08-12T18:41:09.684Z",
"endTimestamp": "2019-08-12T18:41:09.684Z",
"status": "string",
"errorMessage": "string",
"inputs": {
"locale": "string",
"stage": "development",
"source": {
"annotationId": "string"
}
},
"id": "string"
}
]
}
Response fields
Field | Description | Type |
---|---|---|
paginationContext.nextToken |
When this fields exists, it means the response contains incomplete results. This fields contains the value to use for the nextToken parameter in a subsequent request. |
String |
_links |
Links for API navigation. The structure of this field is defined in the JSON Hypertext Application Language specification. | _links object (see specification) |
evaluations |
Information about an evaluation, including the evaluation ID and the status. | Object |
Error response
HTTP 400 Bad Request
HTTP 401 Unauthorized
HTTP 403 Forbidden
HTTP 404 Not Found
HTTP 429 Too Many Requests
HTTP 500 Internal Server Error
Get the status of an evaluation
Get information about a the specified evaluation, including its status.
Request
GET /v1/skills/{skillId}/nluEvaluations/{evaluationId}
Request parameters
Field | Description | Type |
---|---|---|
skillId |
Required. Part of the URI path. The ID of the skill whose evaluation to get. | String |
evaluationId |
Required. Part of the URI path. The ID of the evaluation to get. | String |
Response
HTTP 200 OK
Response body
{
"startTimestamp": "2019-08-12T18:46:31.674Z",
"endTimestamp": "2019-08-12T18:46:31.674Z",
"status": "string",
"errorMessage": "string",
"inputs": {
"locale": "string",
"stage": "string",
"source": {
"annotationId": "string"
}
},
"_links": {
"results": {
"href": "string"
}
}
}
Response fields
Field | Description | Type |
---|---|---|
startTimestamp |
The date and time when the evaluation started. | String in ISO 8601 date and time format |
endTimestamp |
The date and time when the evaluation ended. | String in ISO 8601 date and time format |
inputs |
The source of the evaluation, including the locale and stage evaluated, and the annotation set that was used in the evaluation. | |
_links |
Links for API navigation. The structure of this field is defined in the JSON Hypertext Application Language specification. Includes the location where you can download the evaluation results. | _links object (see specification) |
Error responses
HTTP 400 Bad Request
HTTP 401 Unauthorized
HTTP 403 Forbidden
HTTP 404 Not Found
HTTP 429 Too Many Requests
HTTP 500 Internal Server Error
Get the results of an evaluation
Get the results of an evaluation. You can optionally page and filter the results.
Request
GET /v1/skills/{skillId}/nluEvaluations/{evaluationId}/results
Request parameters
Field | Description | Type |
---|---|---|
skillId |
Required. Part of the URI path. The ID of the skill whose evaluation to get. | String |
evaluationId |
Required. Part of the URI path. The ID of the evaluation to get. | String |
sort.field |
Optional. Part of the query string. A field for sorting the results returned in the response. Valid values are STATUS , ACTUAL_INTENT , and EXPECTED_INTENT . Results are sorted alphabetically. |
String |
testCaseStatus |
Optional. Part of the query string. A filter to return only test cases with the specified status. Valid values are PASSED and FAILED . |
String |
actualIntentName |
Optional. Part of the query string. A filter to return only test cases that resolve to the specified intent. | String |
expectedIntentName |
Optional. Part of the query string. A filter to return only test cases that are expected to be the specified intent. | String |
nextToken |
Optional. Part of the query string. Use this parameter to get more results, after you receive a response with incomplete results. Set it to the value of nextToken from the incomplete response you just received. |
String |
maxResults |
Optional. Part of the query string. The maximum number of items to return in the response. When you don't include this parameter, the default maximum is 1000. When you include this parameter, the response might contain fewer items than the value you specify, but will never contain more. You cannot specify a value higher than 1000. | Integer |
Response
HTTP 200 OK
Response body
{
"paginationContext": {
"nextToken": "string",
"totalCount": "string"
},
"_links": {
"self": {
"href": "string"
},
"next": {
"href": "string"
}
},
"totalFailed": 0,
"testCases": [
{
"status": "PASSED",
"inputs": {
"utterance": "string",
"referenceTimestamp": "2019-08-12T18:46:31.674Z"
},
"actual": {
"domain": "string",
"intent": {
"name": "string",
"confirmationStatus": "NONE",
"slots": {
"slot1": {
"name": "string",
"value": "string",
"confirmationStatus": "NONE",
"resolutions": {
"resolutionsPerAuthority": [
{
"resolution1": {
"authority": "string",
"status": {
"code": "ER_SUCCESS_MATCH"
},
"values": [
{
"name": "string",
"id": "string"
}
]
}
}
]
}
}
}
}
},
"expected": [
{
"domain": "string",
"intent": {
"name": "string",
"slots": {
"slot1": {
"value": "string"
}
}
}
}
]
}
]
}
Response fields
Field | Description | Type |
---|---|---|
paginationContext.nextToken |
When this fields exists, it means the response contains incomplete results. This fields contains the value to use for the nextToken parameter in a subsequent request. |
String |
paginationContext.totalCount |
The total number of results that match the request parameters. | String |
_links |
Links for API navigation. The structure of this field is defined in the JSON Hypertext Application Language specification. | _links object (see specification) |
totalFailed |
The total number of results that failed. | Integer |
testCases |
A list of test cases, including their status, input utterance, actual intent, and expected intent. | List of objects |
Error responses
HTTP 400 Bad Request
HTTP 401 Unauthorized
HTTP 403 Forbidden
HTTP 404 Not Found
HTTP 429 Too Many Requests
HTTP 500 Internal Server Error