NLU Evaluation Tool API

Use the NLU evaluation tool API to programmatically batch test the natural language understanding (NLU) accuracy of your Alexa skill.

To use the API, you create a data file that contains utterances mapped to the intents and slots you expect to be sent to your skill. This is called an annotation set. Then you start an NLU evaluation using a particular annotation set to determine how well your skill's model performs against your expectations. The tool can help you measure the accuracy of your skill, and run regression testing to help make sure that changes to your interaction model don't degrade the customer experience.

You can use the NLU evaluation tool with skill models for all locales. You can use the tool through the Alexa Skills Kit Command Line Interface (ASK CLI) or the Skill Management API (SMAPI). You can also use the tool within the developer console.

Annotation sets

To use the NLU evaluation tool API, you must provide a data file that contains utterances mapped to their expected intent and slots. This is called an annotation set. Each utterance with its expected intent and slots is called an annotation. You can format this data file as JSON or CSV.

JSON format

To provide an annotation set in JSON format, use the following structure:

{
  "data": [
    {
      "inputs": {
        "utterance": "string",
        "referenceTimestamp": "string (date/time in UTC format)"
      },
      "expected": [
        {
          "intent": {
            "name": "string",
            "slots": {
              "SlotName1": {
                "slotValue": {
                  "value": "string",
                  "type": "Simple"
                }
              },
              "SlotName2": {
                "slotValue": {
                  "type": "List",
                  "values": [
                    {
                      "value": "string",
                      "type": "Simple"
                    },
                    {
                      "value": "string",
                      "type": "Simple"
                    }
                  ]
                }
              }
            }
          }
        }
      ]
    }
  ]
}
Field Description
inputs.utterance

Required. The utterance to test.

  • Do not include the wake word or invocation name. Provide just the utterance as it would be used after the user invokes the skill.
  • You can use either written form or spoken form for the utterance. For example, you can use numerals ("5") or write out numbers ("five"). For more examples, see the rules for custom slot type values.
inputs.referenceTimestamp

Optional A time and date in UTC format to use as the basis for relative date and time values (for example, 2019-08-21T00:00:00.000Z). Use this when the utterance tests AMAZON.DATE or AMAZON.TIME slots with words that represent relative dates and times such as "today," "tomorrow," and "now."

Specify the date and time in UTC format: YYYY-MM-DDThh:mm:ss.sTZD, for example: 1997-07-16T19:20:30.45Z. Provide the full date and time, including milliseconds, and use "Z" as the time zone designator rather than +hh:mm or -hh:mm.

For more information about testing relative date and time values, see Create an annotation with relative dates or times.
expected.intent.name Required. The name of the expected intent for the utterance.
expected.intent.slots.SlotName Optional. The name of the expected slot for the utterance. You can provide more than one expected slot for an utterance.
expected.intent.slots.SlotName.slotValue An object that represents the expected value for the slot. The slotValue object has either a value or values property, but never both. For a single value slot, slotValue has a type of Simple and a value property with the string value of the slot. For a multiple-value slot, slotValue has a type of List and a values property containing an array of slot values.
expected.intent.slots.SlotName.slotValue.type Specifies the type of slot value. Set to Simple for a single value slot. Set to List for a multiple-value slot.
expected.intent.slots.SlotName.slotValue.value For a single-value slot, contains the expected value of the slot as a string. Omit for a multiple-value slot.
expected.intent.slots.SlotName.slotValue.values For a multiple-value slot, contains an array of objects representing the expected values for the slot. Each object in the array has a value property for the string value and a type property. Specify the values in the same order as they are written in the utterance. Omit the values array for a single-value slot.
expected.intent.slots.SlotName.slotValue.values[].value Contains the expected slot value as a string.
expected.intent.slots.SlotName.slotValue.values[].type Specifies the type of the slot value. Set to Simple.

JSON annotation set example

The following example shows the JSON format for an annotation set. In this example, the activity slot can accept multiple values.

{
  "data": [
    {
      "inputs": {
        "utterance": "plan a trip"
      },
      "expected": [
        {
          "intent": {
            "name": "PlanMyTripIntent"
          }
        }
      ]
    },
    {
      "inputs": {
        "utterance": "i want to go to chicago on monday",
        "referenceTimestamp": "2020-12-11T12:00:00.000Z"
      },
      "expected": [
        {
          "intent": {
            "name": "PlanMyTripIntent",
            "slots": {
              "toCity": {
                "slotValue": {
                  "value": "Chicago",
                  "type": "Simple"
                }
              },
              "travelDate": {
                "slotValue": {
                  "value": "2020-12-14",
                  "type": "Simple"
                }
              }
            }
          }
        }
      ]
    },
    {
      "inputs": {
        "utterance": "plan a trip from seattle to boston"
      },
      "expected": [
        {
          "intent": {
            "name": "PlanMyTripIntent",
            "slots": {
              "toCity": {
                "slotValue": {
                  "value": "Boston",
                  "type": "Simple"
                }
              },
              "fromCity": {
                "slotValue": {
                  "value": "Seattle",
                  "type": "Simple"
                }
              }
            }
          }
        }
      ]
    },
    {
      "inputs": {
        "utterance": "plan a trip from chicago to denver on thursday",
        "referenceTimestamp": "2020-12-11T12:00:00.000Z"
      },
      "expected": [
        {
          "intent": {
            "name": "PlanMyTripIntent",
            "slots": {
              "toCity": {
                "slotValue": {
                  "value": "Denver",
                  "type": "Simple"
                }
              },
              "fromCity": {
                "slotValue": {
                  "value": "Chicago",
                  "type": "Simple"
                }
              },
              "travelDate": {
                "slotValue": {
                  "value": "2020-12-17",
                  "type": "Simple"
                }
              }
            }
          }
        }
      ]
    },
    {
      "inputs": {
        "utterance": "plan a trip to missoula to go camping, hiking, and fishing"
      },
      "expected": [
        {
          "intent": {
            "name": "PlanMyTripIntent",
            "slots": {
              "toCity": {
                "slotValue": {
                  "value": "Missoula",
                  "type": "Simple"
                }
              },
              "activity": {
                "slotValue": {
                  "type": "List",
                  "values": [
                    {
                      "value": "camping",
                      "type": "Simple"
                    },
                    {
                      "value": "hiking",
                      "type": "Simple"
                    },
                    {
                      "value": "fishing",
                      "type": "Simple"
                    }
                  ]
                }
              }
            }
          }
        }
      ]
    },
    {
      "inputs": {
        "utterance": "plan a trip to portland to go kayaking"
      },
      "expected": [
        {
          "intent": {
            "name": "PlanMyTripIntent",
            "slots": {
              "toCity": {
                "slotValue": {
                  "value": "Portland",
                  "type": "Simple"
                }
              },
              "activity": {
                "slotValue": {
                  "value": "kayaking",
                  "type": "Simple"
                }
              }
            }
          }
        }
      ]
    }
  ]
}

CSV format

To provide an annotation set in CSV (UTF-8) format, use the following structure:

utterance, referenceTimestamp, intent, slot[SlotName1], slot[SlotName2]
"Utterance to test", 2019-08-21T00:00:00.000Z, ExpectedIntent, "slot value 1", "slot value 2"

For multiple-value slots, duplicate the slot[SlotName] syntax and include an index number in brackets.

In the following example, "SlotName1" is a multiple value slot. For a given utterance, the first value for this slot is under the column slot[SlotName1][0] and the second value is under slot[SlotName1][1]. If the utterance includes only a single value for this slot, the expected value goes under the column with no index number: slot[SlotName1].

utterance, referenceTimestamp, intent, slot[SlotName1][0],slot[SlotName1][1],slot[SlotName1],slot[SlotName2]

For descriptions of these fields, see the JSON format.

The following example shows the CSV format for an annotation set. In this example, the activity slot can accept multiple values. The examples shows an utterance with three values for the slot, and an utterance with one value.

utterance, referenceTimestamp, intent, slot[toCity],slot[fromCity],slot[travelDate],slot[activity][0],slot[activity][1],slot[activity][2],slot[activity]
plan a trip,,PlanMyTripIntent,,,
i want to go to chicago on monday,2019-08-29T00:00:00.000Z,PlanMyTripIntent,chicago,,2019-08-30
plan a trip from seattle to boston,,PlanMyTripIntent,boston,seattle,
plan a trip from chicago to denver on Monday,2019-08-29T00:00:00.000Z,PlanMyTripIntent,denver,chicago,2019-09-02
plan a trip to missoula to go camping hiking and fishing,,PlanMyTripIntent,Missoula,,,camping,hiking,fishing
plan a trip to portland to go kayaking,,PlanMyTripIntent,Portland,,,,,,kayaking

The CSV file must be encoded with UTF-8.

Use SMAPI

After you create an annotation set, you can use SMAPI to create and update annotation sets, then run NLU evaluations.

SMAPI endpoint

The API's endpoint is https://api.amazonalexa.com. Every API request must have an Authorization header whose value is the access token retrieved from Login with Amazon.

SMAPI reference

You can use SMAPI to manage annotation sets, and to run NLU evaluations. The following list contains all of the available operations; click each one to see more information.

Create and manage annotation sets

Run NLU evaluations

Create a new annotation set

Creates a new annotation set for the specified skill. This operation returns the annotation ID that you use to upload or update an annotation set.

Request

POST /v1/skills/{skillId}/nluAnnotationSets

Request body

{
  "locale": "string",
  "name": "string"
}

Request parameters

Field Description Type
skillId Required. Part of the URI path. The identifier of the skill for the annotation set. String
locale Required. Part of the request body. The intended interaction model locale for the annotation set. For example: en-US, en-GB, de-DE, and so on. String
name Required. Part of the request body. A name for the annotation set. Must consist of alphanumeric characters. String

Response

HTTP 201 CREATED

Response headers

Header Description
Location The location of the annotation set. Use this URL to get an annotation set.

Response body

{
  "id": "string"
}

Response fields

Field Description Type
id The identifier for the annotation set. String

Error responses

HTTP 400 Bad Request
HTTP 401 Unauthorized
HTTP 403 Forbidden
HTTP 404 Not Found
HTTP 429 Too Many Requests
HTTP 500 Internal Server Error
HTTP 503 Service Unavailable

List a skill's annotation sets

Gets a list of annotation sets for the specified skill.

Request

GET /v1/skills/{skillId}/nluAnnotationSets

Request parameters

Field Description Type
skillId Required. Part of the URI path. The ID of the skill for which to list annotation sets. String
locale Optional. Part of the query string. A locale for filtering the results returned in the response. For example: en-US, en-GB, de-DE, and so on. String
nextToken Optional. Part of the query string. Use this parameter to get more annotation sets, after you receive a response with incomplete results. Set it to the value of nextToken from the incomplete response you just received. String
maxResults Optional. Part of the query string. The maximum number of items to return in the response. When you don't include this parameter, the default maximum is 10. When you include this parameter, the response might contain fewer items than the value you specify, but will never contain more. You cannot specify a value higher than 100. Integer

Response

HTTP 200 OK

Response body

{
  "annotationSets": [
    {
      "locale": "string",
      "name": "string",
      "numberOfEntries": 0,
      "updatedTimestamp": "2019-08-12T18:46:31.674Z",
      "annotationId": "string"
    }
  ],
  "paginationContext": {
    "nextToken": "string"
  },
  "_links": {
    "self": {
      "href": "string"
    },
    "next": {
      "href": "string"
    }
  }
}

Response fields

Field Description Type
annotationSets A list of annotation set objects, including each annotation set's locale, name, and identifier. Object
nextToken When this fields exists, it means the response contains incomplete results. This fields contains the value to use for the nextToken parameter in a subsequent request. String
_links Links for API navigation. The structure of this field is defined in the JSON Hypertext Application Language specification. _links object (see specification)

Error response

HTTP 400 Bad Request
HTTP 401 Unauthorized
HTTP 403 Forbidden
HTTP 404 Not Found
HTTP 429 Too Many Requests
HTTP 500 Internal Server Error

Upload or update an annotation set

Uploads a new annotation set, or, when an annotation set exists for the specified identifier, updates it with the annotation set in the request.

You can create your annotation set in JSON or CSV format. For more information, see Annotation sets.

Request

POST /v1/skills/{skillId}/nluAnnotationSets/{annotationId}/annotations

Request body

For the request body, send a JSON or CSV payload that contains an annotation set. For more information about creating an annotation set, see Annotation sets.

Request parameters

Field Description Type
skillId Required. Part of the URI path. The identifier of the skill for this annotation set. String
annotationId Required. Part of the URI path. The identifier of the annotation set to upload or update. String
Content-Type Required. Standard HTTP header. Set the value of this header to application/json or text/csv, whichever corresponds to the type of payload in the request. String

Response

HTTP 200 OK

Error responses

HTTP 400 Bad Request
HTTP 401 Unauthorized
HTTP 403 Forbidden
HTTP 404 Not Found
HTTP 429 Too Many Requests
HTTP 500 Internal Server Error

Get an annotation set

Gets the specified annotation set for the specified skill.

Request

GET /v1/skills/{skillId}/nluAnnotationSets/{annotationId}/annotations

Request parameters

Field Description Type
skillId Required. Part of the URI path. The identifier of the skill for the annotation set. String
annotationId Required. Part of the URI path. The identifier of the annotation set to get. String
Accept Required. Standard HTTP header. Set the value of this header to application/json to get the annotation set in JSON format, or text/csv to get it in CSV format. String

Response

HTTP 200 OK

Response body

The response body is a JSON or CSV payload that contains an annotation set. The format depends on the value of the Accept header in the request. For more information about the structure of an annotation set, see Annotation sets.

Error response

HTTP 400 Bad Request
HTTP 401 Unauthorized
HTTP 403 Forbidden
HTTP 404 Not Found
HTTP 429 Too Many Requests
HTTP 500 Internal Server Error

Update an annotation set's properties

Updates the properties of the specified annotation set. Currently, the only property you can update is the name.

Request

PUT /v1/skills/{skillId}/nluAnnotationSets/{annotationId}/properties

Request body

{
  "name": "string"
}

Request parameters

Field Description Type
skillId Required. Part of the URI path. The identifier of the skill for the annotation set. String
annotationId Required. Part of the URI path. The identifier of the annotation set whose properties to update. String
name Required. Part of the request body. A name for the annotation set. String

Response

HTTP 201 CREATED

Error responses

HTTP 400 Bad Request
HTTP 401 Unauthorized
HTTP 403 Forbidden
HTTP 404 Not Found
HTTP 429 Too Many Requests
HTTP 500 Internal Server Error

Get an annotation set's properties

Gets the properties of the specified annotation set, including the name, locale, number of annotation entries, and the timestamp when the annotation set was last updated.

Request

GET /v1/skills/{skillId}/nluAnnotationSets/{annotationId}/properties

Request parameters

Field Description Type
skillId Required. Part of the URI path. The identifier of the skill for the annotation set. String
annotationId Required. Part of the URI path. The identifier of the annotation set whose properties to get. String

Response

HTTP 200 OK

Response body

{
  "locale": "string",
  "name": "string",
  "numberOfEntries": 0,
  "updatedTimestamp": "2019-08-12T18:46:31.674Z"
}

Response fields

Field Description Type
locale The locale of the skill's interaction model for the annotation set. For example: en-US, en-GB, de-DE, and so on. String
name The name of the annotation set. String
numberOfEntries The number of annotation entries in the annotation set. Integer
updatedTimestamp The date and time when the annotation set was last updated. String in ISO 8601 date and time format

Error response

HTTP 400 Bad Request
HTTP 401 Unauthorized
HTTP 403 Forbidden
HTTP 404 Not Found
HTTP 429 Too Many Requests
HTTP 500 Internal Server Error

Delete an annotation set

Deletes an annotation set.

Request

DELETE /v1/skills/{skillId}/nluAnnotationSets/{annotationId}

Request parameters

Field Description Type
skillId Required. Part of the URI path. The identifier of the skill for the annotation set. String
annotationId Required. Part of the URI path. The identifier of the annotation set to delete. String

Response

HTTP 204 NO CONTENT

Error responses

HTTP 400 Bad Request
HTTP 401 Unauthorized
HTTP 403 Forbidden
HTTP 404 Not Found
HTTP 429 Too Many Requests
HTTP 500 Internal Server Error

Start an evaluation

Start an evaluation of the natural language understanding (NLU) model derived from the skill's interaction model, using the specified annotation set.

Request

POST /v1/skills/{skillId}/nluEvaluations

Request body

{
  "stage": "string",
  "locale": "string",
  "source": {
    "annotationId": "string"
  }
}

Request parameters

Field Description Type
skillId Required. Part of the URI path. The ID of the skill whose model to evaluate against. String
stage Required. Part of the request body. The stage of the skill whose model to evaluate against. Valid values are development and live. String
locale Required. Part of the request body. The locale of the interaction model to evaluate against. For example: en-US, en-GB, de-DE, and so on. String
source.annotationId Required. Part of the request body. The identifier of the annotation set to use in the evaluation. You get this identifier when you create a new annotation set or list the annotation sets for a skill. Object

Response

HTTP 200 OK

Response headers

Header Description
Location The location of the evaluation. Use this URL to get the status of the evaluation.

Response body

{
  "id": "string"
}

Response fields

Field Description Type
id An identifier for the evaluation. String

Error responses

HTTP 400 Bad Request
HTTP 401 Unauthorized
HTTP 403 Forbidden
HTTP 404 Not Found
HTTP 429 Too Many Requests
HTTP 500 Internal Server Error

List evaluations

Get a list of evaluations. You can optionally page and filter the results.

Request

GET /v1/skills/{skillId}/nluEvaluations

Request parameters

Field Description Type
skillId Required. Part of the URI path. The ID of the skill for which to list evaluations. String
locale Optional. Part of the query string. A locale for filtering the results returned in the response. For example: en-US, en-GB, de-DE, and so on. String
stage Optional. Part of the query string. A stage for filtering the results returned in the response. Valid values are development and live. String
annotationId Optional. Part of the query string. An identifier for filtering the results returned in the response to only those evaluations started from the specified annotation set. String
nextToken Optional. Part of the query string. Use this parameter to get more evaluations, after you receive a response with incomplete results. Set it to the value of nextToken from the incomplete response you just received. String
maxResults Optional. Part of the query string. The maximum number of items to return in the response. When you don't include this parameter, the default maximum is 10. When you include this parameter, the response might contain fewer items than the value you specify, but will never contain more. You cannot specify a value higher than 100. Integer

Response

HTTP 200 OK

Response body

{
  "paginationContext": {
    "nextToken": "string"
  },
  "_links": {
    "self": {
      "href": "string"
    },
    "next": {
      "href": "string"
    }
  },
  "evaluations": [
    {
      "startTimestamp": "2019-08-12T18:41:09.684Z",
      "endTimestamp": "2019-08-12T18:41:09.684Z",
      "status": "string",
      "errorMessage": "string",
      "inputs": {
        "locale": "string",
        "stage": "development",
        "source": {
          "annotationId": "string"
        }
      },
      "id": "string"
    }
  ]
}

Response fields

Field Description Type
paginationContext.nextToken When this fields exists, it means the response contains incomplete results. This fields contains the value to use for the nextToken parameter in a subsequent request. String
_links Links for API navigation. The structure of this field is defined in the JSON Hypertext Application Language specification. _links object (see specification)
evaluations Information about an evaluation, including the evaluation ID and the status. Object

Error response

HTTP 400 Bad Request
HTTP 401 Unauthorized
HTTP 403 Forbidden
HTTP 404 Not Found
HTTP 429 Too Many Requests
HTTP 500 Internal Server Error

Get the status of an evaluation

Get information about a the specified evaluation, including its status.

Request

GET /v1/skills/{skillId}/nluEvaluations/{evaluationId}

Request parameters

Field Description Type
skillId Required. Part of the URI path. The ID of the skill whose evaluation to get. String
evaluationId Required. Part of the URI path. The ID of the evaluation to get. String

Response

HTTP 200 OK

Response body

{
  "startTimestamp": "2019-08-12T18:46:31.674Z",
  "endTimestamp": "2019-08-12T18:46:31.674Z",
  "status": "string",
  "errorMessage": "string",
  "inputs": {
    "locale": "string",
    "stage": "string",
    "source": {
      "annotationId": "string"
    }
  },
  "_links": {
    "results": {
      "href": "string"
    }
  }
}

Response fields

Field Description Type
startTimestamp The date and time when the evaluation started. String in ISO 8601 date and time format
endTimestamp The date and time when the evaluation ended. String in ISO 8601 date and time format
inputs The source of the evaluation, including the locale and stage evaluated, and the annotation set that was used in the evaluation.
_links Links for API navigation. The structure of this field is defined in the JSON Hypertext Application Language specification. Includes the location where you can download the evaluation results. _links object (see specification)

Error responses

HTTP 400 Bad Request
HTTP 401 Unauthorized
HTTP 403 Forbidden
HTTP 404 Not Found
HTTP 429 Too Many Requests
HTTP 500 Internal Server Error

Get the results of an evaluation

Get the results of an evaluation. You can optionally page and filter the results.

Request

GET /v1/skills/{skillId}/nluEvaluations/{evaluationId}/results

Request parameters

Field Description Type
skillId Required. Part of the URI path. The ID of the skill whose evaluation to get. String
evaluationId Required. Part of the URI path. The ID of the evaluation to get. String
sort.field Optional. Part of the query string. A field for sorting the results returned in the response. Valid values are STATUS, ACTUAL_INTENT, and EXPECTED_INTENT. Results are sorted alphabetically. String
testCaseStatus Optional. Part of the query string. A filter to return only test cases with the specified status. Valid values are PASSED and FAILED. String
actualIntentName Optional. Part of the query string. A filter to return only test cases that resolve to the specified intent. String
expectedIntentName Optional. Part of the query string. A filter to return only test cases that are expected to be the specified intent. String
nextToken Optional. Part of the query string. Use this parameter to get more results, after you receive a response with incomplete results. Set it to the value of nextToken from the incomplete response you just received. String
maxResults Optional. Part of the query string. The maximum number of items to return in the response. When you don't include this parameter, the default maximum is 1000. When you include this parameter, the response might contain fewer items than the value you specify, but will never contain more. You cannot specify a value higher than 1000. Integer

Response

HTTP 200 OK

Response body

{
  "paginationContext": {
    "nextToken": "string",
    "totalCount": "string"
  },
  "_links": {
    "self": {
      "href": "string"
    },
    "next": {
      "href": "string"
    }
  },
  "totalFailed": 0,
  "testCases": [
    {
      "status": "PASSED",
      "inputs": {
        "utterance": "string",
        "referenceTimestamp": "2019-08-12T18:46:31.674Z"
      },
      "actual": {
        "domain": "string",
        "intent": {
          "name": "string",
          "confirmationStatus": "NONE",
          "slots": {
            "slot1": {
              "name": "string",
              "value": "string",
              "confirmationStatus": "NONE",
              "resolutions": {
                "resolutionsPerAuthority": [
                  {
                    "resolution1": {
                      "authority": "string",
                      "status": {
                        "code": "ER_SUCCESS_MATCH"
                      },
                      "values": [
                        {
                          "name": "string",
                          "id": "string"
                        }
                      ]
                    }
                  }
                ]
              }
            }
          }
        }
      },
      "expected": [
        {
          "domain": "string",
          "intent": {
            "name": "string",
            "slots": {
              "slot1": {
                "value": "string"
              }
            }
          }
        }
      ]
    }
  ]
}

Response fields

Field Description Type
paginationContext.nextToken When this fields exists, it means the response contains incomplete results. This fields contains the value to use for the nextToken parameter in a subsequent request. String
paginationContext.totalCount The total number of results that match the request parameters. String
_links Links for API navigation. The structure of this field is defined in the JSON Hypertext Application Language specification. _links object (see specification)
totalFailed The total number of results that failed. Integer
testCases A list of test cases, including their status, input utterance, actual intent, and expected intent. List of objects

Error responses

HTTP 400 Bad Request
HTTP 401 Unauthorized
HTTP 403 Forbidden
HTTP 404 Not Found
HTTP 429 Too Many Requests
HTTP 500 Internal Server Error