NLU Evaluation Tool API

Use the NLU evaluation tool API to programmatically batch test the natural language understanding (NLU) accuracy of your Alexa skill.

To use the API, you create a data file that contains utterances mapped to the intents and slots you expect to be sent to your skill. This is called an annotation set. Then you start an NLU evaluation using a particular annotation set to determine how well your skill's model performs against your expectations. The tool can help you measure the accuracy of your skill, and run regression testing to help make sure that changes to your interaction model don't degrade the customer experience.

You can use the NLU evaluation tool with skill models for all locales. You can use the tool through the Alexa Skills Kit Command Line Interface (ASK CLI) or the Skill Management API (SMAPI). You can also use the tool within the developer console.

Annotation sets
Use SMAPI

Annotation sets

To use the NLU evaluation tool API, you must provide a data file that contains utterances mapped to their expected intent and slots. This is called an annotation set. Each utterance with its expected intent and slots is called an annotation. You can format this data file as JSON or CSV.

Note: The maximum number of annotations per annotation set is 10,000.

JSON format

To provide an annotation set in JSON format, use the following structure:

{
  "data": [
    {
      "inputs": {
        "utterance": "string",
        "referenceTimestamp": "string (date/time in UTC format)"
      },
      "expected": [
        {
          "intent": {
            "name": "string",
            "slots": {
              "SlotName1": {
                "slotValue": {
                  "value": "string",
                  "type": "Simple"
                }
              },
              "SlotName2": {
                "slotValue": {
                  "type": "List",
                  "values": [
                    {
                      "value": "string",
                      "type": "Simple"
                    },
                    {
                      "value": "string",
                      "type": "Simple"
                    }
                  ]
                }
              }
            }
          }
        }
      ]
    }
  ]
}

Field	Description
`inputs.utterance`	Required. The utterance to test. Do not include the wake word or invocation name. Provide just the utterance as it would be used after the user invokes the skill. You can use either written form or spoken form for the utterance. For example, you can use numerals ("5") or write out numbers ("five"). For more examples, see the rules for custom slot type values.
`inputs.referenceTimestamp`	Optional A time and date in UTC format to use as the basis for relative date and time values (for example, 2019-08-21T00:00:00.000Z). Use this when the utterance tests `AMAZON.DATE` or `AMAZON.TIME` slots with words that represent relative dates and times such as "today," "tomorrow," and "now." Specify the date and time in UTC format: `YYYY-MM-DDThh:mm:ss.sTZD`, for example: 1997-07-16T19:20:30.45Z. Provide the full date and time, including milliseconds, and use "Z" as the time zone designator rather than `+hh:mm` or `-hh:mm`. For more information about testing relative date and time values, see Create an annotation with relative dates or times.
`expected.intent.name`	Required. The name of the expected intent for the utterance.
`expected.intent.slots.SlotName`	Optional. The name of the expected slot for the utterance. You can provide more than one expected slot for an utterance.
`expected.intent.slots.SlotName.slotValue`	An object that represents the expected value for the slot. The `slotValue` object has either a `value` or `values` property, but never both. For a single value slot, `slotValue` has a `type` of `Simple` and a `value` property with the string value of the slot. For a multiple-value slot, `slotValue` has a `type` of `List` and a `values` property containing an array of slot values.
`expected.intent.slots.SlotName.slotValue.type`	Specifies the type of slot value. Set to `Simple` for a single value slot. Set to `List` for a multiple-value slot.
`expected.intent.slots.SlotName.slotValue.value`	For a single-value slot, contains the expected value of the slot as a string. Omit for a multiple-value slot.
`expected.intent.slots.SlotName.slotValue.values`	For a multiple-value slot, contains an array of objects representing the expected values for the slot. Each object in the array has a `value` property for the string value and a `type` property. Specify the values in the same order as they are written in the `utterance`. Omit the `values` array for a single-value slot.
`expected.intent.slots.SlotName.slotValue.values[].value`	Contains the expected slot value as a string.
`expected.intent.slots.SlotName.slotValue.values[].type`	Specifies the type of the slot value. Set to `Simple`.

JSON annotation set example

The following example shows the JSON format for an annotation set. In this example, the activity slot can accept multiple values.

{
  "data": [
    {
      "inputs": {
        "utterance": "plan a trip"
      },
      "expected": [
        {
          "intent": {
            "name": "PlanMyTripIntent"
          }
        }
      ]
    },
    {
      "inputs": {
        "utterance": "i want to go to chicago on monday",
        "referenceTimestamp": "2020-12-11T12:00:00.000Z"
      },
      "expected": [
        {
          "intent": {
            "name": "PlanMyTripIntent",
            "slots": {
              "toCity": {
                "slotValue": {
                  "value": "Chicago",
                  "type": "Simple"
                }
              },
              "travelDate": {
                "slotValue": {
                  "value": "2020-12-14",
                  "type": "Simple"
                }
              }
            }
          }
        }
      ]
    },
    {
      "inputs": {
        "utterance": "plan a trip from seattle to boston"
      },
      "expected": [
        {
          "intent": {
            "name": "PlanMyTripIntent",
            "slots": {
              "toCity": {
                "slotValue": {
                  "value": "Boston",
                  "type": "Simple"
                }
              },
              "fromCity": {
                "slotValue": {
                  "value": "Seattle",
                  "type": "Simple"
                }
              }
            }
          }
        }
      ]
    },
    {
      "inputs": {
        "utterance": "plan a trip from chicago to denver on thursday",
        "referenceTimestamp": "2020-12-11T12:00:00.000Z"
      },
      "expected": [
        {
          "intent": {
            "name": "PlanMyTripIntent",
            "slots": {
              "toCity": {
                "slotValue": {
                  "value": "Denver",
                  "type": "Simple"
                }
              },
              "fromCity": {
                "slotValue": {
                  "value": "Chicago",
                  "type": "Simple"
                }
              },
              "travelDate": {
                "slotValue": {
                  "value": "2020-12-17",
                  "type": "Simple"
                }
              }
            }
          }
        }
      ]
    },
    {
      "inputs": {
        "utterance": "plan a trip to missoula to go camping, hiking, and fishing"
      },
      "expected": [
        {
          "intent": {
            "name": "PlanMyTripIntent",
            "slots": {
              "toCity": {
                "slotValue": {
                  "value": "Missoula",
                  "type": "Simple"
                }
              },
              "activity": {
                "slotValue": {
                  "type": "List",
                  "values": [
                    {
                      "value": "camping",
                      "type": "Simple"
                    },
                    {
                      "value": "hiking",
                      "type": "Simple"
                    },
                    {
                      "value": "fishing",
                      "type": "Simple"
                    }
                  ]
                }
              }
            }
          }
        }
      ]
    },
    {
      "inputs": {
        "utterance": "plan a trip to portland to go kayaking"
      },
      "expected": [
        {
          "intent": {
            "name": "PlanMyTripIntent",
            "slots": {
              "toCity": {
                "slotValue": {
                  "value": "Portland",
                  "type": "Simple"
                }
              },
              "activity": {
                "slotValue": {
                  "value": "kayaking",
                  "type": "Simple"
                }
              }
            }
          }
        }
      ]
    }
  ]
}

CSV format

To provide an annotation set in CSV (UTF-8) format, use the following structure:

utterance, referenceTimestamp, intent, slot[SlotName1], slot[SlotName2]
"Utterance to test", 2019-08-21T00:00:00.000Z, ExpectedIntent, "slot value 1", "slot value 2"

For multiple-value slots, duplicate the slot[SlotName] syntax and include an index number in brackets.

In the following example, "SlotName1" is a multiple value slot. For a given utterance, the first value for this slot is under the column slot[SlotName1][0] and the second value is under slot[SlotName1][1]. If the utterance includes only a single value for this slot, the expected value goes under the column with no index number: slot[SlotName1].

utterance, referenceTimestamp, intent, slot[SlotName1][0],slot[SlotName1][1],slot[SlotName1],slot[SlotName2]

For descriptions of these fields, see the JSON format.

The following example shows the CSV format for an annotation set. In this example, the activity slot can accept multiple values. The examples shows an utterance with three values for the slot, and an utterance with one value.

utterance, referenceTimestamp, intent, slot[toCity],slot[fromCity],slot[travelDate],slot[activity][0],slot[activity][1],slot[activity][2],slot[activity]
plan a trip,,PlanMyTripIntent,,,
i want to go to chicago on monday,2019-08-29T00:00:00.000Z,PlanMyTripIntent,chicago,,2019-08-30
plan a trip from seattle to boston,,PlanMyTripIntent,boston,seattle,
plan a trip from chicago to denver on Monday,2019-08-29T00:00:00.000Z,PlanMyTripIntent,denver,chicago,2019-09-02
plan a trip to missoula to go camping hiking and fishing,,PlanMyTripIntent,Missoula,,,camping,hiking,fishing
plan a trip to portland to go kayaking,,PlanMyTripIntent,Portland,,,,,,kayaking

The CSV file must be encoded with UTF-8.

Use SMAPI

After you create an annotation set, you can use SMAPI to create and update annotation sets, then execute NLU evaluations.

SMAPI endpoint

The API's endpoint is https://api.amazonalexa.com. Every API request must have an Authorization header whose value is the access token retrieved from Login with Amazon.

SMAPI reference

You can use SMAPI to manage annotation sets, and to run NLU evaluations. The following list contains all of the available operations; click each one to see more information.

Create and manage annotation sets

Create a new annotation set for a skill
List a skill's annotation sets
Upload or update an annotation set
Get an annotation set
Update an annotation set's properties
Get an annotation set's properties
Delete an annotation set

Run NLU evaluations

Start an evaluation
List your evaluations
Get the status of an evaluation
Get the results of an evaluation

Create a new annotation set

Creates a new annotation set for the specified skill. This operation returns the annotation ID that you use to upload or update an annotation set.

Request

POST /v1/skills/{skillId}/nluAnnotationSets

Request body

{
  "locale": "string",
  "name": "string"
}

Request parameters

Field	Description	Type
`skillId`	Required. Part of the URI path. The identifier of the skill for the annotation set.	String
`locale`	Required. Part of the request body. The intended interaction model locale for the annotation set. For example: `en-US`, `en-GB`, `de-DE`, and so on.	String
`name`	Required. Part of the request body. A name for the annotation set. Must consist of alphanumeric characters.	String

Response

HTTP 201 CREATED

Response headers

Header	Description
`Location`	The location of the annotation set. Use this URL to get an annotation set.

Response body

{
  "id": "string"
}

Response fields

Field	Description	Type
`id`	The identifier for the annotation set.	String

Error responses

HTTP 400 Bad Request
HTTP 401 Unauthorized
HTTP 403 Forbidden
HTTP 404 Not Found
HTTP 429 Too Many Requests
HTTP 500 Internal Server Error
HTTP 503 Service Unavailable

List a skill's annotation sets

Gets a list of annotation sets for the specified skill.

Request

GET /v1/skills/{skillId}/nluAnnotationSets

Request parameters

Field	Description	Type
`skillId`	Required. Part of the URI path. The ID of the skill for which to list annotation sets.	String
`locale`	Optional. Part of the query string. A locale for filtering the results returned in the response. For example: `en-US`, `en-GB`, `de-DE`, and so on.	String
`nextToken`	Optional. Part of the query string. Use this parameter to get more annotation sets, after you receive a response with incomplete results. Set it to the value of `nextToken` from the incomplete response you just received.	String
`maxResults`	Optional. Part of the query string. The maximum number of items to return in the response. When you don't include this parameter, the default maximum is 10. When you include this parameter, the response might contain fewer items than the value you specify, but will never contain more. You cannot specify a value higher than 100.	Integer

Response

HTTP 200 OK

Response body

{
  "annotationSets": [
    {
      "locale": "string",
      "name": "string",
      "numberOfEntries": 0,
      "updatedTimestamp": "2019-08-12T18:46:31.674Z",
      "annotationId": "string"
    }
  ],
  "paginationContext": {
    "nextToken": "string"
  },
  "_links": {
    "self": {
      "href": "string"
    },
    "next": {
      "href": "string"
    }
  }
}

Response fields

Field	Description	Type
`annotationSets`	A list of annotation set objects, including each annotation set's locale, name, and identifier.	Object
`nextToken`	When this fields exists, it means the response contains incomplete results. This fields contains the value to use for the `nextToken` parameter in a subsequent request.	String
`_links`	Links for API navigation. The structure of this field is defined in the JSON Hypertext Application Language specification.	`_links` object (see specification)

Error response

HTTP 400 Bad Request
HTTP 401 Unauthorized
HTTP 403 Forbidden
HTTP 404 Not Found
HTTP 429 Too Many Requests
HTTP 500 Internal Server Error

Upload or update an annotation set

Uploads a new annotation set, or, when an annotation set exists for the specified identifier, updates it with the annotation set in the request.

You can create your annotation set in JSON or CSV format. For more information, see Annotation sets.

Request

POST /v1/skills/{skillId}/nluAnnotationSets/{annotationId}/annotations

Request body

For the request body, send a JSON or CSV payload that contains an annotation set. For more information about creating an annotation set, see Annotation sets.

Request parameters

Field	Description	Type
`skillId`	Required. Part of the URI path. The identifier of the skill for this annotation set.	String
`annotationId`	Required. Part of the URI path. The identifier of the annotation set to upload or update.	String
`Content-Type`	Required. Standard HTTP header. Set the value of this header to `application/json` or `text/csv`, whichever corresponds to the type of payload in the request.	String

Response

HTTP 200 OK

Error responses

HTTP 400 Bad Request
HTTP 401 Unauthorized
HTTP 403 Forbidden
HTTP 404 Not Found
HTTP 429 Too Many Requests
HTTP 500 Internal Server Error

Get an annotation set

Gets the specified annotation set for the specified skill.

Request

GET /v1/skills/{skillId}/nluAnnotationSets/{annotationId}/annotations

Request parameters

Field	Description	Type
`skillId`	Required. Part of the URI path. The identifier of the skill for the annotation set.	String
`annotationId`	Required. Part of the URI path. The identifier of the annotation set to get.	String
`Accept`	Required. Standard HTTP header. Set the value of this header to `application/json` to get the annotation set in JSON format, or `text/csv` to get it in CSV format.	String

Response

HTTP 200 OK

Response body

The response body is a JSON or CSV payload that contains an annotation set. The format depends on the value of the Accept header in the request. For more information about the structure of an annotation set, see Annotation sets.

Error response

HTTP 400 Bad Request
HTTP 401 Unauthorized
HTTP 403 Forbidden
HTTP 404 Not Found
HTTP 429 Too Many Requests
HTTP 500 Internal Server Error

Update an annotation set's properties

Updates the properties of the specified annotation set. Currently, the only property you can update is the name.

Request

PUT /v1/skills/{skillId}/nluAnnotationSets/{annotationId}/properties

Request body

{
  "name": "string"
}

Request parameters

Field	Description	Type
`skillId`	Required. Part of the URI path. The identifier of the skill for the annotation set.	String
`annotationId`	Required. Part of the URI path. The identifier of the annotation set whose properties to update.	String
`name`	Required. Part of the request body. A name for the annotation set.	String

Response

HTTP 201 CREATED

Error responses

HTTP 400 Bad Request
HTTP 401 Unauthorized
HTTP 403 Forbidden
HTTP 404 Not Found
HTTP 429 Too Many Requests
HTTP 500 Internal Server Error

Get an annotation set's properties

Gets the properties of the specified annotation set, including the name, locale, number of annotation entries, and the timestamp when the annotation set was last updated.

Request

GET /v1/skills/{skillId}/nluAnnotationSets/{annotationId}/properties

Request parameters

Field	Description	Type
`skillId`	Required. Part of the URI path. The identifier of the skill for the annotation set.	String
`annotationId`	Required. Part of the URI path. The identifier of the annotation set whose properties to get.	String

Response

HTTP 200 OK

Response body

{
  "locale": "string",
  "name": "string",
  "numberOfEntries": 0,
  "updatedTimestamp": "2019-08-12T18:46:31.674Z"
}

Response fields

Field	Description	Type
`locale`	The locale of the skill's interaction model for the annotation set. For example: `en-US`, `en-GB`, `de-DE`, and so on.	String
`name`	The name of the annotation set.	String
`numberOfEntries`	The number of annotation entries in the annotation set.	Integer
`updatedTimestamp`	The date and time when the annotation set was last updated.	String in ISO 8601 date and time format

Error response

HTTP 400 Bad Request
HTTP 401 Unauthorized
HTTP 403 Forbidden
HTTP 404 Not Found
HTTP 429 Too Many Requests
HTTP 500 Internal Server Error

Delete an annotation set

Deletes an annotation set.

Request

DELETE /v1/skills/{skillId}/nluAnnotationSets/{annotationId}

Request parameters

Field	Description	Type
`skillId`	Required. Part of the URI path. The identifier of the skill for the annotation set.	String
`annotationId`	Required. Part of the URI path. The identifier of the annotation set to delete.	String

Response

HTTP 204 NO CONTENT

Error responses

HTTP 400 Bad Request
HTTP 401 Unauthorized
HTTP 403 Forbidden
HTTP 404 Not Found
HTTP 429 Too Many Requests
HTTP 500 Internal Server Error

Start an evaluation

Start an evaluation of the natural language understanding (NLU) model derived from the skill's interaction model, using the specified annotation set.

Request

POST /v1/skills/{skillId}/nluEvaluations

Request body

{
  "stage": "string",
  "locale": "string",
  "source": {
    "annotationId": "string"
  }
}

Request parameters

Field	Description	Type
`skillId`	Required. Part of the URI path. The ID of the skill whose model to evaluate against.	String
`stage`	Required. Part of the request body. The stage of the skill whose model to evaluate against. Valid values are `development` and `live`.	String
`locale`	Required. Part of the request body. The locale of the interaction model to evaluate against. For example: `en-US`, `en-GB`, `de-DE`, and so on.	String
`source.annotationId`	Required. Part of the request body. The identifier of the annotation set to use in the evaluation. You get this identifier when you create a new annotation set or list the annotation sets for a skill.	Object

Response

HTTP 200 OK

Response headers

Header	Description
`Location`	The location of the evaluation. Use this URL to get the status of the evaluation.

Response body

{
  "id": "string"
}

Response fields

Field	Description	Type
`id`	An identifier for the evaluation.	String

Error responses

HTTP 400 Bad Request
HTTP 401 Unauthorized
HTTP 403 Forbidden
HTTP 404 Not Found
HTTP 429 Too Many Requests
HTTP 500 Internal Server Error

List evaluations

Get a list of evaluations. You can optionally page and filter the results.

Request

GET /v1/skills/{skillId}/nluEvaluations

Request parameters

Field	Description	Type
`skillId`	Required. Part of the URI path. The ID of the skill for which to list evaluations.	String
`locale`	Optional. Part of the query string. A locale for filtering the results returned in the response. For example: `en-US`, `en-GB`, `de-DE`, and so on.	String
`stage`	Optional. Part of the query string. A stage for filtering the results returned in the response. Valid values are `development` and `live`.	String
`annotationId`	Optional. Part of the query string. An identifier for filtering the results returned in the response to only those evaluations started from the specified annotation set.	String
`nextToken`	Optional. Part of the query string. Use this parameter to get more evaluations, after you receive a response with incomplete results. Set it to the value of `nextToken` from the incomplete response you just received.	String
`maxResults`	Optional. Part of the query string. The maximum number of items to return in the response. When you don't include this parameter, the default maximum is 10. When you include this parameter, the response might contain fewer items than the value you specify, but will never contain more. You cannot specify a value higher than 100.	Integer

Response

HTTP 200 OK

Response body

{
  "paginationContext": {
    "nextToken": "string"
  },
  "_links": {
    "self": {
      "href": "string"
    },
    "next": {
      "href": "string"
    }
  },
  "evaluations": [
    {
      "startTimestamp": "2019-08-12T18:41:09.684Z",
      "endTimestamp": "2019-08-12T18:41:09.684Z",
      "status": "string",
      "errorMessage": "string",
      "inputs": {
        "locale": "string",
        "stage": "development",
        "source": {
          "annotationId": "string"
        }
      },
      "id": "string"
    }
  ]
}

Response fields

Field	Description	Type
`paginationContext.nextToken`	When this fields exists, it means the response contains incomplete results. This fields contains the value to use for the `nextToken` parameter in a subsequent request.	String
`_links`	Links for API navigation. The structure of this field is defined in the JSON Hypertext Application Language specification.	`_links` object (see specification)
`evaluations`	Information about an evaluation, including the evaluation ID and the status.	Object

Error response

HTTP 400 Bad Request
HTTP 401 Unauthorized
HTTP 403 Forbidden
HTTP 404 Not Found
HTTP 429 Too Many Requests
HTTP 500 Internal Server Error

Get the status of an evaluation

Get information about a the specified evaluation, including its status.

Request

GET /v1/skills/{skillId}/nluEvaluations/{evaluationId}

Request parameters

Field	Description	Type
`skillId`	Required. Part of the URI path. The ID of the skill whose evaluation to get.	String
`evaluationId`	Required. Part of the URI path. The ID of the evaluation to get.	String

Response

HTTP 200 OK

Response body

{
  "startTimestamp": "2019-08-12T18:46:31.674Z",
  "endTimestamp": "2019-08-12T18:46:31.674Z",
  "status": "string",
  "errorMessage": "string",
  "inputs": {
    "locale": "string",
    "stage": "string",
    "source": {
      "annotationId": "string"
    }
  },
  "_links": {
    "results": {
      "href": "string"
    }
  }
}

Response fields

Field	Description	Type
`startTimestamp`	The date and time when the evaluation started.	String in ISO 8601 date and time format
`endTimestamp`	The date and time when the evaluation ended.	String in ISO 8601 date and time format
`inputs`	The source of the evaluation, including the locale and stage evaluated, and the annotation set that was used in the evaluation.
`_links`	Links for API navigation. The structure of this field is defined in the JSON Hypertext Application Language specification. Includes the location where you can download the evaluation results.	`_links` object (see specification)

Error responses

HTTP 400 Bad Request
HTTP 401 Unauthorized
HTTP 403 Forbidden
HTTP 404 Not Found
HTTP 429 Too Many Requests
HTTP 500 Internal Server Error

Get the results of an evaluation

Get the results of an evaluation. You can optionally page and filter the results.

Request

GET /v1/skills/{skillId}/nluEvaluations/{evaluationId}/results

Request parameters

Field	Description	Type
`skillId`	Required. Part of the URI path. The ID of the skill whose evaluation to get.	String
`evaluationId`	Required. Part of the URI path. The ID of the evaluation to get.	String
`sort.field`	Optional. Part of the query string. A field for sorting the results returned in the response. Valid values are `STATUS`, `ACTUAL_INTENT`, and `EXPECTED_INTENT`. Results are sorted alphabetically.	String
`testCaseStatus`	Optional. Part of the query string. A filter to return only test cases with the specified status. Valid values are `PASSED` and `FAILED`.	String
`actualIntentName`	Optional. Part of the query string. A filter to return only test cases that resolve to the specified intent.	String
`expectedIntentName`	Optional. Part of the query string. A filter to return only test cases that are expected to be the specified intent.	String
`nextToken`	Optional. Part of the query string. Use this parameter to get more results, after you receive a response with incomplete results. Set it to the value of `nextToken` from the incomplete response you just received.	String
`maxResults`	Optional. Part of the query string. The maximum number of items to return in the response. When you don't include this parameter, the default maximum is 1000. When you include this parameter, the response might contain fewer items than the value you specify, but will never contain more. You cannot specify a value higher than 1000.	Integer

Response

HTTP 200 OK

Response body

{
  "paginationContext": {
    "nextToken": "string",
    "totalCount": "string"
  },
  "_links": {
    "self": {
      "href": "string"
    },
    "next": {
      "href": "string"
    }
  },
  "totalFailed": 0,
  "testCases": [
    {
      "status": "PASSED",
      "inputs": {
        "utterance": "string",
        "referenceTimestamp": "2019-08-12T18:46:31.674Z"
      },
      "actual": {
        "domain": "string",
        "intent": {
          "name": "string",
          "confirmationStatus": "NONE",
          "slots": {
            "slot1": {
              "name": "string",
              "value": "string",
              "confirmationStatus": "NONE",
              "resolutions": {
                "resolutionsPerAuthority": [
                  {
                    "resolution1": {
                      "authority": "string",
                      "status": {
                        "code": "ER_SUCCESS_MATCH"
                      },
                      "values": [
                        {
                          "name": "string",
                          "id": "string"
                        }
                      ]
                    }
                  }
                ]
              }
            }
          }
        }
      },
      "expected": [
        {
          "domain": "string",
          "intent": {
            "name": "string",
            "slots": {
              "slot1": {
                "value": "string"
              }
            }
          }
        }
      ]
    }
  ]
}

Response fields

Field	Description	Type
`paginationContext.nextToken`	When this fields exists, it means the response contains incomplete results. This fields contains the value to use for the `nextToken` parameter in a subsequent request.	String
`paginationContext.totalCount`	The total number of results that match the request parameters.	String
`_links`	Links for API navigation. The structure of this field is defined in the JSON Hypertext Application Language specification.	`_links` object (see specification)
`totalFailed`	The total number of results that failed.	Integer
`testCases`	A list of test cases, including their status, input utterance, actual intent, and expected intent.	List of objects

Error responses

HTTP 400 Bad Request
HTTP 401 Unauthorized
HTTP 403 Forbidden
HTTP 404 Not Found
HTTP 429 Too Many Requests
HTTP 500 Internal Server Error

NLU Evaluation Tool API

Annotation sets

JSON format

CSV format

Use SMAPI

SMAPI endpoint

SMAPI reference

Create a new annotation set

List a skill's annotation sets

Upload or update an annotation set

Get an annotation set

Update an annotation set's properties

Get an annotation set's properties

Delete an annotation set

Start an evaluation

List evaluations

Get the status of an evaluation

Get the results of an evaluation

Alexa Skills Kit

Resources

Alexa Voice Service

AVS Resources

Connected Devices

Agreements

Blogs

Support