Natural Language Understanding Annotation Set REST API Reference

Use the Natural Language Understanding Annotation (NLU) Annotation Set REST API to define annotation sets for your skill testing. An annotation set is a set of utterances mapped to expected intents and slots for your skill's interaction model. Each utterance with its expected intent and slots is called an annotation.

After you create an annotation set and add annotations, you can test the NLU accuracy of the interaction model. For more details, see NLU Evaluation REST API Reference.

If you want to use create the annotation set in the Alexa developer console, see Batch Test Your NLU Model.

API endpoint

The endpoint of the Annotation Set API is https://api.amazonalexa.com.

Authentication

Each API request must have an authorization header whose value is the access token retrieved from Login with Amazon (LWA). For details, see Get an Access Token for SMAPI.

Operations

The Annotation Set API includes the following operations.

Operation HTTP method and URI

Create annotation set

POST /v1/skills/{skillId}/nluAnnotationSets

Delete annotation set

DELETE /v1/skills/{skillId}/nluAnnotationSets/{annotationId}

Get annotation set contents

GET /v1/skills/{skillId}/nluAnnotationSets/{annotationId}/annotations?maxResults={maxResults}&nextToken={nextToken}

Get annotation set metadata

GET /v1/skills/{skillId}/nluAnnotationSets/{annotationId}

List annotation sets

GET /v1/skills/{skillId}/nluAnnotationSets?nextToken={nextToken}&maxResults={maxResults}

Update annotations

POST /v1/skills/{skillId}/nluAnnotationSets/{annotationId}/annotations

Create annotation set

Create a new annotation set. Add annotations to the set by calling Update annotations.

Request

To create an annotation set, you make a POST request to the nluAnnotationSets resource.

Request path and header example

Copied to clipboard.

POST /v1/skills/{skillId}/nluAnnotationSets
Host: api.amazonalexa.com
Content-Type: application/json
Authorization: Bearer {access token}

Request path and header parameters

Parameter Located in Description Type Required

skillId

Path

Identifies the skill.
Valid values: 1 – 255 characters.

String

Yes

access token

Header

LWA token.

String

Yes

Request body example

Copied to clipboard.

{
    "locale": "en-US",
    "name": "QuickTrivia annotation set1"
}

Request body properties

Property Description Type Required

locale

Locale of the interaction model to test.

String

Yes

name

Name for the annotation set.
Valid values: alphanumeric characters.

String

Yes

Response

A successful response returns HTTP 201 Created, along with a Location parameter in the response header. The response body contains the annotation set ID. Use the ID to reference and add sample utterances to the annotation set. On error, the response returns the appropriate HTTP status code and includes a response body with an error code and human readable message.

Response body example

The following example shows a response.

{
    "id": "annotation.set.1"
}

Response body properties

Property Description Type

id

Identifies the new annotation set.

String

HTTP status codes

Status Description

201 Created

Empty annotation set created successfully. The response header contains a Location parameter that includes the skillId and annotationId. The annotationId is the same as the id in the response body. For example:

HTTP 200 OK
Content-Type: application/json
Location: /v1/skills/{skillId}/nluAnnotationSets/{annotationId}/annotations

400 Bad Request

Indicates that one or more properties in the request body are invalid.
The following example shows the response body with the error and message.

{
     "message": "The property is outside the allowed range.",
     "code": "INVALID_STRING_LENGTH"
}

401 Unauthorized

Request didn't include the authorization token or the token is invalid or expired. Or, the client doesn't have access to the resource.

403 Forbidden

Indicates that the authorization token is valid, but the requested operation isn't allowed.

404 Not Found

Requested resource not found.

429 Too Many Requests

Skill has exceeded the permitted rate limit (specified number of requests per unit of time). The skill can retry the request by using exponential back-off.

500 Server Error

Error occurred on the server. The skill can retry the request by using exponential back-off.

503 Service Unavailable

Server is down for maintenance, overloaded, or otherwise unavailable to handle the incoming request.

Delete annotation set

Delete the specified NLU annotation set.

Request

To delete an annotation set, you make a DELETE request to the nluAnnotationSets resource.

Request path and header example

Copied to clipboard.

DELETE /v1/skills/{skillId}/nluAnnotationSets/{annotationId}
Host: api.amazonalexa.com
Content-Type: application/json
Authorization: Bearer {access token}
Accept: application/json

Request path and header parameters

Parameter Located in Description Type Required

skillId

Path

Identifies the skill.
Valid values: 1 – 255 characters.

String

Yes

annotationId

Path

Identifies the NLU annotation set to delete.

String

Yes

access token

Header

LWA token.

String

Yes

Request body example

The request has no body.

Request body properties

The request has no body.

Response

A successful response returns HTTP 204 No Content. On error, the response returns the appropriate HTTP status code and includes a response body with an error code and human readable message.

Response body example

The response has no body.

Response body properties

The response has no body.

HTTP status codes

Status Description

204 No Content

Annotation set deleted successfully.

400 Bad Request

Server can't process the request due to a client error. For example, the payload is malformed, the annotation set is malformed or empty.

The response body contains a violations array with the errors. The following example shows the response body with a validation error.

{
    "message": "Payload validations failed",
    "violations": [{
        "message": "The \"name\" property at property path \"$.name\" is outside the allowed range.",
        "code": "INVALID_STRING_LENGTH"
    }]
}

401 Unauthorized

Authorization token is invalid, expired, or doesn't have access to the resource.

403 Forbidden

The requested operation isn't allowed.
The response body contains a violations array with the errors.

404 Not Found

The requested resource isn't found. For example, the skillId isn't found.

The following example shows the response body with the error and message.

{
     "message": "The skill cannot be found.",
     "code": "RESOURCE_NOT_FOUND"
}

409 Conflict

Request to run an evaluation for a given skill conflicts with an evaluation request that's currently in progress.

429 Too Many Requests

User has exceeded the permitted rate limit.

500 Internal Server Error

Server has encountered an error.

503 Service Unavailable

Server is down for maintenance, overloaded, or otherwise unavailable to handle the incoming request.

Get annotation set contents

Download the contents of the specified NLU annotation set in comma-separated values (CSV) or JSON format.

Request

To download the contents of the annotation set, you make a GET request to the nluAnnotationSets resource.

Request path and header example

Copied to clipboard.

GET /v1/skills/{skillId}/nluAnnotationSets/{annotationId}/annotations?maxResults={maxResults}&nextToken={nextToken}
Host: api.amazonalexa.com
Content-Type: application/json
Authorization: Bearer {access token}
Accept: {accept-type}

Request path and header parameters

Parameter Located in Description Type Required

skillId

Path

Identifies the skill.
Valid values: 1 – 255 characters.

String

Yes

annotationId

Path

Identifies the annotation set.

String

Yes

maxResults

Query

Maximum number of items to return.
Include when the accept header indicates application/json format only.
Default: 1000.

Number

No

nextToken

Query

Token that you use to get the next set of annotations, after you receive a response with truncated results. Set this parameter to the value of nextToken from the truncated response you just received.
Include when the accept header indicates application/json format only.

String

No

access token

Header

LWA token.

String

Yes

accept-type

Header

Specifies the requested format of the returned contents.
Valid values: text/csv, application/json.

String

Yes

Request body example

The request has no body.

Request body properties

The request has no body.

Response

A successful response returns HTTP 200 OK, along with the contents of the annotation set in the specified format.

On error, the response returns the appropriate HTTP status code and includes a response body with an error code and human readable message.

CSV response body example

When you specify CSV format, the response body includes the entire contents. The first row of the response body contains comma-delimited attribute names of the annotation properties. Subsequent rows define the annotations. The annotation properties in each row follow the strict ordering of the attribute property names defined in the first row.

The following example shows a response in CSV format.

utterance, referenceTimestamp, intent, slot[toCity], slot[fromCity], slot[travelDate]

plan a trip,,PlanMyTripIntent,,,

i want to go to chicago on monday,2019-08-29T00:00:00.000Z,PlanMyTripIntent,chicago,,2019-08-30

plan a trip from seattle to boston,,PlanMyTripIntent,boston,seattle,

plan a trip from chicago to denver on Monday,2019-08-29T00:00:00.000Z,PlanMyTripIntent,denver,chicago,2019-09-02

CSV response body properties

The order of the columns doesn't matter because the first line of the CSV payload specifies the column ordering.

Property Description Type

utterance

Utterance to test.

  • Don't include the wake word or invocation name.
  • You can use either written form or spoken form for the utterance. For example, you can use numerals ("5") or write out numbers ("five"). For more examples, see the rules for custom slot type values.

String

referenceTimestamp

Date and time to use as the basis for relative date and time values. Include when the utterance tests. Use this when the utterance tests AMAZON.DATE or AMAZON.TIME slots with words that represent relative dates and times such as "today," "tomorrow," and "now."
Defined in ISO 8601 format, YYYY-MM-DDThh:mm:ssZ.
For more details about testing relative date and time values, see Create an annotation with relative dates or times.

String

intent`

Name of the expected intent for the utterance.

String

slot

List of expected slots for the utterance.

Array of objects

slot[slotname]

Expected value for the slot.

String

slots[slotname][index]

Expected values for the slot.
For a slot that accepts multiple values, add an index number in brackets. To account for a single value for this slot, add the slot without an index.

String

JSON response body example

The following example shows a response in JSON format.

{
    "annotations": [{

            "uploadId": "upload.1",
            "filePathInUpload": "hello_world.mp3",
            "evaluationWeight": 5,
            "expectedTranscription": "alexa hello world",
            "audioAsset": {
                "downloadUrl": "https://s3.amazon.com/asset1",
                "expiryTime": "2019-10-12T10:20:50.52Z"
            }
        },
        {
            "uploadId": "upload.2",
            "filePathInUpload": "ask say hello.wav",
            "evaluationWeight": 10,
            "expectedTranscription": "ask hello world skill say hello",
            "audioAsset": {
                "downloadUrl": "https://s3.amazon.com/asset2",
                "expiryTime": "2019-10-12T10:20:50.52Z"
            }
        }
    ]
}

JSON response body properties

Property Description Type

inputs

Input to the NLU evaluation.

Object

inputs.utterance

Utterance to test.

  • Don't include the wake word or invocation name.
  • You can use either written form or spoken form for the utterance. For example, you can use numerals ("5") or write out numbers ("five"). For more examples, see the rules for custom slot type values.

String

inputs.referenceTimestamp

Date and time to use as the basis for relative date and time values. Include when the utterance tests. Use this when the utterance tests AMAZON.DATE or AMAZON.TIME slots with words that represent relative dates and times such as "today," "tomorrow," and "now."
Defined in ISO 8601 format, YYYY-MM-DDThh:mm:ssZ.
For more details about testing relative date and time values, see Create an annotation with relative dates or times.

String

expected

Expected results.

Object

expected.intent

Expected intent and slot values.

Object

expected.intent.name

Name of the expected intent for the utterance.

String

expected.intent.slots

List of expected slots for the utterance.

Array of objects

expected.intent.slots.<slotname>.slotValue

Expected value for the slot. The slotValue object has either a value or values property, but never both.

Object

expected.intent.slots.<slotname>.slotValue.type

Slot type.
Valid values: For a single value slot, set to Simple and include a value property with the string value of the slot. For a multiple-value slot, set to List and include a values property with a list of slot values.

String

expected.intent.slots.<slotname>.slotValue.value

Include for a single-value slot only. Contains the expected value of the slot. Omit for a multiple-value slot.

String

expected.intent.slots.<slotname>.slotValue.values

Include for a multiple-value slot only. Contains a list of expected values for the slot. Each object in the array has a value property for the string value and a type property. Specify the values in the same order as they're written in the utterance. Omit for a single-value slot.

Array of objects

expected.intent.slots.<slotname>.slotValue.values[].type

Slot type.
Valid value: Simple.

String

expected.intent.slots.<slotname>.slotValue.values[].value

Contains the expected value of the slot.

String

HTTP status codes

Status Description

200 OK

Response body contains the contents of the NLU annotation set.

400 Bad Request

Server can't process the request due to a client error. For example, the payload is malformed, the annotation set is malformed or empty.

The response body contains a violations array with the errors. The following example shows the response body with a validation error.

{
    "message": "Payload validations failed",
    "violations": [{
        "message": "The \"name\" property at property path \"$.name\" is outside the allowed range.",
        "code": "INVALID_STRING_LENGTH"
    }]
}

401 Unauthorized

Authorization token is invalid, expired, or doesn't have access to the resource.

403 Forbidden

The requested operation isn't allowed.
The response body contains a violations array with the errors.

404 Not Found

The requested resource isn't found. For example, the skillId isn't found.

The following example shows the response body with the error and message.

{
     "message": "The skill cannot be found.",
     "code": "RESOURCE_NOT_FOUND"
}

409 Conflict

Request to run an evaluation for a given skill conflicts with an evaluation request that's currently in progress.

429 Too Many Requests

User has exceeded the permitted rate limit.

500 Internal Server Error

Server has encountered an error.

503 Service Unavailable

Server is down for maintenance, overloaded, or otherwise unavailable to handle the incoming request.

Get annotation set metadata

Get the metadata for the specified annotation set.

Request

To get the metadata for an annotation set, you make a GET request to the nluAnnotationSets resource.

Request path and header example

Copied to clipboard.

GET /v1/skills/{skillId}/nluAnnotationSets/{annotationId}
Host: api.amazonalexa.com
Content-Type: application/json
Authorization: Bearer {access token}
Accept: application/json

Request path and header parameters

Parameter Located in Description Type Required

skillId

Path

Identifies the skill.
Valid values: 1 - 255 characters.

String

Yes

annotationId

Path

Identifies the annotation set.

String

Yes

access token

Header

LWA token.

String

Yes

Request body example

The request has no body.

Request body properties

The request has no body.

Response

A successful response returns HTTP 200 OK, along with the annotation set metadata. On error, the response returns the appropriate HTTP status code and includes a response body with an error code and human readable message.

Response body example

{
    "locale": "en-US",
    "name": "QuickTrivia annotation set1",
    "numberOfEntries": 12,
    "updatedTimestamp": "2019-08-12T18:46:31.674Z"
}

Response body properties

Property Description Type

locale

Locale of the annotation set.

String

name

Name of the annotation set.

String

numberOfEntries

Number of annotations in the annotation set.

Integer

updatedTimestamp

Timestamp of the last update to the annotation set.
Defined in ISO 8601 format, YYYY-MM-DDThh:mm:ssZ.

String

HTTP status codes

Status Description

200 OK

Response body contains the metadata for the NLU annotation set.

400 Bad Request

Server can't process the request due to a client error. For example, the payload is malformed, the annotation set is malformed or empty.

The response body contains a violations array with the errors. The following example shows the response body with a validation error.

{
    "message": "Payload validations failed",
    "violations": [{
        "message": "The \"name\" property at property path \"$.name\" is outside the allowed range.",
        "code": "INVALID_STRING_LENGTH"
    }]
}

401 Unauthorized

Authorization token is invalid, expired, or doesn't have access to the resource.

403 Forbidden

The requested operation isn't allowed.
The response body contains a violations array with the errors.

404 Not Found

The requested resource isn't found. For example, the skillId isn't found.

The following example shows the response body with the error and message.

{
     "message": "The skill cannot be found.",
     "code": "RESOURCE_NOT_FOUND"
}

409 Conflict

Request to run an evaluation for a given skill conflicts with an evaluation request that's currently in progress.

429 Too Many Requests

User has exceeded the permitted rate limit.

500 Internal Server Error

Server has encountered an error.

503 Service Unavailable

Server is down for maintenance, overloaded, or otherwise unavailable to handle the incoming request.

List annotation sets

Get all NLU annotation sets for the specified skill.

Request

To list all annotation sets for the specified skill, you make a GET request to the nluAnnotationSets resource.

Request path and header example

Copied to clipboard.

GET /v1/skills/{skillId}/nluAnnotationSets?nextToken={nextToken}&maxResults={maxResults}
Host: api.amazonalexa.com
Content-Type: application/json
Authorization: Bearer {access token}
Accept: application/json

Request path and header parameters

Parameter Located in Description Type Required

skillId

Path

Identifies the skill.
Valid values: 1 – 255 characters.

String

Yes

maxResults

Query

Maximum number of items to return.
Default: 1000.

Number

No

nextToken

Query

Token that you use to get more items in the list, after you receive a response with truncated results. Set this parameter to the value of nextToken from the truncated response you just received.

String

No

access token

Header

LWA token.

String

Yes

Request body example

The request has no body.

Request body properties

The request has no body.

Response

A successful response returns HTTP 200 OK, along with a list of annotation sets. On error, the response returns the appropriate HTTP status code and includes a response body with an error code and human readable message.

Response body example

{
    "annotationSets": [{
        "locale": "en-US",
        "name": "QuickTrivia annotation set1",
        "numberOfEntries": 1,
        "updatedTimestamp": "2019-08-12T18:46:31.674Z",
        "annotationId": "annotation.set.1"
    }],
    "paginationContext": {
        "nextToken": "someToken.1"
    },
    "_links": {
        "self": {
            "href": "/v1/skills/amzn1.ask.skill.1/nluAnnotationSets"
        },
        "next": {
            "href": "/v1/skills/amzn1.ask.skill.1/nluAnnotationSets?nextToken=token.1&maxResults=100"
        }
    }
}

Response body properties

Property Description Type

annotationSets

List of annotation sets.

Array of objects

annotationSets[].locale

Locale of the annotation set.

String

annotationSets[].name

Name of the annotation set.

String

annotationSets[].numberOfEntries

Number of utterances in the annotation set.

Integer

annotationSets[].updatedTimestamp

Timestamp of the last update to the annotation set.
Defined in ISO 8601 format, YYYY-MM-DDThh:mm:ssZ.

String

annotationSets[].annotationId

Identifies the NLU annotation set.

String

paginationContext

Indicates there are more matching results. If present, the response contains a subset of the results. When there are no more matching results, paginationContext isn't included in the response.

Object

paginationContext.nextToken

Identifies the next set of annotations to return. The token expires in 24 hours.
Use this value for the nextToken parameter in a subsequent List annotation sets request.

String

_links

Links for result navigation.
Defined in JSON Hypertext Application Language specification.

_links object

HTTP status codes

Status Description

200 OK

Response body contains a list of NLU annotation sets.

400 Bad Request

Server can't process the request due to a client error. For example, the payload is malformed, the annotation set is malformed or empty.

The response body contains a violations array with the errors. The following example shows the response body with a validation error.

{
    "message": "Payload validations failed",
    "violations": [{
        "message": "The \"name\" property at property path \"$.name\" is outside the allowed range.",
        "code": "INVALID_STRING_LENGTH"
    }]
}

401 Unauthorized

Authorization token is invalid, expired, or doesn't have access to the resource.

403 Forbidden

The requested operation isn't allowed.
The response body contains a violations array with the errors.

404 Not Found

The requested resource isn't found. For example, the skillId isn't found.

The following example shows the response body with the error and message.

{
     "message": "The skill cannot be found.",
     "code": "RESOURCE_NOT_FOUND"
}

409 Conflict

Request to run an evaluation for a given skill conflicts with an evaluation request that's currently in progress.

429 Too Many Requests

User has exceeded the permitted rate limit.

500 Internal Server Error

Server has encountered an error.

503 Service Unavailable

Server is down for maintenance, overloaded, or otherwise unavailable to handle the incoming request.

Update annotations

Add or update annotations in an existing annotation set. Before you add or update annotations, make sure that you create an annotation set.

Request

To update the annotation set, you make a POST request to the nluAnnotationSets resource.

Request path and header example

Copied to clipboard.

POST /v1/skills/{skillId}/nluAnnotationSets/{annotationId}/annotations
Host: api.amazonalexa.com
Content-Type: {content-type}
Authorization: Bearer {access token}
Accept: application/json

Request path and header parameters

Parameter Located in Description Type Required

skillId

Path

Identifies the skill.
Valid values: 1 – 255 characters.

String

Yes

annotationId

Path

Identifies the annotation set.

String

Yes

content-type

Header

Specifies the format of the included updates.
Valid values: text/csv, application/json.

String

Yes

access token

Header

LWA token.

String

Yes

CSV request body example

When you specify CSV format, the request body includes the entire contents. The first row of the request body must contain comma-delimited attribute names of the annotation properties. Subsequent rows define the annotations. The annotation properties in each row must follow the strict ordering of the attribute property names defined in the first row. For multi-value slot names, duplicate the slot[SlotName] and add an index number in brackets. Include the slot[SlotName] at the end to use when the utterance includes a single value for the multi-value slot.

The following example shows a request in UTF8-encoded CSV format.

utterance, referenceTimestamp, intent, slot[toCity],slot[fromCity],slot[travelDate],slot[activity][0],slot[activity][1],slot[activity][2],slot[activity]

plan a trip,,PlanMyTripIntent,,,

i want to go to chicago on monday,2019-08-29T00:00:00.000Z,PlanMyTripIntent,chicago,,2019-08-30

plan a trip from seattle to boston,,PlanMyTripIntent,boston,seattle,

plan a trip from chicago to denver on Monday,2019-08-29T00:00:00.000Z,PlanMyTripIntent,denver,chicago,2019-09-02

plan a trip to missoula to go camping hiking and fishing,,PlanMyTripIntent,Missoula,,,camping,hiking,fishing

plan a trip to portland to go kayaking,,PlanMyTripIntent,Portland,,,,,,kayaking

CSV request body properties

The order of the columns doesn't matter because the first line of the CSV payload specifies the column ordering.

Property Description Type Required

utterance

Utterance to test.

  • Don't include the wake word or invocation name.
  • You can use either written form or spoken form for the utterance. For example, you can use numerals ("5") or write out numbers ("five"). For more examples, see the rules for custom slot type values.

String

Yes

referenceTimestamp

Date and time to use as the basis for relative date and time values. Include when the utterance tests. Use this when the utterance tests AMAZON.DATE or AMAZON.TIME slots with words that represent relative dates and times such as "today," "tomorrow," and "now."
Defined in ISO 8601 format, YYYY-MM-DDThh:mm:ssZ.
For more details about testing relative date and time values, see Create an annotation with relative dates or times.

String

No

intent`

Name of the expected intent for the utterance.

String

Yes

slot

List of expected slots for the utterance.

Array of objects

No

slot[slotname]

Expected value for the slot.

String

No

slots[slotname][index]

For a slot that accepts multiple values, add an index number in brackets. To account for a single value for this slot, add the slot without an index.

String

No

JSON request body example

The following example shows a request in JSON format.

{
    "data": [{
            "inputs": {
                "utterance": "plan a trip"
            },
            "expected": [{
                "intent": {
                    "name": "PlanMyTripIntent"
                }
            }]
        },
        {
            "inputs": {
                "utterance": "i want to go to chicago on monday",
                "referenceTimestamp": "2020-12-11T12:00:00.000Z"
            },
            "expected": [{
                "intent": {
                    "name": "PlanMyTripIntent",
                    "slots": {
                        "toCity": {
                            "slotValue": {
                                "value": "Chicago",
                                "type": "Simple"
                            }
                        },
                        "travelDate": {
                            "slotValue": {
                                "value": "2020-12-14",
                                "type": "Simple"
                            }
                        }
                    }
                }
            }]
        },
        {
            "inputs": {
                "utterance": "plan a trip from seattle to boston"
            },
            "expected": [{
                "intent": {
                    "name": "PlanMyTripIntent",
                    "slots": {
                        "toCity": {
                            "slotValue": {
                                "value": "Boston",
                                "type": "Simple"
                            }
                        },
                        "fromCity": {
                            "slotValue": {
                                "value": "Seattle",
                                "type": "Simple"
                            }
                        }
                    }
                }
            }]
        },
        {
            "inputs": {
                "utterance": "plan a trip from chicago to denver on thursday",
                "referenceTimestamp": "2020-12-11T12:00:00.000Z"
            },
            "expected": [{
                "intent": {
                    "name": "PlanMyTripIntent",
                    "slots": {
                        "toCity": {
                            "slotValue": {
                                "value": "Denver",
                                "type": "Simple"
                            }
                        },
                        "fromCity": {
                            "slotValue": {
                                "value": "Chicago",
                                "type": "Simple"
                            }
                        },
                        "travelDate": {
                            "slotValue": {
                                "value": "2020-12-17",
                                "type": "Simple"
                            }
                        }
                    }
                }
            }]
        },
        {
            "inputs": {
                "utterance": "plan a trip to missoula to go camping, hiking, and fishing"
            },
            "expected": [{
                "intent": {
                    "name": "PlanMyTripIntent",
                    "slots": {
                        "toCity": {
                            "slotValue": {
                                "value": "Missoula",
                                "type": "Simple"
                            }
                        },
                        "activity": {
                            "slotValue": {
                                "type": "List",
                                "values": [{
                                        "value": "camping",
                                        "type": "Simple"
                                    },
                                    {
                                        "value": "hiking",
                                        "type": "Simple"
                                    },
                                    {
                                        "value": "fishing",
                                        "type": "Simple"
                                    }
                                ]
                            }
                        }
                    }
                }
            }]
        },
        {
            "inputs": {
                "utterance": "plan a trip to portland to go kayaking"
            },
            "expected": [{
                "intent": {
                    "name": "PlanMyTripIntent",
                    "slots": {
                        "toCity": {
                            "slotValue": {
                                "value": "Portland",
                                "type": "Simple"
                            }
                        },
                        "activity": {
                            "slotValue": {
                                "value": "kayaking",
                                "type": "Simple"
                            }
                        }
                    }
                }
            }]
        }
    ]
}

JSON request body properties

Property Description Type Required

inputs

Input to the NLU evaluation.

Object

Yes

inputs.utterance

Utterance to test.

  • Don't include the wake word or invocation name.
  • You can use either written form or spoken form for the utterance. For example, you can use numerals ("5") or write out numbers ("five"). For more examples, see the rules for custom slot type values.

String

Yes

inputs.referenceTimestamp

Date and time to use as the basis for relative date and time values. Include when the utterance tests. Use this when the utterance tests AMAZON.DATE or AMAZON.TIME slots with words that represent relative dates and times such as "today," "tomorrow," and "now."
Defined in ISO 8601 format, YYYY-MM-DDThh:mm:ssZ.
For more details about testing relative date and time values, see Create an annotation with relative dates or times.

String

No

expected

Expected results.

Object

Yes

expected.intent

Expected intent and slot values.

Object

Yes

expected.intent.name

Name of the expected intent for the utterance.

String

Yes

expected.intent.slots

List of expected slots for the utterance.

Array of objects

No

expected.intent.slots.<slotname>.slotValue

Expected value for the slot. The slotValue object has either a value or values property, but never both.

Object

No

expected.intent.slots.<slotname>.slotValue.type

Slot type.
Valid values: For a single value slot, set to Simple and include a value property with the string value of the slot. For a multiple-value slot, set to List and include a values property with a list of slot values.

String

Yes

expected.intent.slots.<slotname>.slotValue.value

Include for a single-value slot only. Contains the expected value of the slot. Omit for a multiple-value slot.

String

No

expected.intent.slots.<slotname>.slotValue.values

Include for a multiple-value slot only. Contains a list of expected values for the slot. Each object in the array has a value property for the string value and a type property. Specify the values in the same order as they're written in the utterance. Omit for a single-value slot.

Array of objects

No

expected.intent.slots.<slotname>.slotValue.values[].type

Slot type.
Valid value: Simple.

String

Yes

expected.intent.slots.<slotname>.slotValue.values[].value

Contains the expected value of the slot.

String

Yes

Response

A successful response returns HTTP 204 No Content. On error, the response returns the appropriate HTTP status code and includes a response body with an error code and human readable message.

Response body example

The response has no body.

Response body properties

The response has no body.

HTTP status codes

Status Description

HTTP 204 No Content

Annotation set updated successfully.

400 Bad Request

Server can't process the request due to a client error. For example, the payload is malformed, the annotation set is malformed or empty.

The response body contains a violations array with the errors. The following example shows the response body with a validation error.

{
    "message": "Payload validations failed",
    "violations": [{
        "message": "The \"name\" property at property path \"$.name\" is outside the allowed range.",
        "code": "INVALID_STRING_LENGTH"
    }]
}

401 Unauthorized

Authorization token is invalid, expired, or doesn't have access to the resource.

403 Forbidden

The requested operation isn't allowed.
The response body contains a violations array with the errors.

404 Not Found

The requested resource isn't found. For example, the skillId isn't found.

The following example shows the response body with the error and message.

{
     "message": "The skill cannot be found.",
     "code": "RESOURCE_NOT_FOUND"
}

409 Conflict

Request to run an evaluation for a given skill conflicts with an evaluation request that's currently in progress.

429 Too Many Requests

User has exceeded the permitted rate limit.

500 Internal Server Error

Server has encountered an error.

503 Service Unavailable

Server is down for maintenance, overloaded, or otherwise unavailable to handle the incoming request.

Update metadata

Update the metadata for the specified annotation set.

Request

To update the annotation set, you make a PUT request to the nluAnnotationSets resource.

Request path and header example

Copied to clipboard.

PUT /v1/skills/{skillId}/nluAnnotationSets/{annotationId}
Host: api.amazonalexa.com
Content-Type: application/json
Authorization: Bearer {access token}
Accept: application/json

Request path and header parameters

Parameter Located in Description Type Required

skillId

Path

Identifies the skill.
Valid values: 1 – 255 characters.

String

Yes

annotationId

Path

Identifies the NLU annotation set.

String

Yes

access token

Header

LWA token.

String

Yes

Request body example

Copied to clipboard.

{
    "name": "QuickTrivia annotation set10"
}

Request body properties

Property Description Type Required

name

Name for the NLU annotation set.
Valid value: alphanumeric characters.

String

Yes

Response

A successful response returns HTTP 201 Created. On error, the response returns the appropriate HTTP status code and includes a response body with an error code and human readable message.

Response body example

The response has no body.

Response body properties

The response has no body.

HTTP status codes

Status Description

HTTP 201 Created

Annotation set properties updated successfully.

400 Bad Request

Server can't process the request due to a client error. For example, the payload is malformed, the annotation set is malformed or empty.

The response body contains a violations array with the errors. The following example shows the response body with a validation error.

{
    "message": "Payload validations failed",
    "violations": [{
        "message": "The \"name\" property at property path \"$.name\" is outside the allowed range.",
        "code": "INVALID_STRING_LENGTH"
    }]
}

401 Unauthorized

Authorization token is invalid, expired, or doesn't have access to the resource.

403 Forbidden

The requested operation isn't allowed.
The response body contains a violations array with the errors.

404 Not Found

The requested resource isn't found. For example, the skillId isn't found.

The following example shows the response body with the error and message.

{
     "message": "The skill cannot be found.",
     "code": "RESOURCE_NOT_FOUND"
}

409 Conflict

Request to run an evaluation for a given skill conflicts with an evaluation request that's currently in progress.

429 Too Many Requests

User has exceeded the permitted rate limit.

500 Internal Server Error

Server has encountered an error.

503 Service Unavailable

Server is down for maintenance, overloaded, or otherwise unavailable to handle the incoming request.

Object definitions

The NLU Annotation Set API defines the following objects.

Locale values

The following table shows valid values for the locale property.

Locale code Language

ar-SA

Arabic (SA)

de-DE

German (DE)

en-AU

English (AU)

en-CA

English (CA)

en-GB

English (UK)

en-IN

English (IN)

en-US

English (US)

es-ES

Spanish (ES)

es-MX

Spanish (MX)

es-US

Spanish (US)

fr-CA

French (CA)

fr-FR

French (FR)

hi-IN

Hindi (IN)

it-IT

Italian (IT)

ja-JP

Japanese (JP)

pt-BR

Portuguese (BR)