Alexa.UIController Interface

Implement the Alexa.UIController interface in your Alexa skill so that users can voice-control onscreen navigation. You can voice-enable the user interface of entertainment devices, such as televisions and game consoles. For details about video skills, see Understand the Video Skill API.

When you implement the UIController interface, you can send Alexa the full details of all user interface elements that appear on the screen, such as buttons, dialogues with options, menu items, and search results. For example, you can send a change event when a dialog appears on the screen with the options continue watching and exit. After that, the user can say "Alexa, continue watching." For more scenarios, see Alexa.UIController Interface (VSK Fire TV).

If you want to voice-enable onscreen navigation without sending the full user interface details, implement the Alexa.KeypadController interface instead.

For the list of languages that the UIController interface supports, see List of Alexa Interfaces and Supported Languages.

Utterances

When you use the Alexa.UIController interface, the voice interaction model is already built for you. After the user says one of the following utterances, Alexa sends a corresponding directive to your skill.

The following examples show some user utterances:

Alexa, select The Aeronauts.
Alexa, select number one.

If you want users to be able to say "Alexa, go back" or "Alexa, scroll right", also implement the Alexa.KeypadController interface.

If you want users to be able to expand an element by saying "Alexa, show details" or "Alexa, show more", also implement the Alexa.MediaDetailsNavigator interface.

Properties and objects

The entity object

The Alexa.UIController interface uses the entity object to represent the content of a user interface element.

Entity object example

{
  "entity": {
    "type": "AMAZON.VideoObject",
    "name": {
      "value": "The Marvelous Mrs. Maisel",
      "variants": ["Mrs. Maisel"]
    },
    "externalIds": {
      "entityId": "video-mno"
    }
  }
}

Entity object schema

Field Description Type Required
type The type of the entity. Valid values are AMAZON.VideoObject, AMAZON.ItemList, AMAZON.SoftwareApplication, AMAZON.Thing. String Yes
name Information about the name of the entity. Object No
name.value The name of the entity, such as "The Marvelous Mrs. Maisel". String Yes, when the name object is included.
name.variants Alternate names for the entity, such as "Mrs. Maisel". List<String> No
externalIds External identifiers for the entity. Map<String,String> No

The element object

The Alexa.UIController interface uses the element object to represent an individual item that appears on the user's screen.

Element object example

{
  "element": {
    "elementId": "elementId-003",
    "ordinal": 3,
    "uiSupportedActions": ["SELECT"],
    "entity": {
      "type": "AMAZON.VideoObject",
      "name": {
        "value": "The Dressmaker"
      },
      "externalIds": {
        "entityId": "video-ghi"
      }
    }
  }
}

Element object schema

Field Description Type Required
elementId A unique identifier for the user interface element. String Yes
ordinal The number of the element when the user can refer to the element by number, such as "Alexa, select number one." Integer No
uiSupportedActions The actions that a user can perform on the element. Valid values are SELECT, EXPAND, SCROLL_RIGHT, SCROLL_LEFT, SCROLL_UP, SCROLL_DOWN, SCROLL_FORWARD. List<String> Yes
entity Information about the element. An entity object. Yes
elements Child elements of the element. Only include the element.elements field for element objects in the uiElements property. A list of element objects. No

The scene object

The Alexa.UIController interface uses the scene object to identify the user interface that currently appears on the user's screen.

Scene object example

{
  "scene": {
    "sceneId": "<Home Screen 1234>"
  }
}

Scene object schema

Field Description Type Required
sceneId An unique identifier for a set of UI elements that appear on the user's screen. String Yes

The uiElements property

The Alexa.UIController interface uses the uiElements property to represent all the elements in the current user interface. The uiElements property value is an object.

uiElements object schema

Field Description Type
scene Identifying information about the user interface that currently appears on the user's screen. A scene object.
elements All the elements in the user interface. A list of element objects.

Discovery

You describe endpoints that support Alexa.UIController using the standard discovery mechanism described in Alexa.Discovery.

Set retrievable to true for the properties that you report when Alexa sends your skill a state report request. Set proactivelyReported to true for the properties that you proactively report to Alexa in a change report.

For the full list of display categories, see display categories.

Discover response example

The following example shows a Discover.Response message for a device that supports the Alexa.UIController and Alexa.PowerController interfaces.

Copied to clipboard.

{
  "event": {
    "header": {
      "namespace": "Alexa.Discovery",
      "name": "Discover.Response",
      "payloadVersion": "3",
      "messageId": "<message id>"
    },
    "payload": {
      "endpoints": [
        {
          "endpointId": "<unique ID of the endpoint>",
          "manufacturerName": "<the manufacturer name of the endpoint>",
          "description": "<a description that is shown in the Alexa app>",
          "friendlyName": "<device name, displayed in the Alexa app>",
          "displayCategories": ["TV"],
          "cookie": {},
          "capabilities": [
            {
              "type": "AlexaInterface",
              "interface": "Alexa.UIController",
              "version": "3.0",
              "properties": {
                "supported": [
                  {
                    "name": "uiElements"
                  }
                ],
                "proactivelyReported": true,
                "retrievable": false
              }
            },
            {
              "type": "AlexaInterface",
              "interface": "Alexa.PowerController",
              "version": "3",
              "properties": {
                "supported": [
                  {
                    "name": "powerState"
                  }
                ],
                "proactivelyReported": true,
                "retrievable": true
              }
            },
            {
              "type": "AlexaInterface",
              "interface": "Alexa",
              "version": "3"
            }
          ]
        }
      ]
    }
  }
}

Directives

ActionOnUIElement directive

Support the ActionOnUIElement directive so that users can voice-control your user interface.

ActionOnUIElement directive payload details

Field Description Type
scene The user interface scene that the user is trying to act on. A scene object.
action The action that the user wants to perform. Valid values are SELECT, EXPAND, SCROLL_RIGHT, SCROLL_LEFT, SCROLL_UP, SCROLL_DOWN, SCROLL_FORWARD. String
element The element that the user wants to perform the action on. An element object.

ActionOnUIElement directive example

The following example illustrates a ActionOnUIElement directive that Alexa sends to your skill.

{
  "directive": {
    "header": {
      "namespace": "Alexa.UIController",
      "name": "ActionOnUIElement",
      "messageId": "<message id>",
      "correlationToken": "<an opaque correlation token>",
      "payloadVersion": "3.0"
    },
    "endpoint": {
      "endpointId": "<endpoint id>",
      "cookie": {}
    },
    "payload": {
      "scene": {"sceneId": "<Home Screen 1234>"},
      "action": "SCROLL_FORWARD",
      "element": {
        "elementId": "<list-001>",
        "uiSupportedActions": ["SCROLL_FORWARD", "SCROLL_FORWARD"],
        "ordinal": 12,
        "entity": {
          "type": "AMAZON.ItemList",
          "name": {
            "value": "Suggested for You",
            "variants": ["Suggested"]
          }
        }
      }
    }
  }
}

ActionOnUIElement response event

If you handle a ActionOnUIElement directive successfully, respond with an Alexa.Response event. In the context object, include the values of all relevant properties.

ActionOnUIElement response event example

Copied to clipboard.

{
  "event": {
    "header": {
      "namespace": "Alexa",
      "name": "Response",
      "messageId": "<message id>",
      "correlationToken": "<an opaque correlation token>",
      "payloadVersion": "3"
    },
    "endpoint": {
      "endpointId": "<endpoint id>"
    },
    "payload": {}
  },
  "context": {}
}

ActionOnUIElement directive error handling

If you can't handle a ActionOnUIElement directive successfully, respond with an Alexa.ErrorResponse event.

State reporting

Alexa sends a ReportState directive to request information about the state of an endpoint. When Alexa sends a ReportState directive, you send a StateReport event in response. The response contains the current state of all the retrievable properties in the context object. You identify your retrievable properties in your discovery response. For details about state reports, see Understand State and Change Reporting.

StateReport response event example

Copied to clipboard.

{
  "event": {
    "header": {
      "namespace": "Alexa",
      "name": "StateReport",
      "messageId": "<message id>",
      "correlationToken": "<an opaque correlation token>",
      "payloadVersion": "3"
    },
    "endpoint": {
      "endpointId": "<endpoint id>"
    },
    "payload": {}
  },
  "context": {
    "properties": [
      {
        "namespace": "Alexa.PowerController",
        "name": "powerState",
        "value": "OFF",
        "timeOfSample": "2017-02-03T16:20:50.52Z",
        "uncertaintyInMilliseconds": 0
      }
    ]
  }
}

Change reporting

You send a ChangeReport event to proactively report changes in the state of an endpoint. You identify the properties that you proactively report in your discovery response. For details about change reports, see Understand State and Change Reporting.

ChangeReport event example

Copied to clipboard.

{  
  "event": {
    "header": {
      "namespace": "Alexa",
      "name": "ChangeReport",
      "messageId": "<message id>",
      "payloadVersion": "3"
    },
    "endpoint": {
      "scope": {
        "type": "BearerToken",
        "token": "<an OAuth2 bearer token>"
      },
      "endpointId": "<endpoint id>"
    },
    "payload": {
      "change": {
        "cause": {
          "type": "VOICE_INTERACTION"
        },
        "properties": [
          {
            "namespace": "Alexa.UIController",
            "name": "uiElements",
            "value": {
              "scene": {"sceneId": "<Home Screen 1234>"},
              "elements": [
                {
                  "elementId": "<list-001>",
                  "ordinal": 12,
                  "uiSupportedActions": ["SCROLL_FORWARD", "SCROLL_FORWARD"],
                  "entity": {
                    "type": "AMAZON.ItemList",
                    "name": {
                      "value": "Suggested for You",
                      "variants": ["Suggested"]
                    }
                  },
                  "elements": [
                    {
                      "elementId": "elementId-001",
                      "ordinal": 1,
                      "uiSupportedActions": ["SELECT"],
                      "entity": {
                        "type": "AMAZON.VideoObject",
                        "name": {
                          "value": "Captain Fantastic"
                        },
                        "externalIds": {
                          "entityId": "video-abc"
                        }
                      }
                    },
                    {
                      "elementId": "elementId-002",
                      "ordinal": 2,
                      "uiSupportedActions": ["SELECT"],
                      "entity": {
                        "type": "AMAZON.VideoObject",
                        "name": {
                          "value": "The Aeronauts"
                        },
                        "externalIds": {
                          "entityId": "video-def"
                        }
                      }
                    },
                    {
                      "elementId": "elementId-003",
                      "ordinal": 3,
                      "uiSupportedActions": ["SELECT"],
                      "entity": {
                        "type": "AMAZON.VideoObject",
                        "name": {
                          "value": "The Dressmaker"
                        },
                        "externalIds": {
                          "entityId": "video-ghi"
                        }
                      }
                    }
                  ]
                }
              ]
            },
            "timeOfSample": "2020-02-03T16:20:50.52Z",
            "uncertaintyInMilliseconds": 0
          }
        ]
      }
    }
  },
  "context": {
    "properties": [
      {
        "namespace": "Alexa.PowerController",
        "name": "powerState",
        "value": "ON",
        "timeOfSample": "2017-02-03T16:20:50.52Z",
        "uncertaintyInMilliseconds": 500
      }
    ]
  }
}

ChangeReport event example to reset UI

If you need to reset the UIController state, for example when the user opens an app on the device that you don't control, send a change report with an empty uiElements property.

Copied to clipboard.

{  
  "event": {
    "header": {
      "namespace": "Alexa",
      "name": "ChangeReport",
      "messageId": "<message id>",
      "payloadVersion": "3"
    },
    "endpoint": {
      "scope": {
        "type": "BearerToken",
        "token": "<an OAuth2 bearer token>"
      },
      "endpointId": "<endpoint id>"
    },
    "payload": {
      "change": {
        "cause": {
          "type": "VOICE_INTERACTION"
        },
        "properties": [
          {
            "namespace": "Alexa.UIController",
            "name": "uiElements",
            "value": {
            },
            "timeOfSample": "2020-02-03T16:20:50.52Z",
            "uncertaintyInMilliseconds": 0
          }
        ]
      }
    }
  },
  "context": {
    "properties": [
      {
        "namespace": "Alexa.PowerController",
        "name": "powerState",
        "value": "ON",
        "timeOfSample": "2017-02-03T16:20:50.52Z",
        "uncertaintyInMilliseconds": 500
      }
    ]
  }
}