Alexa.UIController Interface 3.0

Implement the Alexa.UIController interface in your Alexa skill so that users can voice-control onscreen navigation. You can voice-enable the user interface of entertainment devices, such as televisions and game consoles. For details about video skills, see Understand the Video Skill API.

When you implement the Alexa.UIController interface, you can send Alexa the full details of all user interface elements that appear on the screen, such as buttons, dialogues with options, menu items, and search results. For example, you can send a change event when a dialog appears on the screen with the options continue watching and exit. After that, the user can say "Alexa, continue watching." For more scenarios, see the Alexa.UIController Interface for Fire TV.

If you want to voice-enable onscreen navigation without sending the full user interface details, implement the Alexa.KeypadController interface instead.

For the list of languages that the UIController interface supports, see List of Alexa Interfaces and Supported Languages. For the definitions of the message properties, see Alexa Interface Message and Property Reference.

Utterances

The Alexa.UIController interface uses the pre-built voice interaction model. After the user says one of the following utterances, Alexa sends a corresponding directive to your skill.

The following examples show some user utterances:

Alexa, select The Aeronauts.
Alexa, select number one.

If you want users to be able to say "Alexa, go back" or "Alexa, scroll right", also implement the Alexa.KeypadController interface.

Properties and objects

Reportable properties

The Alexa.UIController interface supports the focusedUIElement and uiElements properties. You identify that you support the properties in your discovery response.

FocusedUIElement property

The Alexa.UIController interface uses the focusedUIElement property to represent the element on the user interface that has the focus. The focusedUIElement property value is an object.

The following table shows the definition of the FocusedUIElement object.

Property Description Type Required
scene Identifying information about the user interface that currently appears on the user's screen. Scene object Yes
element Identifies the element in the user interface that has the focus. Element object Yes

UiElements property

The Alexa.UIController interface uses the uiElements property to represent all the elements in the current user interface. The uiElements property value is an object.

The following table shows the definition of the UiElements object.

Property Description Type Required
scene Identifying information about the user interface that currently appears on the user's screen. Scene object Yes
elements Identifies all the elements in the user interface. List of Element objects Yes

Entity object

The Alexa.UIController interface uses the Entity object to represent the content of a user interface element.

The following example shows a video entity.

{
  "entity": {
    "type": "AMAZON.VideoObject",
    "name": {
      "value": "The Marvelous Mrs. Maisel",
      "variants": ["Mrs. Maisel"]
    },
    "externalIds": {
      "entityId": "video-mno"
    }
  }
}

The following table shows the definition of the Entity object.

Property Description Type Required
type The type of the entity. Valid values are AMAZON.VideoObject, AMAZON.ItemList, AMAZON.SoftwareApplication, AMAZON.Thing. String Yes
name Information about the name of the entity. Object No
name.value The name of the entity, such as "The Marvelous Mrs. Maisel." String Yes, when you include the name object
name.variants Alternate names for the entity, such as "Mrs. Maisel." List<String> No
externalIds External identifiers for the entity. Map<String,String> No

Element object

The Alexa.UIController interface uses the Element object to represent an individual item that appears on the user's screen.

The following example shows a video element.

{
  "element": {
    "elementId": "elementId-003",
    "ordinal": 3,
    "uiSupportedActions": ["SELECT"],
    "entity": {
      "type": "AMAZON.VideoObject",
      "name": {
        "value": "The Dressmaker"
      },
      "externalIds": {
        "entityId": "video-ghi"
      }
    }
  }
}

The following table shows the definition of the Element object.

Property Description Type Required
elementId Unique identifier for the user interface element. String Yes
ordinal Number of the element when the user can refer to the element by number, such as "Alexa, select number one." Integer No
uiSupportedActions Actions that a user can perform on the element. Valid values are SELECT, EXPAND, SCROLL_RIGHT, SCROLL_LEFT, SCROLL_UP, SCROLL_DOWN, SCROLL_FORWARD, SCROLL_BACKWARD. List<String> Yes
entity Information about the element. Entity object Yes
elements Child elements of the element. Only include the element.elements field for element objects in the uiElements property. List of Element objects No

Scene object

The Alexa.UIController interface uses the Scene object to identify the user interface that currently appears on the user's screen.

The following example shows the home screen.

{
  "scene": {
    "sceneId": "Home Screen 1234"
  }
}

The following table shows the definition of the Scene object.

Property Description Type Required
sceneId An unique identifier for a set of UI elements that appear on the user's screen. String Yes

Discovery

You describe endpoints that support Alexa.UIController by using the standard discovery mechanism described in Alexa.Discovery.

Set retrievable to true for the properties that you report when Alexa sends your skill a state report request. Set proactivelyReported to true for the properties that you proactively report to Alexa in a change report.

For the full list of display categories, see display categories.

Discover response example

The following example shows a Discover.Response message for a device that supports the Alexa.UIController and Alexa.PowerController interfaces.

Copied to clipboard.

{
  "event": {
    "header": {
      "namespace": "Alexa.Discovery",
      "name": "Discover.Response",
      "payloadVersion": "3",
      "messageId": "Unique identifier, preferably a version 4 UUID"
    },
    "payload": {
      "endpoints": [
        {
          "endpointId": "Unique ID of the endpoint",
          "manufacturerName": "Manufacturer of the endpoint",
          "description": "Description to be shown in the Alexa app",
          "friendlyName": "Device name, displayed in the Alexa app",
          "displayCategories": ["TV"],
          "cookie": {},
          "capabilities": [
            {
              "type": "AlexaInterface",
              "interface": "Alexa.UIController",
              "version": "3.0",
              "properties": {
                "supported": [
                  {
                    "name": "uiElements"
                  },
                  {
                    "name": "focusedUIElement"
                  }
                ],
                "proactivelyReported": true,
                "retrievable": false
              }
            },
            {
              "type": "AlexaInterface",
              "interface": "Alexa.PowerController",
              "version": "3",
              "properties": {
                "supported": [
                  {
                    "name": "powerState"
                  }
                ],
                "proactivelyReported": true,
                "retrievable": true
              }
            },
            {
              "type": "AlexaInterface",
              "interface": "Alexa",
              "version": "3"
            }
          ]
        }
      ]
    }
  }
}

Directives

Alexa sends the following Alexa.UIController interface directives to your skill.

ActionOnUIElement directive

Support the ActionOnUIElement directive so that users can voice-control your user interface.

ActionOnUIElement directive example

The following example shows a ActionOnUIElement directive that Alexa sends to your skill.

{
  "directive": {
    "header": {
      "namespace": "Alexa.UIController",
      "name": "ActionOnUIElement",
      "messageId": "Unique version 4 UUID",
      "correlationToken": "Opaque correlation token",
      "payloadVersion": "3.0"
    },
    "endpoint": {
      "endpointId": "Endpoint id",
      "cookie": {}
    },
    "payload": {
      "scene": {"sceneId": "Home Screen 1234"},
      "action": "SCROLL_FORWARD",
      "element": {
        "elementId": "list-001",
        "uiSupportedActions": ["SCROLL_FORWARD", "SCROLL_RIGHT"],
        "ordinal": 12,
        "entity": {
          "type": "AMAZON.ItemList",
          "name": {
            "value": "Suggested for You",
            "variants": ["Suggested"]
          }
        }
      }
    }
  }
}

ActionOnUIElement directive payload

The following table shows the payload details for the ActionOnUIElement directive.

Property Description Type Required
scene Identifies the user interface scene that the user is trying to act on. Scene object Yes
action Identifies the action that the user wants to perform.
Valid values: SELECT, EXPAND, SCROLL_RIGHT, SCROLL_LEFT, SCROLL_UP, SCROLL_DOWN, SCROLL_FORWARD,SCROLL_BACKWARD.		 String Yes
element Identifies the element that the user wants to perform the action on. Element object Yes

ActionOnUIElement response

If you handle a ActionOnUIElement directive successfully, respond with an Alexa.Response event. In the context object, include the values of all relevant properties.

The following example shows a ActionOnUIElement response.

Copied to clipboard.

{
  "event": {
    "header": {
      "namespace": "Alexa",
      "name": "Response",
      "messageId": "Unique identifier, preferably a version 4 UUID",
      "correlationToken": "Opaque correlation token that matches the request",
      "payloadVersion": "3"
    },
    "endpoint": {
      "endpointId": "Endpoint id"
    },
    "payload": {}
  },
  "context": {}
}

ActionOnUIElement directive error handling

If you can't handle a ActionOnUIElement directive successfully and the error is specific to video, respond with an Alexa.Video.ErrorResponse event. For general errors, respond with a generic Alexa.ErrorResponse event.

State reporting

Alexa sends a ReportState directive to request information about the state of an endpoint. When Alexa sends a ReportState directive, you send a StateReport event in response. The response contains the current state of all the retrievable properties in the context object. You identify your retrievable properties in your discovery response. For details about state reports, see Understand State and Change Reporting.

StateReport response example

Copied to clipboard.

{
  "event": {
    "header": {
      "namespace": "Alexa",
      "name": "StateReport",
      "messageId": "Unique identifier, preferably a version 4 UUID",
      "correlationToken": "Opaque correlation token that matches the request",
      "payloadVersion": "3"
    },
    "endpoint": {
      "endpointId": "Endpoint id"
    },
    "payload": {}
  },
  "context": {
    "properties": [
      {
        "namespace": "Alexa.PowerController",
        "name": "powerState",
        "value": "OFF",
        "timeOfSample": "2020-02-03T16:20:50.52Z",
        "uncertaintyInMilliseconds": 0
      }
    ]
  }
}

Change reporting

You send a ChangeReport event to report changes proactively in the state of an endpoint. You identify the properties that you proactively report in your discovery response. For details about change reports, see Understand State and Change Reporting.

ChangeReport event example

Copied to clipboard.

{
    "event": {
        "header": {
            "namespace": "Alexa",
            "name": "ChangeReport",
            "messageId": "Unique identifier, preferably a version 4 UUID",
            "payloadVersion": "3"
        },
        "endpoint": {
            "scope": {
                "type": "BearerToken",
                "token": "OAuth2.0 bearer token"
            },
            "endpointId": "Endpoint id"
        },
        "payload": {
            "change": {
                "cause": {
                    "type": "VOICE_INTERACTION"
                },
                "properties": [{
                        "namespace": "Alexa.UIController",
                        "name": "uiElements",
                        "value": {
                            "scene": {
                                "sceneId": "Home Screen 1234"
                            },
                            "elements": [{
                                "elementId": "list-001",
                                "ordinal": 12,
                                "uiSupportedActions": ["SCROLL_FORWARD", "SCROLL_RIGHT"],
                                "entity": {
                                    "type": "AMAZON.ItemList",
                                    "name": {
                                        "value": "Suggested for You",
                                        "variants": ["Suggested"]
                                    }
                                },
                                "elements": [{
                                        "elementId": "elementId-001",
                                        "ordinal": 1,
                                        "uiSupportedActions": ["SELECT"],
                                        "entity": {
                                            "type": "AMAZON.VideoObject",
                                            "name": {
                                                "value": "Captain Fantastic"
                                            },
                                            "externalIds": {
                                                "entityId": "video-abc"
                                            }
                                        }
                                    },
                                    {
                                        "elementId": "elementId-002",
                                        "ordinal": 2,
                                        "uiSupportedActions": ["SELECT"],
                                        "entity": {
                                            "type": "AMAZON.VideoObject",
                                            "name": {
                                                "value": "The Aeronauts"
                                            },
                                            "externalIds": {
                                                "entityId": "video-def"
                                            }
                                        }
                                    },
                                    {
                                        "elementId": "elementId-003",
                                        "ordinal": 3,
                                        "uiSupportedActions": ["SELECT"],
                                        "entity": {
                                            "type": "AMAZON.VideoObject",
                                            "name": {
                                                "value": "The Dressmaker"
                                            },
                                            "externalIds": {
                                                "entityId": "video-ghi"
                                            }
                                        }
                                    }
                                ]
                            }]
                        },
                        "timeOfSample": "2020-02-03T16:20:50.52Z",
                        "uncertaintyInMilliseconds": 0
                    },
                    {
                        "namespace": "Alexa.UIController",
                        "name": "focusedUIElement",
                        "value": {
                            "scene": {
                                "sceneId": "Home Screen 1234"
                            },
                            "element": {
                                "elementId": "elementId-003",
                                "ordinal": 3,
                                "uiSupportedActions": ["SELECT"],
                                "entity": {
                                    "type": "AMAZON.VideoObject",
                                    "name": {
                                        "value": "Captain Fantastic"
                                    },
                                    "externalIds": {
                                        "entityId": "video-abc"
                                    }
                                }
                            }
                        },
                        "timeOfSample": "2020-02-03T16:20:50.52Z",
                        "uncertaintyInMilliseconds": 0
                    }
                ]
            }
        }
    },
    "context": {
        "properties": [{
            "namespace": "Alexa.PowerController",
            "name": "powerState",
            "value": "ON",
            "timeOfSample": "2020-02-03T16:20:50.52Z",
            "uncertaintyInMilliseconds": 500
        }]
    }
}

ChangeReport event example to reset UI

To reset the UIController state, for example, when the user opens an app on the device that you don't control, send a change report with an empty uiElements property.

Copied to clipboard.

{  
  "event": {
    "header": {
      "namespace": "Alexa",
      "name": "ChangeReport",
      "messageId": "Unique identifier, preferably a version 4 UUID",
      "payloadVersion": "3"
    },
    "endpoint": {
      "scope": {
        "type": "BearerToken",
        "token": "OAuth2.0 bearer token"
      },
      "endpointId": "Endpoint id"
    },
    "payload": {
      "change": {
        "cause": {
          "type": "VOICE_INTERACTION"
        },
        "properties": [
          {
            "namespace": "Alexa.UIController",
            "name": "uiElements",
            "value": {},
            "timeOfSample": "2020-02-03T16:20:50.52Z",
            "uncertaintyInMilliseconds": 0
          }
        ]
      }
    }
  },
  "context": {
    "properties": [
      {
        "namespace": "Alexa.PowerController",
        "name": "powerState",
        "value": "ON",
        "timeOfSample": "2020-02-03T16:20:50.52Z",
        "uncertaintyInMilliseconds": 500
      }
    ]
  }
}

Last updated: Dec 16, 2022