Amazon Developer

as

Settings
Sign out
Notifications
Alexa
Amazon Appstore
Ring
AWS
Documentation
Support
Contact Us
My Cases
Develop
Test
Publish
Monetize
Engage users
Device specifications
Resources
Skip to main content
The UIController interface takes into account your app’s UI state in order to more intelligently respond to user requests. To use the UIController interface, you need to send Alexa your UI state, listing all elements on the page and the elements in focus. These details are then included in the directives that Alexa sends to your Lambda. If you’re implementing UIController, consider also implementing MediaDetailsNavigator, which is another Alexa interface that uses the UI state report to provide directives for “show more details about <video title>” requests.

Detailed Explanation

For Alexa to more intelligently respond to user’s requests, Alexa needs to know what elements the customer sees on the screen and what actions are available for each element. This information is known as the UI state. A UI state has two different dimensions to describe what’s shown on the UI — the list of elements shown on the screen (which is reported in the uiElements section of the state report), and the element in focus (which is reported in the focusedUIElement section). To let Alexa know your UI state, you (as the app developer) first send a UI report to Amazon (posting the event as a ChangeReport to a specific Alexa gateway endpoint) so that Alexa can dynamically target different screen elements. For example, after launching your app, your app would send a structured list of UI elements on screen (including names, aliases, position, and supported actions for those objects). You send your UI state to the Alexa endpoint whenever your UI changes. When customers make requests related to on-screen UI navigation or selection (e.g., “Alexa, play number three,” “Alexa, select continue watching,” “Alexa, play harry potter,” “Alexa, play this” ), Alexa sends an ActionOnUIElement directive from the UIController interface to your Lambda. The directive contains specific details from the UI state that you previously sent to Alexa. For example, if you have a title on the screen called “Popcorn Movie” in the second position in a list, the UIController interface sends a directive with those details to your Lambda. Overall, the UIController interface lets customers interact in a richer way with voice in your app’s UI, such as selecting their profile by voice or selecting a specific item on the screen by voice.

UIController versus KeypadController

UIController differs from KeypadController in that UIController considers your UI state and includes details from the UI state in the directive it sends. In contrast, KeypadController is stateless and doesn’t receive any specific details about your app’s UI. As a result, UIController is a lot more powerful in the types of utterances it can support. You can use both KeypadController and UIController together. UIController will take precedence over KeypadController whenever the UI state is available. If the UI state is not available, KeypadController will be used as a fallback. If you don’t have KeypadController implemented, the fallback would be the in-app voice scrolling and selection functionality that is built-in to Fire TV.

Supported Utterances

Alexa sends an ActionOnUIElement directive to your Lambda when users make a navigation or selection request related to an element on the screen. Unlike the other utterances, because UIController has access to your UI state, there aren’t a rigid set of supported utterances. The supported utterances and features can be dynamically controlled based on the on-screen context you provide in your UI state report. The utterances your app supports depend on the elements you provide in your UI state reporting. For example, if you describe an element called “Jake’s watchlist” as an ordinal in the third position, then Alexa would support utterances such as “Go to Jake’s watchlist” or “select the third one.” (Ordinals are identifiers that indicate position, such as number one, number two, etc.) Depending on the detail in your state reporting, the following utterances and scenarios can be supported:
  • You can scroll a list by saying, “Scroll/Page right/left/up/down.” (Note: “Scroll back” is not supported.)
  • You can see more/all the titles that are not visible in the shortlist by saying “Show/See more.” (Note: “Show/see all” is not supported.)
  • You can select an item from the list by saying “Select/Pick/Choose this/that/the first one/number one,” or ask Alexa to play a title from the list, like “Select/Play/Choose/Watch Mockingjay.”
  • You can also use your voice to emulate a d-pad controller on your remote by saying “Go right/left/up/down” followed by “Select/Choose/Watch/Play.”
  • On a video detail page, you can take the following actions showing on the screen: “Add to watchlist,” “Watch Now,” “Watch Trailer,” etc.
  • You can navigate to any UI location, such as recently watched, watch lists, recommended videos.
  • You can respond to a dialog screen and pick an option without needing to reach for the remote control.
For specific examples of utterances in multiple locales, see the following list. These utterances include utterances from KeypadController and RemoteVideoPlayer: SearchAndPlay.

Sending UI State Reports

You need to send a UI state report to inform Alexa about the currently visible items on the customer’s screen.

VSK-Enabled Apps

For VSK-enabled apps, you send the UI state report via the Alexa Client Library. The UI Controller requires Alexa Client Library version 1.4.9 and later. See Step 3: Integrate the Alexa Client Library for information about downloading and installing the Alexa Client Library.

Step 1 - Add the UI Controller to Alexa Client Library Supported Capabilities

Before you can send UI state reports, you need to add support for the UI Controller in the Alexa Client Library. In the initializeAlexaClient method, add getAlexaUIControllerCapability to the list of supportedCapabilities.

Step 2 - Send UI Reports through Alexa Client Library

To send your UI state report, call the setUIState method in the AlexaClientManager class of the Alexa Client Library.
The setUIState method takes a states parameter. The states parameter is a map of user interface properties representing elements displayed on screen. A key string, representing the property name to set (i.e. uiElements, focusedUIElement), is matched with a value string, the JSON encoded contents of the value key associated with this property.

VSK-Enabled Devices

For third party devices, you send the UI state report as a POST request to an Alexa event gateway endpoint every time there is a actionable change with your UI State. The steps to send UI state report events are listed.

Step 1 - Request Permission to Send Events

UI state report events are sent to the Alexa event gateway, which requires an authentication token for each customer. In the Alexa console, find your skill and, select the PERMISSIONS tab on the left, and turn on Send Alexa Events. Record your Alexa Client Id and Alexa Client Secret (shown on this screen), which will be used in the next step to authenticate the customer with Login with Amazon (LwA). For more information, see Configure Permissions for Customer Information in Your Skill in the Alexa documentation. (State reporting is common with the Alexa Smart Home APIs.)

Step 2 - Add Code to Handle AcceptGrant Directives

When you request permission to send Alexa events, your skill’s Lambda function will receive an AcceptGrant directive for each customer. The AcceptGrant directive contains an authorization code that you will use to obtain an authentication token for the customer. You must add code to obtain and store the tokens in the same region where your customer is located. Here’s an example AcceptGrant directive:
AcceptGrant directive payload details
FieldDescriptionType
grantInformation that identifies a user in Amazon Alexa systems.Object
grant.typeThe type of grant. Currently, the only valid value is OAuth2.AuthorizationCode.String
grant.codeAn authorization code for the user.String
granteeInformation that identifies a user in a linked account service or system.Object
grantee.typeThe type of grantee. Currently, the only valid value is BearerToken.String
grantee.tokenThe user access token received by Alexa during the account linking process.String
If the AcceptGrant directive was successfully handled, respond with an AcceptGrant.Response event. AcceptGrant response event example
If you can’t handle an AcceptGrant directive successfully, respond with an Alexa.Authorization.ErrorResponse event. Here are some reasons why the handling might be unsuccessful:
  • You are unable to call Login with Amazon to exchange the authorization code for access and refresh tokens.
  • You are unable to store the access and refresh tokens for the user.
  • Any other error that occurs while you try to retrieve and store the access and refresh tokens.
Here’s an example Alexa.Authorization.ErrorResponse:
A user will not be able to enable your skill if an error occurs during AcceptGrant processing. See the following for more details:

Step 3 - Indicate that Properties Are Proactively Reported During Discovery

Alexa sends a Discover directive through the Discovery Interface to understand your app’s capabilities. In addition to declaring support for the UIController interface, during discovery you must also indicate that you will send change reports. You do this by indicating the properties are proactivelyReported in your discovery response. Here’s how you can configure your response to the Discover directive to indicate that you support the UIController capability:
For more information about state reporting and authorization in general, see the following:

Step 4 - Send UI State Reports to the Alexa Event Gateway

The final step is to begin sending UI state reports to the Alexa event gateway when a reportable property changes. When your UI state changes for any reason, send a UI state report. Each message to the event gateway should include the authentication token as an HTTP header and in the body of the message. To send your UI state to the Alexa event gateway, submit the request header as follows:
(Change <access-token-from-Amazon to the actual authentication token you obtained in Step 2.) The request body must contain a JSON object with your UI state details as described in UI State Report Example and Schema. See the “Example” tab in the next section for details. For more details about authorization, see the following in the Alexa documentation:

UI State Report Example and Schema

The following tabs provide a UI state report example (which shows sample populated values) and a schema (which shows data types rather than examples). A tree diagram is also included to show the schema’s visual hierarchy.
(The context and properties objects aren’t used with Fire TV implementations. The context object would include only updates from other Alexa components such as Alexa.PowerState. Leave this empty if there’s no state update from other components when the Change Report event is triggered.)

Data Structures

This section describes the data structures for the state report schema as previously described. These tree diagrams are similar to the earlier schema but are paired down here to show the required schema for uiElements and focusedUIElement sections of your UI state report.

Schema for uiElements versus focusedUIElements

The properties array has two main objects: one with a namespace of uiElements, and another with a namespace of focusedUIElement. Their schemas are slightly different:
  • uiElements lists all elements on the screen; this section uses an elements array (plural) and also contains a nested array similarly called elements where you can provide an optional list of sub-elements.
  • focusedUIElement lists the element in focus; this section uses an element object (singular).
The following tabs shows the different schemas for uiElements and focusedUIElement lists.

Payload Definitions

The following table defines the payload for the UI state reports that you send. Note that elements repeated (e.g., uiSupportedActions) in both the uiElements and focusedUIElement sections have the same definitions.
Payload Descriptions
FieldDescriptionData Type
change
required
Contains details about the change in state.Object
cause
required
Provides the reason for the change in state.Object
type
required
The type of state change. One option is allowed for Fire TV implementations: AMAZON.PHYSICAL_INTERACTION.Enum
properties
optional
Contains the details about the state change. The properties element contains details for state changes relating to both uiElements (a list of elements) and focusedUIElement objects.Array
namespace
required
The namespace for the payload. For Fire TV implementations, one option is allowed: Alexa.UIController.String
name
required
The name of the interface for the payload. Two options are allowed: uiElement and focusedUIElement. If you’re providing a list of elements, use uiElement. In contrast, if you’re reporting about the element in focus, use focusedUIElement. See the sample UI State Report Example for details and context.Example: uiElementsString
value
optional
Contains a scene object that describes the elements on the customer’s screen.Object
scene
required
Includes a sceneId property containing information about the scene displayed on the customer’s UI.Object
sceneId
required
ID of the scene rendered on the customer’s UI.Example: sceneId-001String
elements
optional
List of UI elements rendered on this scene. Used only with uiElements state reporting.Array
element
required
Element that has the visual focus on this scene. Used only with focusedUIElement state reporting.List<Element>
entity
required
A Thing (a generic object) that describes this element.Object
type
required
Type of this entity. See the Enum<EntityTypes> section for details.Example: AMAZON.ItemListEnum<EntityTypes>
name
optional
Name of this entity. Contains two possible properties: value and variants.Object
value
optional
Preferred name of this entity.Example: SuperbadString
variants
optional
Additional names used to refer this entity.Example: ExtremebadString
externalIds
optional
Identifiers for this entity unique within the scope of this endpoint. The entity.externalIds field will be opaque to Alexa and will only be used to include in directives from Alexa. The identifiers reported in entity.externalIds are required to be understood by the endpoint on all directive commands that accept an entity structure.Example: "mediaBrowseId": "d9fd6168-81b8-4a28-a660-ba35141ca5b6"Map<String, String>
elementId
required
Identifier of this UI element unique within the scope of the reported scene. The elementId field will be opaque to Alexa and will only be used to include in directives from Alexa to this endpoint. The endpoint can use this field to include information it needs to be able to locate this element on the rendered UI in response to a directive command.Example: elementId_11String
ordinal
optional
The number of the element that can be used in spoken language selection. For example, if this value is one then “Alexa select number 1” will select this element.Example: Number oneInteger
uiSupportedActions
required
A list of enumerated actions a user can take on this UI element through the UIController interface. See Enum<uiSupportedActions> for details about enum options.Example: SELECTEnum
elements
optional
Optional list of nested UI elements. This element is unique to uiElements.List<ElementWithChildren>
timeOfSample
optional
The time at which the property value was recorded.Example: 2018-02-03T16:20:49.52ZISO 8601 time
uncertaintyInMilliseconds
optional
The number of milliseconds that have elapsed since the property value was last confirmed.Example: 0Integer

Enum<EntityTypes>

The type enum for entity allows the following options:
  • AMAZON.VideoObject: Entity type to represent a video item rendered on a UI. This entity is an abstraction for various types of videos (e.g. movie, tv shows, trailers, etc.). Example screen:
    Example: “Interstellar” from this screen

    Example: “Interstellar” from this screen

  • AMAZON.ItemList: Entity type to represent a list of items. This entity can be used to represent a collection of entities of same or different types. Example screen:
    Example: “Recents” from this screen

    Example: “Recents” from this screen

  • AMAZON.SoftwareApplication: Entity type to represent a software application. Example screen:
    Example: “IMDb TV” from this screen

    Example: “IMDb TV” from this screen

  • AMAZON.Thing: Entity type to represent generic entities whose specific type is either unknown to the API integrator or undefined in the UIController API specification. (Note that MediaDetailsNavigator does not support showing more details about this entity type.) Example screen:
    Example: A screen reader application reporting “Captain Fantastic” from this screen

    Example: A screen reader application reporting “Captain Fantastic” from this screen

Enum<uiSupportedActions>

The uiSupportedActions enum allows the following options.
NameDescriptionDirective Handling Expectation
SELECTAction that selects a UI elementThe app should do an equivalent of a single click on the provided element
EXPANDAction that “expands” or otherwise “reveals more” on a UI element.The app should expand the hidden, obstructed, truncated, or otherwise expandable content of the provided element.
SCROLL_RIGHTAction that scrolls in the right direction on a scrollable elementThe app should perform a scroll action on the specified element in the right direction such that the sense of the scroll is clear, either through animation or by maintaining some elements on the screen
SCROLL_LEFTAction that scrolls in the left direction on a scrollable elementThe app should perform a scroll action on the specified element in the left direction such that the sense of the scroll is clear, either through animation or by maintaining some elements on the screen
SCROLL_UPAction that scrolls in the upwards direction on a scrollable elementThe app should perform a scroll action on the specified element in the upwards direction such that the sense of the scroll is clear, either through animation or by maintaining some elements on the screen
SCROLL_DOWNAction that scrolls in the downwards direction on a scrollable elementThe app should perform a scroll action on the specified element in the downwards direction such that the sense of the scroll is clear, either through animation or by maintaining some elements on the screen
SCROLL_FORWARDAction that navigates in the forward direction on a scrollable elementThe app should perform a scroll action on the specified element in the forward direction such that the sense of the scroll is clear, either through animation or by maintaining some elements on the screen.
SCROLL_BACKWARDAction that navigates in the backward direction on a scrollable elementThe app should perform a scroll action on the specified element in the backward direction such that the sense of the scroll is clear, either through animation or by maintaining some elements on the screen.

ActionOnUIElement Directive Example

When the customer makes a request described in the Supported Utterances, the UIController interface sends a directive with the name ActionOnUIElement to your Lambda. The following is a sample directive. Note that the information you report in your UI state is used in the directive. For example, "sceneId": "sceneId-001" was reported in the UI state that the app sent to Alexa.

Examples of UI State Reports and UIController Directives

The following are examples showing sample screens, the UI state report sent, the customer’s utterance, and the UIController directive that Alexa sends to your Lambda.

Named/Ordinal Selection Example

In this example, the screen shows content in different ordinals (positions such as number 1, 2, 3, etc.). The user selects content by ordinal or by title. Screen the user sees: Example 1
Customer utterance: “Alexa, select number two” / “Alexa, select interstellar"

"Select This” Example

In this example, the user selected a title (“The Marvelous Mrs. Maisel”) to view the details. The app presents the user with the content details. The user then says “select this.” This speech pattern is referred to as “anaphora” because the user changes the referent from a title (“The Marvelous Mrs. Maisel”) to a shorter abbreviation (“this”); in both cases, the referent refers to the same item. Screen the user sees: Example 2
Customer utterance: “Alexa, select this”

Named Play Example

In this example, the user sees a list of titles on the screen and selects one of the titles by name. Because the user asks to play the title, a RemoteVideoPlayer: SearchAndPlay directive is sent rather than a UIController directive. Screen the user sees: Example 3
Customer utterance: “Alexa, play Interstellar"

"Play This” Example

In this example, the user previously asked to see more detail about the title “The Marvelous Mrs. Maisel,” so the app showed the content details page. Now the user says “Play this.” As with the previous “Select this” example, this is also an example of anaphora. Except in this case, the directive sent is a RemoteVideoPlayer: SearchAndPlay directive rather than a UIController directive. Screen the user sees: Example 4
Customer utterance: “Alexa, play this”

On-screen Selection Example

Some applications can read the on-screen content by either scraping the UI or through accessibility frameworks. These applications can read the on screen content text and provide simple on screen selection functionalities without fully understanding content being rendered These applications can also use the send state UI reports to the Alexa gateway endpoint to report the on-screen information to Alexa. The UIController interface includes a generic entity type AMAZON.Thing that can be used by screen-reader applications to report the on screen text data in the same format as some of the more advanced applications would use. The following is a simple UI state report and interaction. Screen the user sees: Example 5
Customer utterance: “Alexa, select captain fantastic”

Sample App Integration

You can find a sample integration of UIController in the cloudside sample app. The sample app includes Lambda code that shows how to declare support with the Discover directive for UIController and how to handle the ActionOnUIElement directives that UIController sends. The following sections contain detailed use cases for UI Controller APIs from the sample app. You do not need to implement any of the classes or methods to test the sample app. The following code snippets show you where to implement, and how to use, required methods. You need to use your own logic in your application code. To test with the sample app, change the UI content of any screen and test voice targeting the new UI contents with Alexa.

MainFragment Class

The MainFragment class is associated with the sample app’s home screen. The sample app reports what’s on this page and what element is currently in focus.
To see what’s in focus and the current UI state reported to Alexa, run an adb (Android debug bridge) command in your terminal adb logcat | grep MainFragment -i The sample apps reports the UI State for this screen to Alexa whenever the customer makes a selection on this screen.

VideoDetailsFragment

In the sample app, the VideoDetailsFragment class is associated with a specific Movie object. This class provides all the Movie details like a trailer, purchase options, and movie description.
To see what’s in focus and the current UI state reported to Alexa, run an adb (Android debug bridge) command in your terminal. adb logcat | grep VideoDetailsFragment -i The sample apps reports the UI State for this screen to Alexa whenever the customer makes a selection on this screen.

UI State Report

The sample app uses several classes to generate the UI state event payload:
  • Element
  • ElementWithChildren
  • Entity
  • EntityType
  • EntityName
  • Scene
  • UIAction
  • FocusedUIElement
  • UIElement
You can use these classes and the methods within them to generate the UI state payload with your logic. For example, the Element class defines each UI element on a screen. The sample app defines this behavior in the following code snippet:

UIElementUtil Class

The UIElementUtil class is a utility class that builds all the UI elements required for generating UI state event payloads.

onMessage method

The sample app receives the directives from Alexa through Lambda. The directive is called ActionOnUIElement, in the onMessage method.

MediaDetailsNavigator

If you’ve implemented UIController, consider also implementing MediaDetailsNavigator, as this interface builds on the same UI state report you’re already sending to also include support for utterances such as “Show more details about <video title>”.
Last modified on March 29, 2021