UIController interface takes into account your app’s UI state in order to more intelligently respond to user requests. To use the UIController interface, you need to send Alexa your UI state, listing all elements on the page and the elements in focus. These details are then included in the directives that Alexa sends to your Lambda.
If you’re implementing UIController, consider also implementing MediaDetailsNavigator, which is another Alexa interface that uses the UI state report to provide directives for “show more details about <video title>” requests.
Detailed Explanation
For Alexa to more intelligently respond to user’s requests, Alexa needs to know what elements the customer sees on the screen and what actions are available for each element. This information is known as the UI state. A UI state has two different dimensions to describe what’s shown on the UI — the list of elements shown on the screen (which is reported in theuiElements section of the state report), and the element in focus (which is reported in the focusedUIElement section).
To let Alexa know your UI state, you (as the app developer) first send a UI report to Amazon (posting the event as a ChangeReport to a specific Alexa gateway endpoint) so that Alexa can dynamically target different screen elements. For example, after launching your app, your app would send a structured list of UI elements on screen (including names, aliases, position, and supported actions for those objects). You send your UI state to the Alexa endpoint whenever your UI changes.
When customers make requests related to on-screen UI navigation or selection (e.g., “Alexa, play number three,” “Alexa, select continue watching,” “Alexa, play harry potter,” “Alexa, play this” ), Alexa sends an ActionOnUIElement directive from the UIController interface to your Lambda. The directive contains specific details from the UI state that you previously sent to Alexa. For example, if you have a title on the screen called “Popcorn Movie” in the second position in a list, the UIController interface sends a directive with those details to your Lambda.
Overall, the UIController interface lets customers interact in a richer way with voice in your app’s UI, such as selecting their profile by voice or selecting a specific item on the screen by voice.
UIController versus KeypadController
UIController differs from KeypadController in that UIController considers your UI state and includes details from the UI state in the directive it sends. In contrast, KeypadController is stateless and doesn’t receive any specific details about your app’s UI. As a result, UIController is a lot more powerful in the types of utterances it can support.
You can use both KeypadController and UIController together. UIController will take precedence over KeypadController whenever the UI state is available. If the UI state is not available, KeypadController will be used as a fallback. If you don’t have KeypadController implemented, the fallback would be the in-app voice scrolling and selection functionality that is built-in to Fire TV.
Supported Utterances
Alexa sends anActionOnUIElement directive to your Lambda when users make a navigation or selection request related to an element on the screen. Unlike the other utterances, because UIController has access to your UI state, there aren’t a rigid set of supported utterances. The supported utterances and features can be dynamically controlled based on the on-screen context you provide in your UI state report. The utterances your app supports depend on the elements you provide in your UI state reporting.
For example, if you describe an element called “Jake’s watchlist” as an ordinal in the third position, then Alexa would support utterances such as “Go to Jake’s watchlist” or “select the third one.” (Ordinals are identifiers that indicate position, such as number one, number two, etc.)
Depending on the detail in your state reporting, the following utterances and scenarios can be supported:
- You can scroll a list by saying, “Scroll/Page right/left/up/down.” (Note: “Scroll back” is not supported.)
- You can see more/all the titles that are not visible in the shortlist by saying “Show/See more.” (Note: “Show/see all” is not supported.)
- You can select an item from the list by saying “Select/Pick/Choose this/that/the first one/number one,” or ask Alexa to play a title from the list, like “Select/Play/Choose/Watch Mockingjay.”
- You can also use your voice to emulate a d-pad controller on your remote by saying “Go right/left/up/down” followed by “Select/Choose/Watch/Play.”
- On a video detail page, you can take the following actions showing on the screen: “Add to watchlist,” “Watch Now,” “Watch Trailer,” etc.
- You can navigate to any UI location, such as recently watched, watch lists, recommended videos.
- You can respond to a dialog screen and pick an option without needing to reach for the remote control.
KeypadController and RemoteVideoPlayer: SearchAndPlay.
Sending UI State Reports
You need to send a UI state report to inform Alexa about the currently visible items on the customer’s screen.VSK-Enabled Apps
For VSK-enabled apps, you send the UI state report via the Alexa Client Library. The UI Controller requires Alexa Client Library version 1.4.9 and later. See Step 3: Integrate the Alexa Client Library for information about downloading and installing the Alexa Client Library.Step 1 - Add the UI Controller to Alexa Client Library Supported Capabilities
Before you can send UI state reports, you need to add support for the UI Controller in the Alexa Client Library. In theinitializeAlexaClient method, add getAlexaUIControllerCapability to the list of supportedCapabilities.
Step 2 - Send UI Reports through Alexa Client Library
To send your UI state report, call thesetUIState method in the AlexaClientManager class of the Alexa Client Library.
setUIState method takes a states parameter. The states parameter is a map of user interface properties representing elements displayed on screen. A key string, representing the property name to set (i.e. uiElements, focusedUIElement), is matched with a value string, the JSON encoded contents of the value key associated with this property.
VSK-Enabled Devices
For third party devices, you send the UI state report as a POST request to an Alexa event gateway endpoint every time there is a actionable change with your UI State. The steps to send UI state report events are listed.Step 1 - Request Permission to Send Events
UI state report events are sent to the Alexa event gateway, which requires an authentication token for each customer. In the Alexa console, find your skill and, select the PERMISSIONS tab on the left, and turn on Send Alexa Events. Record your Alexa Client Id and Alexa Client Secret (shown on this screen), which will be used in the next step to authenticate the customer with Login with Amazon (LwA). For more information, see Configure Permissions for Customer Information in Your Skill in the Alexa documentation. (State reporting is common with the Alexa Smart Home APIs.)Step 2 - Add Code to Handle AcceptGrant Directives
When you request permission to send Alexa events, your skill’s Lambda function will receive an AcceptGrant directive for each customer. The AcceptGrant directive contains an authorization code that you will use to obtain an authentication token for the customer. You must add code to obtain and store the tokens in the same region where your customer is located. Here’s an example AcceptGrant directive:
| Field | Description | Type |
|---|---|---|
grant | Information that identifies a user in Amazon Alexa systems. | Object |
grant.type | The type of grant. Currently, the only valid value is OAuth2.AuthorizationCode. | String |
grant.code | An authorization code for the user. | String |
grantee | Information that identifies a user in a linked account service or system. | Object |
grantee.type | The type of grantee. Currently, the only valid value is BearerToken. | String |
grantee.token | The user access token received by Alexa during the account linking process. | String |
AcceptGrant directive was successfully handled, respond with an AcceptGrant.Response event.
AcceptGrant response event example
AcceptGrant directive successfully, respond with an Alexa.Authorization.ErrorResponse event. Here are some reasons why the handling might be unsuccessful:
- You are unable to call Login with Amazon to exchange the authorization code for access and refresh tokens.
- You are unable to store the access and refresh tokens for the user.
- Any other error that occurs while you try to retrieve and store the access and refresh tokens.
Alexa.Authorization.ErrorResponse:
AcceptGrant processing.
See the following for more details:
Step 3 - Indicate that Properties Are Proactively Reported During Discovery
Alexa sends aDiscover directive through the Discovery Interface to understand your app’s capabilities. In addition to declaring support for the UIController interface, during discovery you must also indicate that you will send change reports. You do this by indicating the properties are proactivelyReported in your discovery response.
Here’s how you can configure your response to the Discover directive to indicate that you support the UIController capability:
Step 4 - Send UI State Reports to the Alexa Event Gateway
The final step is to begin sending UI state reports to the Alexa event gateway when a reportable property changes. When your UI state changes for any reason, send a UI state report. Each message to the event gateway should include the authentication token as an HTTP header and in the body of the message. To send your UI state to the Alexa event gateway, submit the request header as follows:<access-token-from-Amazon to the actual authentication token you obtained in Step 2.) The request body must contain a JSON object with your UI state details as described in UI State Report Example and Schema. See the “Example” tab in the next section for details.
For more details about authorization, see the following in the Alexa documentation:
UI State Report Example and Schema
The following tabs provide a UI state report example (which shows sample populated values) and a schema (which shows data types rather than examples). A tree diagram is also included to show the schema’s visual hierarchy.- Example
- Schema
- Tree
context and properties objects aren’t used with Fire TV implementations. The context object would include only updates from other Alexa components such as Alexa.PowerState. Leave this empty if there’s no state update from other components when the Change Report event is triggered.)
Data Structures
This section describes the data structures for the state report schema as previously described. These tree diagrams are similar to the earlier schema but are paired down here to show the required schema foruiElements and focusedUIElement sections of your UI state report.
Schema for uiElements versus focusedUIElements
The properties array has two main objects: one with a namespace of uiElements, and another with a namespace of focusedUIElement. Their schemas are slightly different:
uiElementslists all elements on the screen; this section uses anelementsarray (plural) and also contains a nested array similarly calledelementswhere you can provide an optional list of sub-elements.focusedUIElementlists the element in focus; this section uses anelementobject (singular).
uiElements and focusedUIElement lists.
- uiElements Schema
- focusedUIElement Schema
Payload Definitions
The following table defines the payload for the UI state reports that you send. Note that elements repeated (e.g.,uiSupportedActions) in both the uiElements and focusedUIElement sections have the same definitions.
| Field | Description | Data Type |
|---|---|---|
changerequired | Contains details about the change in state. | Object |
causerequired | Provides the reason for the change in state. | Object |
typerequired | The type of state change. One option is allowed for Fire TV implementations: AMAZON.PHYSICAL_INTERACTION. | Enum |
propertiesoptional | Contains the details about the state change. The properties element contains details for state changes relating to both uiElements (a list of elements) and focusedUIElement objects. | Array |
namespacerequired | The namespace for the payload. For Fire TV implementations, one option is allowed: Alexa.UIController. | String |
namerequired | The name of the interface for the payload. Two options are allowed: uiElement and focusedUIElement. If you’re providing a list of elements, use uiElement. In contrast, if you’re reporting about the element in focus, use focusedUIElement. See the sample UI State Report Example for details and context.Example: uiElements | String |
valueoptional | Contains a scene object that describes the elements on the customer’s screen. | Object |
scenerequired | Includes a sceneId property containing information about the scene displayed on the customer’s UI. | Object |
sceneIdrequired | ID of the scene rendered on the customer’s UI.Example: sceneId-001 | String |
elementsoptional | List of UI elements rendered on this scene. Used only with uiElements state reporting. | Array |
elementrequired | Element that has the visual focus on this scene. Used only with focusedUIElement state reporting. | List<Element> |
entityrequired | A Thing (a generic object) that describes this element. | Object |
typerequired | Type of this entity. See the Enum<EntityTypes> section for details.Example: AMAZON.ItemList | Enum<EntityTypes> |
nameoptional | Name of this entity. Contains two possible properties: value and variants. | Object |
valueoptional | Preferred name of this entity.Example: Superbad | String |
variantsoptional | Additional names used to refer this entity.Example: Extremebad | String |
externalIdsoptional | Identifiers for this entity unique within the scope of this endpoint. The entity.externalIds field will be opaque to Alexa and will only be used to include in directives from Alexa. The identifiers reported in entity.externalIds are required to be understood by the endpoint on all directive commands that accept an entity structure.Example: "mediaBrowseId": "d9fd6168-81b8-4a28-a660-ba35141ca5b6" | Map<String, String> |
elementIdrequired | Identifier of this UI element unique within the scope of the reported scene. The elementId field will be opaque to Alexa and will only be used to include in directives from Alexa to this endpoint. The endpoint can use this field to include information it needs to be able to locate this element on the rendered UI in response to a directive command.Example: elementId_11 | String |
ordinaloptional | The number of the element that can be used in spoken language selection. For example, if this value is one then “Alexa select number 1” will select this element.Example: Number one | Integer |
uiSupportedActionsrequired | A list of enumerated actions a user can take on this UI element through the UIController interface. See Enum<uiSupportedActions> for details about enum options.Example: SELECT | Enum |
elementsoptional | Optional list of nested UI elements. This element is unique to uiElements. | List<ElementWithChildren> |
timeOfSampleoptional | The time at which the property value was recorded.Example: 2018-02-03T16:20:49.52Z | ISO 8601 time |
uncertaintyInMillisecondsoptional | The number of milliseconds that have elapsed since the property value was last confirmed.Example: 0 | Integer |
Enum<EntityTypes>
Thetype enum for entity allows the following options:
-
AMAZON.VideoObject: Entity type to represent a video item rendered on a UI. This entity is an abstraction for various types of videos (e.g. movie, tv shows, trailers, etc.). Example screen:
Example: “Interstellar” from this screen
-
AMAZON.ItemList: Entity type to represent a list of items. This entity can be used to represent a collection of entities of same or different types. Example screen:
Example: “Recents” from this screen
-
AMAZON.SoftwareApplication: Entity type to represent a software application. Example screen:
Example: “IMDb TV” from this screen
-
AMAZON.Thing: Entity type to represent generic entities whose specific type is either unknown to the API integrator or undefined in theUIControllerAPI specification. (Note thatMediaDetailsNavigatordoes not support showing more details about this entity type.) Example screen:
Example: A screen reader application reporting “Captain Fantastic” from this screen
Enum<uiSupportedActions>
TheuiSupportedActions enum allows the following options.
| Name | Description | Directive Handling Expectation |
|---|---|---|
SELECT | Action that selects a UI element | The app should do an equivalent of a single click on the provided element |
EXPAND | Action that “expands” or otherwise “reveals more” on a UI element. | The app should expand the hidden, obstructed, truncated, or otherwise expandable content of the provided element. |
SCROLL_RIGHT | Action that scrolls in the right direction on a scrollable element | The app should perform a scroll action on the specified element in the right direction such that the sense of the scroll is clear, either through animation or by maintaining some elements on the screen |
SCROLL_LEFT | Action that scrolls in the left direction on a scrollable element | The app should perform a scroll action on the specified element in the left direction such that the sense of the scroll is clear, either through animation or by maintaining some elements on the screen |
SCROLL_UP | Action that scrolls in the upwards direction on a scrollable element | The app should perform a scroll action on the specified element in the upwards direction such that the sense of the scroll is clear, either through animation or by maintaining some elements on the screen |
SCROLL_DOWN | Action that scrolls in the downwards direction on a scrollable element | The app should perform a scroll action on the specified element in the downwards direction such that the sense of the scroll is clear, either through animation or by maintaining some elements on the screen |
SCROLL_FORWARD | Action that navigates in the forward direction on a scrollable element | The app should perform a scroll action on the specified element in the forward direction such that the sense of the scroll is clear, either through animation or by maintaining some elements on the screen. |
SCROLL_BACKWARD | Action that navigates in the backward direction on a scrollable element | The app should perform a scroll action on the specified element in the backward direction such that the sense of the scroll is clear, either through animation or by maintaining some elements on the screen. |
ActionOnUIElement Directive Example
When the customer makes a request described in the Supported Utterances, theUIController interface sends a directive with the name ActionOnUIElement to your Lambda. The following is a sample directive. Note that the information you report in your UI state is used in the directive. For example, "sceneId": "sceneId-001" was reported in the UI state that the app sent to Alexa.
Examples of UI State Reports and UIController Directives
The following are examples showing sample screens, the UI state report sent, the customer’s utterance, and theUIController directive that Alexa sends to your Lambda.
Named/Ordinal Selection Example
In this example, the screen shows content in different ordinals (positions such as number 1, 2, 3, etc.). The user selects content by ordinal or by title. Screen the user sees:
UI state report you send
UI state report you send
UIController directive you receive
UIController directive you receive
"Select This” Example
In this example, the user selected a title (“The Marvelous Mrs. Maisel”) to view the details. The app presents the user with the content details. The user then says “select this.” This speech pattern is referred to as “anaphora” because the user changes the referent from a title (“The Marvelous Mrs. Maisel”) to a shorter abbreviation (“this”); in both cases, the referent refers to the same item. Screen the user sees:
UI state report you send
UI state report you send
UIController directive you receive
UIController directive you receive
Named Play Example
In this example, the user sees a list of titles on the screen and selects one of the titles by name. Because the user asks to play the title, aRemoteVideoPlayer: SearchAndPlay directive is sent rather than a UIController directive.
Screen the user sees:
UI state report you send
UI state report you send
UIController directive you receive
UIController directive you receive
"Play This” Example
In this example, the user previously asked to see more detail about the title “The Marvelous Mrs. Maisel,” so the app showed the content details page. Now the user says “Play this.” As with the previous “Select this” example, this is also an example of anaphora. Except in this case, the directive sent is aRemoteVideoPlayer: SearchAndPlay directive rather than a UIController directive.
Screen the user sees:
UI state report you send
UI state report you send
SearchAndPlay directive you receive
SearchAndPlay directive you receive
On-screen Selection Example
Some applications can read the on-screen content by either scraping the UI or through accessibility frameworks. These applications can read the on screen content text and provide simple on screen selection functionalities without fully understanding content being rendered These applications can also use the send state UI reports to the Alexa gateway endpoint to report the on-screen information to Alexa. TheUIController interface includes a generic entity type AMAZON.Thing that can be used by screen-reader applications to report the on screen text data in the same format as some of the more advanced applications would use. The following is a simple UI state report and interaction.
Screen the user sees:
UI state report you send
UI state report you send
UIController directive you receive
UIController directive you receive
Sample App Integration
You can find a sample integration ofUIController in the cloudside sample app. The sample app includes Lambda code that shows how to declare support with the Discover directive for UIController and how to handle the ActionOnUIElement directives that UIController sends.
The following sections contain detailed use cases for UI Controller APIs from the sample app. You do not need to implement any of the classes or methods to test the sample app. The following code snippets show you where to implement, and how to use, required methods. You need to use your own logic in your application code.
To test with the sample app, change the UI content of any screen and test voice targeting the new UI contents with Alexa.
MainFragment Class
TheMainFragment class is associated with the sample app’s home screen. The sample app reports what’s on this page and what element is currently in focus.
adb logcat | grep MainFragment -i
The sample apps reports the UI State for this screen to Alexa whenever the customer makes a selection on this screen.
VideoDetailsFragment
In the sample app, theVideoDetailsFragment class is associated with a specific Movie object. This class provides all the Movie details like a trailer, purchase options, and movie description.
adb logcat | grep VideoDetailsFragment -i
The sample apps reports the UI State for this screen to Alexa whenever the customer makes a selection on this screen.
UI State Report
The sample app uses several classes to generate the UI state event payload:ElementElementWithChildrenEntityEntityTypeEntityNameSceneUIActionFocusedUIElementUIElement
Element class defines each UI element on a screen. The sample app defines this behavior in the following code snippet:
UIElementUtil Class
TheUIElementUtil class is a utility class that builds all the UI elements required for generating UI state event payloads.
onMessage method
The sample app receives the directives from Alexa through Lambda. The directive is calledActionOnUIElement, in the onMessage method.
MediaDetailsNavigator
If you’ve implementedUIController, consider also implementing MediaDetailsNavigator, as this interface builds on the same UI state report you’re already sending to also include support for utterances such as “Show more details about <video title>”.
