Use the Smart Screen Sample Applications

With the release of the Alexa Voice Service (AVS) Device SDK 3.0, Amazon provides two companion products for enabling screen-based Alexa Built-in device makers to complete their smart-screen Alexa integration:

  1. The Inter-Process Communication (IPC) Server Sample Application.
  2. The Alexa Smart Screen Web Components Node.js framework and Sample Web Application available on a separate Github release.

Together, through communication by using the IPC Client Framework, these applications provide a complete implementation for enabling Alexa smart-screen features on your device.

The IPC Server Sample App communicates with the Alexa Smart Screen Sample Web App through the IPC Client Framework Messages.
Communication between the IPC Server Sample App and the Smart Screen Sample Web App

Configure and run the Sample Apps

After completing the required quick start guides for building both the IPC Server Sample Application and Smart Screen Sample Web Application, you’re ready to configure and run each application. You configure the IPC Server Sample App through the two configuration files discussed in IPCServerSampleAppConfig. You then configure the Smart Screen Sample Web App using its own Sample Web App configuration.

Used together, these configurations enable you to implement a wide range of device display variations with distinct interaction modalities. For more information about the full collection of Sample IPC Server and Web App client configs for different device implementations, see Sample IPC Server configurations.

Use the following command to run the IPC Server Sample App with one of the provided samples.

  cd $HOME/sdk-folder/ipc-server-app-build/src
  ./IPCServerSampleApp \
  -C $HOME/sdk-folder/sdk-build/Integration/AlexaClientSDKConfig.json \
  -C $HOME/sdk-folder/src/avs-device-sdk/SampleApplications/IPCServerSampleApplication/config/IPCServerSampleAppConfig.json \
  -C $HOME/sdk-folder/src/avs-device-sdk/SampleApplications/IPCServerSampleApplication/config/samples/<SAMPLE_CONFIG>.json  \
  -L INFO

Then in your Chromium-based browser, run the Sample Web App.

  cd <path-to-web-components-repo-clone>/alexa-smart-screen-web-components

  open ./samples/alexa-smart-screen-sample-app/dist/index.html

Authorize with Login With Amazon (LWA)

Before you can use the applications to interact with Alexa, you must first complete authorization through LWA. This is facilitated by the implementation of the completeAuthorization IPC API in both applications.

After both applications are running and connected, complete LWA authorization following the instructions in the Sample Web App user interface in your browser. Simply select and copy the provided code, and then click the link to open the authorization flow in a separate tab.

AVS SDK Smart Screen Sample App Authorization

After authorization is complete, the Sample Web App receives the setAuthorizationState IPC message from the IPC Server App with the state REFRESHED, triggering a transition to the Sample Web App home screen.

AVS SDK Smart Screen Sample App Ready to Use

Test the Sample Apps

When using the default IPCServerSampleAppConfig, you see a Sample App home screen for a TV-type device that implements windows for both fullscreen and lower-third TV overlay Alexa visual responses. The home screen also provides instructions about the input required to start an Alexa interaction, through the use of a press and hold key, representative of using a microphone button on a TV remote control.

  1. With the Web App focused, press and hold the A key on your keyboard to open the microphone and begin talking to Alexa.

    Notice the text-based attention system in the lower-right of the screen transition from IDLE to LISTENING while holding the A key.

  2. Next say "Tell me a joke".

    The attention system text changes from LISTENING to THINKING, then Alexa responds with a joke and a fullscreen visual card.

    AVS SDK Smart Screen Sample App Ready to Use
  3. Now press and hold the A key and say "What’s the weather?".

    Alexa responds with the weather report for your local area, and a lower-third overlay card as supported by this device configuration.

    AVS SDK Smart Screen Sample App Ready to Use

Interactions

The Smart Screen Sample Web App allows for a variety of interactions for controlling the Alexa experience on your device integration.

Speech input

The Sample Web App config offers control of the input type and device key that are used to initiate Alexa interactions through the audioInputInitiator property of the Sample Web App config and the talkKey property of the deviceKeys section. These configurations determine the key press that's used to send the associated recognizeSpeechRequest IPC message to the Server to start speech recognition.

All speech input requires that the device you’re using to run the IPC Server Sample App has a connected microphone.

Wake word

If your build of the IPC Server Sample App uses a wake word engine, no input from the Web App is required to start an Alexa interaction. Just start by speaking the "Alexa" wake word. You can set the audioInputInitiator property of the Sample Web App config to WAKEWORD to inform the Web App that you're using a wake word engine, which updates the Sample home screen accordingly.

Press and hold to talk

The default IPCServerSampleAppConfig uses the PRESS_AND_HOLD audio input type, which means you must hold the input button/key for the duration of your speech interaction, and release when ready for Alexa to respond. By default, this input key is the A key.

For example, with the Sample Web App open and in focus:

  • You (Press and hold A): "What's the weather like in Portland?" (Release A)
  • Alexa: "Right now in Portland, it's 77 degrees with sun…", and a weather visual card is shown.
AVS SDK Smart Screen Sample App Ready to Use

For multi-turn interactions, the PRESS_AND_HOLD audio input type prompts you again to press and hold the input key to answer any follow-up questions Alexa might have.

Tap-to-talk

The Smart Screen Web App also supports the TAP audio input type, which is where a single button press opens the microphone for a speech interaction with Alexa, similar to saying the wake word. Again, by default this key is the A key.

The following example uses the Hub Large Landscape Sample configuration to show this interaction type. With the Web App open and in focus:

  • You (Tap A): "What's the population of Seattle?"
  • Alexa: "The population of Seattle is…", and an accompanying information card is shown.
AVS SDK Smart Screen Sample App Ready to Use

For multi-turn interactions, the TAP audio input type automatically reopens the microphone for you to answer any follow-up questions Alexa might have.

In addition to speech interactions, the Smart Screen Sample Apps also provide control of basic navigation of Alexa experiences on your device through the IPC Client Framework’s navigationEvent API. These interactions represent the types of input expected from devices, such as remote control input on a TV, or system graphic user interface (GUI) buttons on a touch display.

Currently Amazon supports two navigation events: Back and Exit.

Back

The BACK event enables you to use device button input to navigate the back stack of any Alexa experiences initiated by the applications.

All the sample IPC Server configs use the B key to initiate back navigation.

For example, with the Sample Web App open and in focus:

  • You (Press and hold A): "Play some music." (Release A)
  • Alexa: "Here’s some music you like…". Music is played, and the music card is shown.
  • You (Press and hold A): "What's the weather like in Seattle?" (Release A)
  • Alexa: "Right now in Seattle, it's 74 degrees with sun…", and a weather visual card is shown over the music card while music playback is attenuated.
  • You (Press B once) The weather card is cleared and the application returns to music card with music playback at full volume.
  • You (Press B again) The music card is cleared and music playback is terminated.
Logo Logo Logo

Exit

The EXIT event enables you to use device button input to exit ALL Alexa experiences initiated by the applications.

All the sample IPCServer configs use the ESC key to initiate exit navigation.

For example, with the Sample Web App open and in focus:

  • You (Press and hold A): "Play some music." (Release A)
  • Alexa: "Here’s some music you like…". Music is played, and the music card is shown.
  • You (Press and hold A): "What's the weather like in Seattle?" (Release A)
  • Alexa: "Right now in Seattle, it's 74 degrees with sun…", and a weather visual card is shown over the music card while music playback is attenuated.
  • You (Press ESC): Both the weather and music card are cleared, and all speech and music playback are terminated.
Logo Logo Logo

Captions and DoNotDisturb

The Smart Screen Sample Web App also allows for control of Alexa Captions and DoNotDisturb states through the Web App.

Captions

The AlexaCaptions IPC API is implemented in both applications to allow integrations that have been built with captions enabled to toggle and render Alexa captions for supported responses.

All the sample IPCServer configs use the C key to toggle captions enablement through the captionsStateChanged IPC message, and the Sample Web App uses the alexa-smart-screen-sample-captions component to render captions received through the renderCaptions IPC message.

For example, using the Hub Large Landscape config with the Sample Web App open and in focus:

  • You (Press C) This toggles the captions enabled state to ON in the Alexa client.
  • You (Press and hold A): "What's the weather like in Portland?" (Release A)
  • Alexa: "Right now in Portland, it's 77 degrees with sun…". A weather visual card is shown and captions are rendered over it. Captions are cleared when speech finishes.

    AVS SDK Smart Screen Sample App Ready to Use

DoNotDisturb

The DoNotDisturb IPC API is implemented in both applications to allow for toggling the DoNotDisturb state, and informing the Web App when it has changed.

All the sample IPCServer configs use the D key to toggle DoNotDisturb enablement in the Alexa client through the doNotDisturbStateChanged IPC message.

The Sample Web App is also notified of DoNotDisturb state changes in the Alexa client that are the result of a user voice interaction or an update from the Alexa Mobile App.

For example, with the Sample Web App open and in focus:

  • You (Press D) This toggles the DoNotDisturb enabled state to ON in the Alexa client, and the Sample Web App’s text attention system flashes to show DO NOT DISTURB ENABLED.
  • You (Press D again) This toggles the DoNotDisturb enabled state to OFF in the Alexa client, and the Sample Web App’s text attention system flashes to show DO NOT DISTURB DISABLED.
Logo Logo

Use display windows

The Smart Screen sample applications provide you with the ability to configure and implement a variety of different display and window implementations for rendering Alexa visual experiences on your device. Together, the applications do this through three key mechanisms:

  1. Assertion of visual characteristics to the Alexa service through the IPC Server Sample Application config.
    • Device Display: physical device display characteristics.
    • Window Templates: definitions of windows that can be used on the device used for rendering Alexa visual content.
    • Interaction Modes: definition of the types of interaction modalities applied to windows and any content rendered within them.
  2. Dynamic configuration of windows in the Sample Web App from the asserted visual characteristics defined by the IPC Server Sample App config.
  3. Coordinated WindowState reporting to ensure that the Alexa service is correctly informed of windows instances created in the Sample Web Application which have been registered with PresentationOrchestrator.
    • This allows responses from Alexa Skills and Amazon sources to target the available windows on your device with their content, such as when sending APL RenderDocument directives to your device.

Following are some examples using these mechanisms to achieve different window configurations on your device.

TV overlay window

If you’re looking to implement Alexa visual experiences on your TV-type devices that are similar to those on FireTV devices, you might want to consider the TV overlay window, which is implemented in the default IPCServerSampleAppConfig for the IPC Server Sample App.

With this window configuration implementation, certain shorter form Alexa visual responses, like weather and general information requests, for example, "Who is Abraham Lincoln?", present in a transient overlay window on the lower third of your screen, while longer form Alexa visual responses, like music playback, present fullscreen.

Configuration

The sample TV Fullscreen And Overlay configuration provides you an example implementation for a TV-type device with both a fullscreen and lower third overlay window for Alexa visual content.

Visual characteristics window config Sample Web App windows config
Note that
  • the definition of two interactionModes, each with the uiMode of TV, which is a requirement for ensuring receipt of overlay window content for TV.
  • the definition of two window templates, one with type STANDARD and the other with type OVERLAY.
Note that
  • the definition of two windows in the array, one using the tvFullscreen window template, and the other using the tvOverlayLandscape defined in the visual characteristics.
  • the assignment of the corresponding tv and tv_overlay interactionModes defined in visual characteristics to the fullscreen and overlay display window definitions respectively.
  • the overlay window is given a higher zOrderIndex than the fullscreen window.
  • the windowPosition is set to bottom for the overlay window.
{
  "type": "AlexaInterface",
  "interface": "Alexa.InteractionMode",
  "version": "1.1",
  "configurations": {
      "interactionModes": [
          {
              "id": "tv",
              "uiMode": "TV",            
              "interactionDistance": {
                  "unit": "INCHES",
                  "value": 130
              },
              "touch": "UNSUPPORTED",
              "keyboard": "SUPPORTED",
              "video": "SUPPORTED",
              "dialog": "SUPPORTED"
          },
          {
              "id": "tv_overlay",
               "uiMode": "TV",              
             "interactionDistance": {
                  "unit": "INCHES",
                  "value": 130
              },
              "touch": "UNSUPPORTED",
              "keyboard": "SUPPORTED",
              "video": "UNSUPPORTED",
              "dialog": "SUPPORTED"
          }
      ]
  }
},
{
    "type": "AlexaInterface",
    "interface": "Alexa.Display.Window",
    "version": "1.0",
    "configurations": {
        "templates": [
        {
            "id": "tvFullscreen",
            "type": "STANDARD",
            "configuration": {
                "sizes": [{
                    "type": "DISCRETE",
                    "id": "fullscreen",
                    "value": {
                        "unit": "PIXEL",
                        "value": {
                            "width": 1920,
                            "height": 1080
                        }
                    }
                }],
                "interactionModes": [
                    "tv"
                ]
            }
        },
        {
          "id": "tvOverlayLandscape",
          "type": "OVERLAY",
          "configuration": {
                "sizes": [{
                    "type": "DISCRETE",
                    "id": "landscapePanel",
                    "value": {
                        "unit": "PIXEL",
                        "value": {
                            "width": 1920,
                            "height": 400
                        }
                    }
                }],
                "interactionModes": [
                    "tv_overlay"
                ]
            }
        }]
    }
}
 
[{
        "id": "tvFullscreen",
        "displayWindowConfig": {
            "templateId": "tvFullscreen",
            "configurations": {
                "landscape": {
                    "sizeConfigurationId": "fullscreen",
                    "interactionMode": "tv"
                }
            }
        },
        "aplRendererParameters": {
            "supportedExtensions": [
                "aplext:backstack:10"
            ]
        },
        "supportedInterfaces": [
            "Alexa.Presentation.APL",
            "TemplateRuntime"
        ],
        "zOrderIndex": 0
    },
    {
        "id": "tvOverlayLandscape",
        "displayWindowConfig": {
            "templateId": "tvOverlayLandscape",
            "configurations": {
                "landscape": {
                    "sizeConfigurationId": "landscapePanel",
                    "interactionMode": "tv_overlay"
                }
            }
        },
        "supportedInterfaces": [
            "Alexa.Presentation.APL",
            "TemplateRuntime"
        ],
        "windowPosition": "bottom",
        "zOrderIndex": 1
    }
]
 

Example usage

With the previous configurations applied, and the Sample Web App open and in focus:

  • You (Press and hold A): "Play some music." (Release A)
  • Alexa: "Here’s some music you like…", and music is played, and the music card is shown in the fullscreen window.
  • You (Press and hold A): "What's the weather like in Seattle?" (Release A)
  • Alexa: "Right now in Seattle, it's 74 degrees with sun…", and a weather card is shown in the overlay window above the music card.
Logo Logo

Re-orienting windows

If you plan to implement a device that may change orientation from landscape to portrait or vice versa, you likely want to implement Alexa visual windows that can respond to these changes in orientation to present the best content to your users for your device’s current orientation.

The Smart Screen Sample Apps support these types of windows, and any dynamic content like APL, that may adapt visual presentation based on orientation, through the use of the following systems:

  1. The three key mechanisms discussed in Use display windows for asserting and configuring windows.
  2. The displayMode ORIENTABLE property of the Sample Web App config.
  3. The optional display window orientation configuration of the Sample Web App config.
  4. The Sample Web App’s implementation of the alexa-smart-screen-window-manager component, and its corresponding API windowManager.updateDisplayOrientationToWindows(DisplayOrientation.PORTRAIT).

Configuration

The sample Hub Orientable config provides you an example implementation of a re-orientable touch display device with similar characteristics to an Echo Show 15.

Visual characteristics window config Sample Web App windows config
Note that
  • the definition of two DISCRETE size configurations for the single hubFullscreen window template: one with landscape dimensions, and the other portrait.
Note that
  • the displayMode property is set to ORIENTABLE.
  • the implementation of both the landscape and portrait definition in the displayWindowConfig , each mapping to the corresponding fullscreenLandscape and fullscreenPortrait size configuration from the window template assertion.
{
  "type": "AlexaInterface",
  "interface": "Alexa.Display.Window",
  "version": "1.0",
  "configurations": {
    "templates": [
      {
        "id": "hubFullscreen",
        "type": "STANDARD",
        "configuration": {
          "sizes": [
            {
              "type": "DISCRETE",
              "id": "fullscreenLandscape",
              "value": {
                "unit": "PIXEL",
                "value": {
                  "width": 1920,
                  "height": 1080
                }
              }
            },
            {
              "type": "DISCRETE",
              "id": "fullscreenPortrait",
              "value": {
                "unit": "PIXEL",
                "value": {
                  "width": 1080,
                  "height": 1920
                }
              }
            }
          ],
          "interactionModes": [
            "hub_fullscreen"
          ]
        }
      }
    ]
  }
}

 
"displayMode": "ORIENTABLE",
"windows": [
  {
    "id": "hubOrientableFullscreen",
    "displayWindowConfig": {
      "templateId": "hubFullscreen",
      "configurations": {
        "landscape" : {
          "sizeConfigurationId": "fullscreenLandscape",
          "interactionMode": "hub_fullscreen"
        },
        "portrait" : {
          "sizeConfigurationId": "fullscreenPortrait",
          "interactionMode": "hub_fullscreen"
        }
      }
    },
    "aplRendererParameters": {
      "supportedExtensions": [
        "aplext:backstack:10"
      ]
    },
    "supportedInterfaces": [
      "Alexa.Presentation.APL",
      "TemplateRuntime"
    ],
    "zOrderIndex": 0
  }
]
 

Example usage

With the Hub Orientable configuration applied, and the Sample Web app open and in focus with a portrait orientation:

  • You (Press and hold A): "What's the weather like in Seattle?" "" (Release A)
  • Alexa: "Right now in Seattle, it's 76 degrees with sun…", and a weather card is shown fullscreen for portrait orientation.
  • You: Adjust your browser window size to reflect a landscape orientation. The weather card dynamically updates and re-renders to the landscape dimension.
Hub Orientable 1 Hub Orientable 2

Resizable windows

If your device implementation of an Alexa client requires windows that may dynamically re-size based on user input or other changing device modalities, you may want to implement resizable windows in the Smart Screen Web App.

The Smart Screen Sample Applications support these types of windows, and any dynamic content like APL, that may adapt visual presentation based on size changes, through the use of the following systems:

  1. The three key mechanisms discussed in Use display windows for asserting and configuring windows.
  2. The CONTINUOUS WindowTemplate size configuration type.
  3. The displayMode RESIZABLE property of the Sample Web App config.
  4. The Sample Web App’s implementation of the alexa-smart-screen-Window-manager component, and its corresponding API windowManager.updateDisplaySizeToWindows(width, height).

Configuration

The sample Resizable Desktop App config provides you an example implementation of an Alexa client, such as one might find in a desktop application, for which the user can change window size within a defined range.

Visual characteristic window config Sample Web App windows config
Note that
  • the definition of a single CONTINUOUS size configuration for the pcFullscreen window template, with both a minimum and maximum dimension range defined.
Note that
  • the displayMode property is set to RESIZABLE.
  • the single implementation of the landscape configuration using the pcFullscreenContinuous size configuration. You can use either landscape or portrait as the base size configuration for the resizable window.
{
  "type": "AlexaInterface",
  "interface": "Alexa.Display.Window",
  "version": "1.0",
  "configurations": {
    "templates": [
      {
        "id": "pcFullscreen",
        "type": "STANDARD",
        "configuration": {
          "sizes": [
            {
              "type": "CONTINUOUS",
              "id": "pcFullscreenContinuous",
              "minimum": {
                "unit": "PIXEL",
                "value": {
                  "width": 960,
                  "height": 480
                }
              },
              "maximum": {
                "unit": "PIXEL",
                "value": {
                  "width": 1920,
                  "height": 1280
                }
              }
            }
          ],
          "interactionModes": [
            "pc_fullscreen"
          ]
        }
      }
    ]
  }
}

 
"displayMode": "RESIZABLE",
"windows": [
  {
    "id": "pcFullscreen",
    "displayWindowConfig": {
      "templateId": "pcFullscreen",
      "configurations": {
        "landscape" : {
          "sizeConfigurationId": "pcFullscreenContinuous",
          "interactionMode": "pc_fullscreen"
        }
      }
    },
    "aplRendererParameters": {
      "supportedExtensions": [
        "aplext:backstack:10"
      ]
    },
    "supportedInterfaces": [
      "Alexa.Presentation.APL",
      "TemplateRuntime"
    ],
    "zOrderIndex": 0
  }
]
 

Example usage

With the Resizable Desktop App config applied, and the Sample Web App open and in focus:

  • You (Press and hold A): "Where is the Golden Bridge?" (Release A)
  • Alexa: "The Golden Gate bridge is…", and an informational card is shown for the current size.
  • You: Adjust your browser window size to scale the window vertically. The informational card dynamically updates and re-renders to the current size.
Resizable Desktop 1 Resizable Desktop 2

Debug mode

You can run the IPC Server Sample App in debug mode to troubleshoot your device and run diagnostic tools. If the IPC Server Sample App is running in debug mode, you see the following message when it first starts.

SDK Version 3.0.0

WARNING! THIS DEVICE HAS BEEN COMPILED IN DEBUG MODE.

RELEASING A PRODUCTION DEVICE IN DEBUG MODE MAY IMPACT DEVICE PERFORMANCE,
DOES NOT COMPLY WITH THE AVS SECURITY REQUIREMENTS,AND COULD RESULT IN SUSPENSION OR TERMINATION OF THE ALEXA SERVICE ON YOUR DEVICES.

Troubleshooting

  • Can't hear sound: If you don't hear any sound, make sure that both your microphone and speakers are both working.
  • Smart Screen Applications are unresponsive: If Alexa isn't responding or the IPC Server Sample App appears stuck, or displaying error messages when you try to speak, try refreshing your browser widow with the Sample Web App to reload that application and re-connect to the IPC Server.

For more details on how to troubleshoot other common issues, see Troubleshooting AVS Device SDK Common Issues.