Optimizing Your Multimodal Experiences on the New Echo Show 15

Ryan Matthews Sep 29, 2021
Share:
Multimodal
Blog_Header_Post_Img

Yesterday, we announced the all-new Echo Show 15, an entirely new type of Echo Show that’s designed to be the digital heart of the home, helping busy families stay organized, connected, and entertained. Echo Show 15 has a screen that can be positioned in either portrait or landscape orientations, and can be mounted on a wall or placed on the counter. This blog covers how to update your skill to support portrait orientation with APL or the Web API for Games.

In addition, we are announcing APL 1.8 that comes with new features you can use to make your multimodal experiences even more engaging for customers. APL 1.8 is already available on Echo Show 1st Generation, Echo Show 2nd Generation, Echo Show 5, Echo Show 8. and Fire TV devices. In the coming days we will also roll it out to Echo Show 10 and Fire tablet devices. As always, you can use the supportedInterfaces object in your skill request to detect the version of APL supported on a device.

Adapting for the Echo Show 15

While the all-new Echo Show 15 is our first Echo device to support portrait mode, customers have been able to enjoy APL portrait mode experiences on Fire tablets (Fire 7, Fire HD8 and Fire HD10) since APL 1.6 which was launched in April-2021. APL supports automatic scaling, which we use to scale existing APL experiences to new viewport profiles they may not have been designed for. However, because a lot of APL experiences are designed for landscape profiles - scaling them to fit on a portrait viewport can lead to results that are not portrait friendly. Check out Matchbox’s Question of the Day below, before and after they've added portrait mode support: 

Two screenshots of Question of the Day showing a BEFORE and AFTER version of portrait mode support.

Question of the Day (click here try it!)

LEFT: portrait mode with automatic scaling

RIGHT: portrait mode with opt-out scaling and responsive templates

 

In this section we will cover how you can add support for the Echo Show 15’s portrait viewport profile (Hub Portrait Medium) or opt out of scaling entirely to create a great portrait experience on these devices. We will also show how to adapt  games that use the Web API to work well in portrait mode.

Adapting APL-based Experiences

Step 1: Decide Your Scaling Strategy

The first decision you need to make is how you want to scale your experience for the Echo Show 15. If you do nothing at all and rely on automatic scaling it will result in the kind of letterboxed experience we just saw for customers using Echo Show 15 in a portrait orientation. You have one of three options:

 

(a) Add Echo Show 15 Viewport Profiles to Your Skill’s Supported Profiles
For the best user experience, we recommend you add the new Echo Show 15 viewport profiles - Hub Landscape Extra Large (landscape) and Hub Portrait Medium (portrait) - to your skill’s list of supported viewport profiles. You can do this by going to the Alexa Developer Console, selecting the Build→ Interfaces tab, and checking the corresponding boxes for these profiles. Once you’ve done this you should also make your APL documents are responsive and test on the new profiles using the Test tab of the developer console.

Screenshot from the Alexa Developer Console showing the new Echo Show 15 viewports.

(b) Opt-Out of Automated Scaling
Starting today, you can also opt out of automated scaling entirely. This means that when new form factors are launched and we add new viewport profiles to support them APL will just assume your skill supports them. You can opt out of scaling by retrieving your skill’s manifest, adding the interfaces code snippet below, and then updating it again using the ASK CLI.

Copied to clipboard
{
    "manifest": {
      "apis": {
        "custom": {
          "endpoint": {
            "uri": "
          },
          "interfaces": [
            {
              "supportedViewports": [               
                {
                  "mode": "HUB",
                  "shape": "RECTANGLE"
                },
                {
                  "mode": "HUB",
                  "shape": "ROUND"
                },
                {
                  "mode": "TV",
                  "shape": "RECTANGLE"
                },
                {
                  "mode": "MOBILE",
                  "shape": "RECTANGLE"
                }
              ],
              "type": "ALEXA_PRESENTATION_APL"
            }
          ]
      },
      "manifestVersion": "1.0"
    }
  }

(c) Rely on Automated Scaling
This is what happens if you don’t enable the Echo Show 15 viewports or opt out of scaling entirely. Your APL document will be scaled up or down to fit the available screen size. Although the scaled content works, there is no guarantee that the end result will look the way you intended. While we don’t recommend this option if you do choose to go with it, we still recommend testing your content on the Echo Show 15 portrait viewport profile (Hub Portrait Medium) to ensure that your content is at least visible on the device. 

 

Step 2: Adapt Your APL for Portrait Orientation

Next, you will want to adapt your APL document(s) to make sure they work well in portrait mode. There are two ways of doing this:

(a) Use a Responsive Template
APL provides a set of responsive templates you can use in your Alexa skills. These layouts automatically work on viewports with different modes, sizes, and shapes (including portrait). The sample below uses the AlexaImageList template. Notice how the content makes the best use of the available screen real estate:

Portrait and landscape versions of the same content, illustrating how each makes the best use of the available real estate.

To begin using a responsive template, go to the APL authoring tool, select “Create Visual Response”, and choose a template from the list provided.

(b) Adapt Your Custom APL Document

We understand that responsive templates can’t meet every need and many developers create their own custom layouts.  To adapt your custom layout, we added the Hub Portrait Medium viewport profile for use on the Echo Show 15 in portrait mode. In addition, there’s the Hub Landscape Extra Large viewport profile in case you want to optimize your APL document for Echo Show 15 in landscape mode. To use these profiles, import the alexa-viewport-profiles package in to your APL document as shown below.

Copied to clipboard
{
   "import": [
    {
      "name": "alexa-viewport-profiles",
      "version": "1.3.0"
    }
  ]
}

Now you can use the viewport profile in when clauses and other expressions in your APL document. To use a resource from the package in your document’s conditional logic, use the reserved character "@" in the resource name. In the following example, this when clause evaluates to true if the customer’s device meets the criteria for the hubPortraitMedium profile:

Copied to clipboard
"when": "${@viewportProfile == @hubPortraitMedium}"

You don’t have to use a specific viewport profile if you want to adapt for portrait mode on all devices (including tablets). The resource @viewportOrientation can be used to tell the orientation of the current viewport. In the sample below, the container layout is changed from left to right to top to bottom depending on the device orientation.

Copied to clipboard
{
  "type": "APL",
  "version": "1.8",
   "onConfigChange": {
     "type": "Reinflate"
   },
  "import": [
    {
      "name": "alexa-styles",
      "version": "1.3.0"
    },
    {
      "name": "alexa-viewport-profiles",
      "version": "1.3.0"
    }
  ],
  "mainTemplate": {
    "parameters": [
      "exampleData"
    ],
    "item": {
      "type": "Container",
      "direction": "${@viewportOrientation == @viewportOrientationLandscape ? 'row' : 'column'}",
      "padding": "@marginHorizontal, @spacingMedium",
      "items": [
        {
          "type": "Image",
          "scale": "best-fit",
          "width": "${@viewportOrientation == @viewportOrientationLandscape ? 0 : '100%'}",
          "height": "${@viewportOrientation == @viewportOrientationLandscape ? '100%' : 0}",
          "grow": 0.5,
          "source": "${exampleData.image}"
        },
        {
          "type": "ScrollView",
          "width": "${@viewportOrientation == @viewportOrientationLandscape ? 0 : '100%'}",
          "height": "${@viewportOrientation == @viewportOrientationLandscape ? '100%' : 0}",
          "grow": 0.5,
          "spacing": "@spacingMedium",
          "item": {
            "type": "Text",
            "fontSize": 20,
            "text": "${exampleData.text}"
          }
        }
      ]
    }
  },
  "settings": {
    "supportsResizing": true
  }
}

Here’s the rendered visuals. When in portrait mode, it’s restructured with improved readability and a better use of the screen real estate:

Screenshots showing a portrait and landscape-optimized view of two pieces of content.

 

Step 3: Enable Dynamic Resizing (Optional)

On devices like tablets where customers can reorient devices while your skill is running you can use the supportsResizing setting to declare that your APL document can be resized. This is also true on Echo Show 15 although the large size of the device means that customers are less likely to reorient it as often or in the middle of a skill session. That said, supportsResizing enables APL documents to rotate on devices that support reorientation. Here’s an example of the supportsResizing setting in an APL document.

Copied to clipboard
{
  "type": "APL",
  "version": "1.8",
  "settings": {
    "supportsResizing": true
  },
  "mainTemplate": {
     ...
  }
}

APL documents that are built responsively should resize automatically and look good when the orientation of a device changes. However, we know it isn’t always possible to only use relative dimensions. This is why we added the new Reinflate command with APL 1.6 earlier this year. Put simply, this command asks the APL runtime to redraw the entire APL document and update all the data bindings. You can choose to preserve specific properties such as scroll offsets of scrollview components so customers don’t lose their place in the interaction.

 

See this walkthrough to learn how to use preserved properties and Reinflate to support devices that can change orientation. Here’s an example of an APL document that supports resizing and reinflates itself when device configuration (e.g., orientation) changes.

Copied to clipboard
{
  "type": "APL",
  "version": "1.6",
  "settings": {
    "supportsResizing": true
  },
  "onConfigChange": {
    "type": "Reinflate"
  },
  "mainTemplate": {
     ...
  }
}

 

Step 4: Preview the Portrait Experience in the Authoring Tool

As you’re updating your APL responses, frequently test your new layouts in the APL authoring tool in the Alexa Developer Console. It’s been updated to simulate the HubLandscapeExtraLarge and HubPortraitMedium viewport profiles so you can see what your APL documents will look like on an Echo Show 15 in landscape and portrait modes respectively. The Test tab in the Alexa developer console also supports these new viewport profiles now.

 

Step 5: Add Rich Audio and Load Images/Video Predictably (Optional)

For skills that use video, images, and/or vector graphics we’re making it easier to tell when something goes wrong with those resources and respond to it. 

(a) Fail Gracefully Using New Image, Video, and Vector Graphic Events

Image and VectorGraphic components now have onLoad/onFail events and Video components now support onTrackReady, onTrackFail, and onTrackUpdate events. These events allow APL commands to be run when any sources load, fails to load, or change status. For example, if an image fails to load because the URL is invalid, the component invokes the new onFail handler which could be configured to run SetValue to change the source to a different placeholder image.

(b) Add Rich Audio Responses

APL for Audio - now generally available - enables developers to create rich, layered audio responses using high fidelity audio (48kHz).

Adapting Web API for Games Experiences

Most standard web frameworks, e.g., Phaser and ThreeJS, have support to detect orientation and perform some action like locking or asking users to flip their device. However, by using responsive web design techniques alone to automatically resize, hide, shrink, or enlarge UI elements based on the viewport dimensions, your skill can support portrait mode without any additional developer effort, as seen below with Volley’s Song Quiz. Note that while framework-specific guidance is beyond the scope of this blog, below are some tips on how to achieve portrait mode support with native CSS and Javascript.

Playlist selection from Volley's Song Quiz skill, in portrait mode.

Song Quiz (click here try it!)

ABOVE: Song Quiz using the responsive HTML techniques below to support portrait mode

 

Load a CSS that’s specific to a portrait or landscape mode

Copied to clipboard
<html>
<head>
  <link rel="stylesheet" type="text/css" media="screen and (orientation: portrait)" href="css/portrait.css" />
  <link rel="stylesheet" type="text/css" media="screen and (orientation: landscape)" href="css/landscape.css" />
</head>
  ...
</html>

 

Two methods in Javascript to detect screen orientation

Copied to clipboard
<script>
const debug = document.getElementById("debug");

// You can check if you're in portrait mode or landscape mode in javascript as follows:
let orientation = (screen.orientation || {}).type || screen.mozOrientation || screen.msOrientation;
if (orientation === "portrait-primary" || orientation === "portrait-secondary") {
    debug.innerHTML += "You are in Portrait Mode";
} else {
    debug.innerHTML += "You are in Landscape Mode";
}

// You can also check the screen orientation using window.orientation
switch(window.orientation) {
    case 0:
        debug.innerHTML += "<br/>Rotation: 0 degrees";
        break;
    case 90:
        debug.innerHTML += "<br/>Rotation: 90 degrees";
 
    case 180:
        debug.innerHTML += "<br/>Rotation: 180 degrees";
        break;
    case -90:
        debug.innerHTML += "<br/>Rotation: -90 degrees";
        break;
    default:
        console.log("Rotation: " + window.orientation.toString());
        break;
}
</script>

 

Detect screen change

window.addEventListener(“orientationchange”, () => {}) is deprecated for desktop browsers, but fully supported for mobile browsers, which is how portrait mode is recognized on Echo Show 15 and Fire tablets:

Copied to clipboard
window.addEventListener("orientationchange", function () {
    console.log("orientationChange");
 
    if (screen.orientation.type === "landscape-primary") {
         debug.innerHTML = "You are in Landscape Mode";
      
    } else {
         debug.innerHTML = "You are in Portrait Mode";
    }
});

 

Lock portrait/landscape mode

This can be used to lock in on a specific mode. For example:

<script>
  screen.orientation.lock('portrait');
</script>

How to test Web API portrait mode support

To emulate a 1080dp X 1920dp resolution during development, modify your underlying HTML5 framework’s viewport/window sizing or simulate the Echo Show 15's dimensions using your browser (e.g., Chrome's Device Mode).

Display Template-Based Experiences

As announced in a previous blog post (and again at Alexa Live), Display Templates have been retired. If you own a skill that uses Display Templates, please migrate it to APL as soon as possible. Any skill that doesn’t migrate may see a significant degradation of their customer experience. For example, while <font> tags will be auto-converted to the <span> tag in our cloud (your code will not change), there is no direct equivalent of the  <action>, <div>, and <img> tags. Therefore, touch-based events using <action>, line breaks using <div>, and inline images using <img> will not work properly. Please review your content and make the necessary adjustments.

We look forward to seeing what you build! If you need help come find us on the Alexa Community Slack, at @mhaddy@aruntalkstech, or @austinvach.

Related Articles

Every New APL Feature Announced at Alexa Live
Getting Started with the Web API for Games
Integrate Quick Links (Beta) Into Your Alexa Custom Skill

Subscribe