How to Build a Multimodal Alexa Skill

1. What are APL commands?

2. Updating our APL documents

3. Adding commands to our document

4. Adding video control

5. ExecuteCommands directive

APL Commands are messages which change the visual or audio presentation of the content delivered to your customer. There are a few different ways you can use commands. The first is by using the directive, Alexa.Presentation.APL.ExecuteCommands. There are also different handler properties in some components and even in the APL document. For an example, the TouchWrapper responsive component has an onPress property which takes in a list of commands. Today, we will be using a similar mechanism within the APL document, the onMount property of our APL document which executes commands immediately upon inflating the document. This is useful for the animation commands we are using in our introduction.

Now, let’s animate some text objects. This is what we want to create:

First, we will need to update out APL Documents to be able to use commands by adding an id property to our components. Since we want the commands to execute immediately on the APL screen loading, we will be putting the commands inside the onMount property of our launch document. In order to target our components with the commands, we’ll need to add some logical ids for our text components. Hold up though! These are defined in the APL package. If you check the hosted version, you will see there are already component ids added by default! These are textTop, textMiddle, textBottom corresponding to the top, middle, and bottom text components, so there is no need to add anything. However, if you would like to add different ids, each text component id is exposed as a parameter in the APL package.

A. In your launch document (launchDocument.json file), add "id": "image", to the AlexaImage component.

B. In your birthday visual, (birthdayDocument.json file), add "id": "birthdayVideo", to your video component.

The command we will be using is the AnimateItem command. This command was added in APL 1.1, so make sure that your APL document is declaring version 1.1 which will look like version: '1.1'. The animate item command will run a fixed-duration animation sequence on one or more properties of a single component. We are going to modify the transform property.

The basic structure of our animate item commands will be :

The transform property requires you to specify where the component is moving to and from. All transformations in the "to" and "from" arrays must match types. In this case, the type is translationX. For more information on valid values, see the animate command values tech doc. We are going to use the ParallelCommand in order to run all of our text animations in parallel. This command has a list of commands to execute called child commands. All child commands execute simultaneously.

Now that we understand the structure of our commands, we will need to create the JSON representing the commands inside the onMount property of our launchDocument.json file. Let’s add some commands. To start, we will animate the text components to all slide in from different directions like in the gif and we want the commands to execute all at the same time, so the 3 text AnimateItem commands will be contained within the parallel command. To arrive at the values for each of these translations, we’ll have to understand the viewport coordinate system.

The viewport is drawn is with coordinate (0,0) contained at the upper left corner. In addition, all of the 'to' properties will be relative to the object’s position in the APL document we built, not relative to the viewport coordinates. For the first object, we’ll translate from 1600, to 0 on the X axis. The middle text will come from the left, so we will translate from -400 (so we can ensure the Text cannot be seen before animation) to 0 on the X axis, and the last item will come from below, which will use 'translateY' from 1200 to 0, since Y is positive going down.

A. Add the set of commands to your onMount property in your launchDocument.json document. Putting the above inside of the parallel command will leave us with:

Once that is working, let’s make the more complex animation for the image component. Looking at how this animation runs, we will need to scale our image from a really small scale to 1 (full size). We are also rotating it from 0 to 360 degrees over this duration which will be 2 seconds. You will notice the path it takes is not quite linear and different from the other animations. This is because it is custom defined. You do not have to stick to the defined properties in the chart below, but can define your own curve with cubic-bezier curves or a linear path. In fact, the named curves all have mathematical definitions listed in the chart below. The coordinates start at (0,0) and go to (1,1). Think of the X coordinate as time and Y as magnitude of the change. Here is the curve I defined "easing": "path(0.25, 0.2, 0.5, 0.5, 0.75, 0.8)", But if you want to write your own, feel free!

B. Put this all together for the image command gives us:

Add this in your launchDocument.json file inside the onMount command list.

C. Now test it out!

D. Once that is working, enter your birthday and test the launchHandler with context when it is not your birthday. You should see the commands applied to this as well. We are not quite done. What about animations when it is their birthday? Since, that experience is defined in the birthdayDocument.json file which we have not added commands to. Let’s fix this.

Did you notice the other change in the above gif? There is a new component added to the birthdayDocument.json document, the AlexaTransportsControls responsive component. You should always have an on screen control for your video or it may not pass certification. Let’s add this. This component is also a part of the alexa-layouts package.

A. Add the AlexaTransportControls component to the container with the video component inside birthdayDocument.json, since we want this aligned to the center, too. This should go after the video component in the items list.

Our component has a secondary control of 0 because we do not want to show the secondary control buttons. These are the skip and rewind buttons if you were playing a series of videos. The primary control size is the size of the play button. The mediaComponentId must reference the VideoComponent earlier in the document.

B. Save and deploy these changes and test for your birthday scenario. Make sure the button is functioning and stops and replays the video when toggled.

Did you notice the clipping on the audio response from Alexa? You may not notice this if your birthday is close enough, but Alexa’s voice response is getting cut off when the video starts to play. To fix this we will need to use the ExecuteCommands Directive.

Alexa is cut off from speaking when the video starts to play. We want Alexa to finish speaking, then start the video automatically. We need to turn off autoplay in order to fix this, but it does not make sense for our customers to have to tell the video to start. We’ll use commands to solve this.

To fix the audio, we are going to have to add the ExecuteCommands directive to our backend as well as a payload for it. The execute commands directive will execute the list of provided commands after Alexa is done speaking. It looks like this:

For our usage, we will need the skill provided token for the ExecuteCommand directive to target, so this can be "birthdayToken". Without this, our command will not know which document to target to execute on.

A. Add a new token field to the APL RenderDocument directive with the value of birthdayToken. Your addDirective(…​) will now look like:

B. In the else block in our HasBirthdayLaunchRequestHandler, we will need to add another directive. This can be chained onto our current render directive. Add the below code to the handlerInput.responseBuilder.

C. Replace the <List_of_commands> with our commands list. This is simply going to be a single command to start the video. Since this happens once Alexa is done speaking, we get the behavior we want! The command looks like this:

You’ll end up with APL directive code that looks like this:

D. Save and deploy and test this out now.

That’s a cool animation isn’t it? Great work on expanding your Cake Time with images, text, video, and animations!

Complete code in Github