Voice is a fast-evolving user interface. Over the last few years, many developers started creating voice experiences by building a simple one-shot “fact skill” using the AWS console at a hackathon or workshop. Fast forward to today and two trends are clear: developers are building multi-turn conversational skills, and they like to write skill code from their laptop using an integrated development environment (IDE).

Developing code on a laptop is great, but deploying your code to test it may cause unnecessary interruptions to your mental flow. It takes time to zip your project, deploy your code to AWS Lambda, fire up the skill, ask a series of questions, stop the skill, locate and refresh the log file in Amazon CloudWatch, and search for debug messages and errors.

The Alexa evangelist team built a simulation tool called TestFlow to streamline skill debugging and testing. You can use the tool to test your skill code without needing to package or deploy your project. TestFlow is a lightweight, command-line dialog simulator. It displays and maintains the details of a mock skill session. Whenever you want to run a test, you just Alt-Tab to a command prompt, run TestFlow, review results, and then Alt-Tab right back to developing.

A voice designer could define a happy-path flow of user requests to the skill. For example, imagine a file listing the launch request, then a help query, and then a stop command.

file: dialogs/default.txt

Here’s another example showing how to define intents, slots, and a pause for user input.

file: dialogs/page.txt

As a developer, you can save dialog sequence files like this to a /dialogs folder in your project. You can use these files in two ways–as a document for specifying and sharing expected skill sequences and as an input to TestFlow.

TestFlow will open both this file and your source file and execute this series of unit tests for you.

To run TestFlow, you just type :  node testflow

How TestFlow Works

You can think of TestFlow as a lightweight skill simulator rather than a traditional testing tool. Both the inputs and outputs to and from TestFlow are reduced to the simplest possible format to remove any friction in the testing process. As TestFlow executes multiple events, it will maintain session state for you, like the real Alexa voice service.

Developers can envision, define, run, and repeat “what if” scenarios quickly, having full control over which text and attributes to display, whether the test runs fully automated or with guided user input, and how slowly the test should run. For example, if you set a delay of 5 seconds between test events, users will have time to read and digest each request and response as it scrolls past. TestFlow is not network-dependent; it is a single file that runs alongside your code, and is perfect for developers who want to develop offline, like when traveling.  Testflow is configurable; you can set display and runtime options directly in the testflow.js file, or hack the script for your own purposes.

Here we see the test begins in automated fashion, then pauses to allow the user to type in a custom slot value of 33 rather than accept the default value.

Developers must understand how to set and get context, or session attributes, in order to remember things and provide intelligent responses to the user. An attribute could be used to remember the value of a user utterance slot, or for internal metadata such as a count of times the user launched the skill.

However, session attributes are largely invisible, requiring the developer to imagine their state or log their state as a skill executes.

Developers can configure TestFlow to show all the session attributes shaded in magenta; or just a single named attribute, similar to a “watch” variable in traditional debugging.

TestFlow prepares a mock request JSON for each execution, containing a User ID and timestamp.

Developers can define unique User IDs for events to run as, as well as whether the event runs as of right now or at a past or future time. In this way, you can simulate skills with multiple users and persistent memory.

You can also specify slot values in the dialog sequence using a single value, or by simulating various entity resolution events, such as testing both synonym with canonical values  See the dialog sequence spec for the full details.

TestFlow supports both Node.JS and Python projects, any version of the Alexa Skills Kit Software Development Kit (SDK), and skills that call other APIs or AWS services.  See the Setup Steps for hints on configuring your local environment.

 

The Testing Tools Landscape

TestFlow is one of dozens of approaches to testing an Alexa skill.  The developer portal Test console is the primary tool to use, especially for reviewing how utterances map to your language model. Echosim.io from a browser or the Alexa app now features Alexa buttons, and can be used to interact with your skill verbally.

The Alexa Skills Management API (SMAPI) and Alexa Skills Kit Command Line Interface (CLI) tools can simulate utterances to an Alexa skill. Also, Bespoken’s BST command line utilities provide an integrated testing framework for skills.

Formal QA testing frameworks can be used to verify the expected output of your skill code, and are often automated as part of a continuous integration workflow. MochaJS and Chai are popular open source Javascript test frameworks. 

TestFlow is unique in that it requires no device, browser, or network; it simplifies inputs and outputs to the minimum.

Get started with the TestFlow here, and follow the tutorial that covers the options and test formats.

More Resources

• Building Engaging Alexa Skills: Why Testing and Automation Matter
• Now It’s Easier to Test Your Alexa Skill for a Great Customer Experience on Echo Spot
• Certify Your Skills Faster Using New Tools in the Alexa Developer Console

Make Money by Creating Engaging Skills Customers Love

You can make money through Alexa skills using in-skill purchasing or Amazon Pay for Alexa Skills. You can also make money for eligible skills that drive some of the highest customer engagement with Alexa Developer Rewards. Download our guide to learn which product best meets your needs.