Regular skill testing can help you deliver consistently great and reliable voice experiences to customers. But as your skill becomes more complex, testing might become a time-consuming task if you have to test hundreds of possible dialogs manually. The right tools can help you simulate complex interactions in an automated way, which reduces the level of effort spent on repetitive tasks.
Test automation can also help you find bugs early in the cycle of the software development process – and the earlier you find a bug, the more time you have to fix it Furthermore, when multiple people work on the same code base, having test automation before checking in the code can prevent regressions from happening. As a result, your customers will benefit from a consistent and reliable voice experience.
We previously shared a high-level overview of automated testing and some best practices here and here, and introduced a cmd-line simulator called TestFlow. Today’s post shares how to create unit tests to validate a skill’s output against predefined assertions.
Types of Automated Tests for Alexa Skills
The classic test pyramid (referenced in the diagram below) outlines the different types of testing you can leverage to create an effective and balanced test automation suite. You should have many more unit tests than tests that validate the voice interaction with Alexa. In this way, you can address the major functional test cases with unit tests and you can treat the voice user interface (VUI) interaction as a sanity test to ensure the voice experience works as expected.
Unit testing involves breaking Alexa skill backend code into pieces and subjecting each piece to a series of tests. We declare the correct outputs and conditions we expect to see from each unit test. The testing tool will report on the number of passing and failing tests.
Choosing a Unit Test Framework
There are several open source unit test frameworks available. Bespoken Tools has an example setup within the Fact Skill sample. Another is BrainMacInosh’s Alexa-Skill-Test-Framework. Both of the frameworks currently support the latest Alexa Skills Kit (ASK) Software Development Kit (SDK).
These frameworks are easy to set up and they support different types of skill features like cards, dialog models, Amazon DynamoDB, and session attributes. Under the covers they rely on standard test packages like Jest or Mocha.
In this post, we’ll use BrainMacInosh’s Alexa-Skill-Test-Framework to test a sample skill. If you’d like to follow along on your own device, make sure you have Node.JS and GitHub installed.
Sample Skill Unit Testing
Step 1: Clone the sample skill to your local desktop and install the node_modules dependency
- Open Hello World Sample Skill on GitHub.
- Run “git clone https://github.com/alexa/skill-sample-nodejs-hello-world.git” to clone the Skill folder to your local desktop.
- Go to skill-sample-nodejs-hello-world folder that you just cloned, navigate to lambda/custom and run “npm install” to install the node_modules from package.json.
Step 2: Set up the test framework from npm
- Make sure you are under the project root folder.
- From the root, create a folder and name it “test.”
- Under “test” folder, run the command to install the test framework:
- npm install alexa-skill-test-framework --save-dev
- npm install mocha --global
- Check the node_modules folder under “test” folder to verify the alexa-skill-test-framework module is installed successfully.
Step 3: Run a sample test case
- Copy the Hello World sample test file: helloworld-tests.js from GitHub and put it under the “test” folder.
- Make the following updates to helloword-test.js:
- Remove the comment out from “const alexaTest = require(‘alexa-skill-test-framework’”.
- Comment out “const alexaTest = require(‘../index’).
- Change alexaTest.initialize(require(‘./helloword.js’), to alexaTest.initialize(require(‘../lambda/custom/index.js’).
The code shoud look like this:
- Navigate to “test” folder, run “mocha helloworld-tests.js”. You will be able to see the test result for your Hello World sample Skill.
Write Your Own Tests Under Test Folder
Example 1: Unit Test for Hello World Skill
Now that we are setup, we will continue to use the Hello World Skill Test example to learn how to write custom unit tests. This test case test snippet demonstrates how to test the launch request in the Hello World Skill. The script runs “alexaTest.getLaunchRequest()” to simulate a launch request, and then defines the expected speech output under “says”. “repromptsNothing”: true means no reprompt is included in the launch request.
Let’s review the first unit test in your test script, LaunchRequest:
describe("Hello World Skill", function () {
// tests the behavior of the skill's LaunchRequest
describe("LaunchRequest", function () {
alexaTest.test([
{
request: alexaTest.getLaunchRequest(),
says: "Welcome to the Alexa Skills Kit, you can say hello!",
repromptsNothing: true,
shouldEndSession: true
}
]);
});
]);
});
};After you run the test, errors are displayed. From the screenshot you should see two errors, and we will be focusing on the first one:
Example 2: Unit Test for Checking Session Attributes
Replace the code in index.js with the code from this helloworld.js file. This skill uses session attributes to store values inside a session.
Here is the portion of the code that sets the session attributes in this sample skill; it calls an attributeManager from handlerInput to get all the session attributes and assigns them to a variable. We can customize the session attributes value for this sample, change “attributes.foo = ‘bar’” to “attributes.activity = ‘eating’”.
const HelloWorldIntentHandler = {
canHandle(handlerInput) {
return handlerInput.requestEnvelope.request.type === 'IntentRequest'
&& handlerInput.requestEnvelope.request.intent.name === 'HelloWorldIntent';
},
handle(handlerInput) {
const speechText = 'Hello World!';
const attributes = handlerInput.attributesManager.getSessionAttributes();
attributes.activity = 'eating';
return handlerInput.responseBuilder
.speak(speechText)
.withSimpleCard('Hello World', speechText)
.getResponse();
},
};To test, in the test file under the “describe” block, we simply need to put the expected pair of values after hasAttributes. If you are using version 1 of the ASK SDK, the test method implementation is the same. Here is the sample test code:
//tests the behavior of the Skill's HelloWorldIntent
describe("HelloWorldIntent", function () {
alexaTest.test([
{
request: alexaTest.getIntentRequest("HelloWorldIntent"),
says: "Hello World!", repromptsNothing: true, shouldEndSession: true,
hasAttributes: {
activity: ‘eating'
}
}
]);
});
};Hopefully you have a better understanding how to leverage unit testing to ease your Skill testing. Reach out via the Alexa developer forums to learn more!
More Resources
- TestFlow: Simulate Conversations with Your Alexa Skill Code to Ease Debugging
- Building Engaging Alexa Skills: Why Testing and Automation Matter
- Certify Your Skills Faster Using New Tools in the Alexa Developer Console
- Now It’s Easier to Test Your Alexa Skill for a Great Customer Experience on Echo Spot