Functional Testing for a Custom Skill
The functional tests verify that:
-
The skill's basic functionality matches the information displayed on the skill's detail card in the Amazon Alexa app. You enter this data on the Distribution page for the skill on the developer console.
See Define Skill Store Details and Availability for more details about defining this card.
-
The skill's core functionality works and provides useful cards to the Amazon Alexa app.
Use the Amazon Alexa app for these tests. The Alexa app is available for Fire OS, Android, iOS, and web browsers. The Alexa app displays all of your skills currently under development, as well as all published skills.
To return to the high-level testing checklist, see Certification Requirements for Custom Skills.
- 3.1. Review and Test Example Phrases
- 3.2. Skill Description and Detail Page
- 3.3. Cards
- 3.4. Account Linking
- 3.5. Audio Playback
- 3.6 Screen Display and Navigation
- 3.7. Video Playback
- 3.8. Duplicates of Samples or Templates
- 3.9 Device Location API
- 3.10 Echo Buttons
- 3.11 Test Skills
- 3.12 Customer Profile API
- Related Topics
3.1. Review and Test Example Phrases
Example phrases are shown on the skill's detail card in the Alexa app to help users understand how to interact with the skill. You select these phrases from your sample utterances to ensure accuracy.
These are the phrases users are most likely to try the first time they interact with the skill. Therefore, make sure that they work well and provide a good user experience.
The example phrases you enter on the Distribution page are automatically validated against several criteria after you enter them. Correct any errors or warnings before your submit. Note that this does not check for all possible problems, so review the tests below as well.
| Test | Expected Results | |
|---|---|---|
1. |
Open the Alexa app, click Skills and then scroll or search to find your skill. Review the skill's detail page. Inspect all example phrases listed in the skill's detail card. |
|
2. |
Edit your skill in the developer console. Your sample utterances are defined within each intent. Click the intent, review the utterances, and verify that your chosen example phrases match existing sample utterances. For more about defining example phrases, see Define Skill Store Details and Availability: Example Phrases for Custom Skills. If you are using the utterance for a built-in intent for an example phrase, the utterance does not need to be in your sample utterances. However, fully test the utterance and verify that your skill returns a valid response. |
Each example phrase is explicitly included in the sample utterances. For instance, if you have this example phrase:
Then you must also have this utterance:
Any slots in the example phrase (such as Seattle in this example) must be filled with an example of a valid slot value. In this example, |
3. |
Launch the skill using each of the provided example phrases. |
Each example phrase renders a contextualized, non-error response. |
3.2. Skill Description and Detail Page
The skill description teaches users about what your skill can do and how to use it. The description is displayed in the Alexa app when users view available skills. Define the description on the Distribution page.
| Test | Expected Results | |
|---|---|---|
1. |
Open the Alexa app, click Skills and then scroll to your skill. Review the short description shown in the list of skills |
|
2. |
Open the detail card for the skill. Review the skill's full description shown on the skill's detail card. |
|
3. |
If either the short description or full description includes any URLs (such as directing users to more information about the skill), visit the URL in a browser and verify the page opens without error. |
|
4. |
If your skill's detail page includes links to a privacy policy or terms of use, open the links in a browser. |
|
5. |
Review the skill title and ensure it is relevant to the core functionality of the skill. |
|
6. |
Review the skill icon and ensure it is not cropped or blank. |
The icon is not cropped or blank. |
| Test | Expected Results | |
|---|---|---|
1. |
Open the Alexa app, click Skills and then scroll to your skill. Review the short description shown in the list of skills |
|
2. |
Open the detail card for the skill. Review the skill's full description shown on the skill's detail card. |
|
3. |
If either the short description or full description includes any URLs (such as directing users to more information about the skill), visit the URL in a browser and verify the page opens without error. |
|
4. |
If your skill's detail page includes links to a privacy policy or terms of use, open the links in a browser. |
|
5. |
Review the skill title and ensure it is relevant to the core functionality of the skill. |
|
6. |
Review the skill icon and ensure it is not cropped or blank. |
The icon is not cropped or blank. |
| Test | Expected Results | |
|---|---|---|
1. |
Open the Alexa app, click Skills and then scroll to your skill. Review the short description shown in the list of skills |
|
2. |
Open the detail card for the skill. Review the skill's full description shown on the skill's detail card. |
|
3. |
If either the short description or full description includes any URLs (such as directing users to more information about the skill), visit the URL in a browser and verify the page opens without error. |
|
4. |
If your skill's detail page includes links to a privacy policy or terms of use, open the links in a browser. |
|
5. |
Review the skill title and ensure it is relevant to the core functionality of the skill. |
|
6. |
Review the skill icon and ensure it is not cropped or blank. |
The icon is not cropped or blank. |
| Test | Expected Results | |
|---|---|---|
1. |
Open the Alexa app, click Skills and then scroll to your skill. Review the short description shown in the list of skills |
|
2. |
Open the detail card for the skill. Review the skill's full description shown on the skill's detail card. |
|
3. |
If either the short description or full description includes any URLs (such as directing users to more information about the skill), visit the URL in a browser and verify the page opens without error. |
|
4. |
If your skill's detail page includes links to a privacy policy or terms of use, open the links in a browser. |
|
5. |
Review the skill title and ensure it is relevant to the core functionality of the skill. |
|
6. |
Review the skill icon and ensure it is not cropped or blank. |
The icon is not cropped or blank. |
| Test | Expected Results | |
|---|---|---|
1. |
Open the Alexa app, click Skills and then scroll to your skill. Review the short description shown in the list of skills |
|
2. |
Open the detail card for the skill. Review the skill's full description shown on the skill's detail card. |
|
3. |
If either the short description or full description includes any URLs (such as directing users to more information about the skill), visit the URL in a browser and verify the page opens without error. |
|
4. |
If your skill's detail page includes links to a privacy policy or terms of use, open the links in a browser. |
|
5. |
Review the skill title and ensure it is relevant to the core functionality of the skill. |
|
6. |
Review the skill icon and ensure it is not cropped or blank. |
The icon is not cropped or blank. |
| Test | Expected Results | |
|---|---|---|
1. |
Open the Alexa app, click Skills and then scroll to your skill. Review the short description shown in the list of skills |
|
2. |
Open the detail card for the skill. Review the skill's full description shown on the skill's detail card. |
|
3. |
If either the short description or full description includes any URLs (such as directing users to more information about the skill), visit the URL in a browser and verify the page opens without error. |
|
4. |
If your skill's detail page includes links to a privacy policy or terms of use, open the links in a browser. |
|
5. |
Review the skill title and ensure it is relevant to the core functionality of the skill. |
|
6. |
Review the skill icon and ensure it is not cropped or blank. |
The icon is not cropped or blank. |
| Test | Expected Results | |
|---|---|---|
1. |
Open the Alexa app, click Skills and then scroll to your skill. Review the short description shown in the list of skills |
|
2. |
Open the detail card for the skill. Review the skill's full description shown on the skill's detail card. |
|
3. |
If either the short description or full description includes any URLs (such as directing users to more information about the skill), visit the URL in a browser and verify the page opens without error. |
|
4. |
If your skill's detail page includes links to a privacy policy or terms of use, open the links in a browser. |
|
5. |
Review the skill title and ensure it is relevant to the core functionality of the skill. |
|
6. |
Review the skill icon and ensure it is not cropped or blank. |
The icon is not cropped or blank. |
| Test | Expected Results | |
|---|---|---|
1. |
Open the Alexa app, click Skills and then scroll to your skill. Review the short description shown in the list of skills |
|
2. |
Open the detail card for the skill. Review the skill's full description shown on the skill's detail card. |
|
3. |
If either the short description or full description includes any URLs (such as directing users to more information about the skill), visit the URL in a browser and verify the page opens without error. |
|
4. |
If your skill's detail page includes links to a privacy policy or terms of use, open the links in a browser. |
|
5. |
Review the skill title and ensure it is relevant to the core functionality of the skill. |
|
6. |
Review the skill icon and ensure it is not cropped or blank. |
The icon is not cropped or blank. |
3.3. Cards
Cards are graphical cards displayed in the Alexa app to describe or enhance the voice interaction.
| Test | Expected Results | |
|---|---|---|
1. |
Walk through the skill's functionality, invoking each intent. For each response that includes a card, review the card in the Alexa app. (Open the Alexa app and navigate to the Activity page.) |
For each card:
|
2. |
If any cards include URLs, visit each URL in a browser and verify the page opens without error. |
For each URL displayed in a card:
|
Note that skills normally only return cards when responding with the information the user requested. Supportive prompts to ask the user for more information do not normally include cards.
Cards can be a useful tool when developing and debugging a skill, but be sure to remove any extraneous debug-type cards before submitting the skill.
3.4. Account Linking
If your skill needs to connect the identity of the end user with a user in another system (account linking), this test verifies that the skill implements the solution described in Account Linking for Custom Skills correctly. At a high level, this means:
- When the user invokes an intent that requires authorization, the skill successfully authenticates and completes the request.
- The skill returns the
LinkAccountcard if a user invokes an intent that requires authorization, but has not yet linked their account.
Link the skill with test credentials before you submit
When you submit your skill for certification, automated tests check for common certification issues. To ensure that your skill passes these tests, use your Amazon developer account to link your skill with a test account in your other system before you submit:
- In your other system, make sure you have a test account to use with your skill. For example, if your skill connects with your "Ride Hailer" service, you need a Ride Hailer test account. Be sure to use a test account for this purpose, as you will need to share the credentials for this account with the certification team.
-
Log in to the Alexa app with the Amazon developer account you used to create the skill.
Important: You must use your Amazon developer account for this step. - Enable your skill in the Alexa app and start the account linking flow.
- Complete the account linking flow using the test account from step 1. Continuing the earlier example, use your test credentials to link the skill to "Ride Hailer".
- Test that the skill is successfully linked to the other service. Your Amazon developer account should now be connected with the test account in the other service.
- When you submit your skill, provide the credentials (account and password) for the test account in the Testing Instructions field.
If your skill does not use account linking, you can skip this test.
| Test | Expected Results | |
|---|---|---|
1. |
In the developer console, review the Account Linking page for the skill and verify that account linking has been configured correctly. |
|
2. |
In the Alexa app, review the detail card for your skill. Verify that the skill description explains that you must log in to the other system to use the skill. |
The description clearly explains the steps and information necessary to successfully link the accounts. |
3. |
Log in to the Alexa app as an Alexa user who has not yet enabled the skill, or log in and then disable the skill. Enable the skill, but do not complete the account linking process, leaving the skill enabled but not linked.
|
|
4. |
Disable the skill or log in to the Alexa app as an Alexa user who has not yet enabled the skill. Enable the skill. Once the login page for your skill is displayed, click all links and verify that they navigate to new pages. |
|
5. |
Disable the skill or log in to the Alexa app as an Alexa user who has not yet enabled the skill. Enable the skill. Once the login page for your skill is displayed, log in and verify the account linking flow. |
|
3.5. Audio Playback
Complete these tests if your skill uses the AudioPlayer interface to stream audio.
| Test | Expected Results | |
|---|---|---|
1. |
Set the volume on the device to 3 and invoke an intent that streams the audio. Listen for several minutes. |
|
2. |
While listening to the audio playback, invoke the following built-in intents:
|
|
3. |
While listening to the audio playback, use a remote control or other hardware device to pause and resume the audio. |
Be sure your skill properly handles |
4. |
While listening to the audio playback, invoke the following built-in intents:
|
Each command either does something that makes sense for your skill (such as "next" advancing to the next track in a playlist) or is handled gracefully. No unexpected errors occur. See Configuring Your Skill for the AudioPlayer Directives. |
5. |
If your skill plays audio that contains a wake word, play that audio on an Alexa device. |
If your skill plays audio that contains a wake word, the wake word in the audio might wake up the device. Make sure that your skill's audio does not wake up the device. One way to do this is to make sure that there are no pauses after the wake word. |
| Test | Expected Results | |
|---|---|---|
1. |
Set the volume on the device to 3 and invoke an intent that streams the audio. Listen for several minutes. |
|
2. |
While listening to the audio playback, invoke the following built-in intents:
|
|
3. |
While listening to the audio playback, use a remote control or other hardware device to pause and resume the audio. |
Be sure your skill properly handles |
4. |
While listening to the audio playback, invoke the following built-in intents:
|
Each command either does something that makes sense for your skill (such as "next" advancing to the next track in a playlist) or is handled gracefully. No unexpected errors occur. See Configuring Your Skill for the AudioPlayer Directives. |
| Test | Expected Results | |
|---|---|---|
1. |
Set the volume on the device to 3 and invoke an intent that streams the audio. Listen for several minutes. |
|
2. |
While listening to the audio playback, invoke the following built-in intents:
|
|
3. |
While listening to the audio playback, use a remote control or other hardware device to pause and resume the audio. |
Be sure your skill properly handles |
4. |
While listening to the audio playback, invoke the following built-in intents:
|
Each command either does something that makes sense for your skill (such as "nächste" advancing to the next track in a playlist) or is handled gracefully. No unexpected errors occur. See Configuring Your Skill for the AudioPlayer Directives. |
| Test | Expected Results | |
|---|---|---|
1. |
Set the volume on the device to 3 and invoke an intent that streams the audio. Listen for several minutes. |
|
2. |
While listening to the audio playback, invoke the following built-in intents:
|
|
3. |
While listening to the audio playback, use a remote control or other hardware device to pause and resume the audio. |
Be sure your skill properly handles |
4. |
While listening to the audio playback, invoke the following built-in intents:
|
Each command either does something that makes sense for your skill (such as "अगला" advancing to the next track in a playlist) or is handled gracefully. No unexpected errors occur. See Configuring Your Skill for the AudioPlayer Directives. |
5. |
If your skill plays audio that contains a wake word, play that audio on an Alexa device. |
If your skill plays audio that contains a wake word, the wake word in the audio might wake up the device. Make sure that your skill's audio does not wake up the device. One way to do this is to make sure that there are no pauses after the wake word. |
| Test | Expected Results | |
|---|---|---|
1. |
Set the volume on the device to 3 and invoke an intent that streams the audio. Listen for several minutes. |
|
2. |
While listening to the audio playback, invoke the following built-in intents:
|
|
3. |
While listening to the audio playback, use a remote control or other hardware device to pause and resume the audio. |
Be sure your skill properly handles |
4. |
While listening to the audio playback, invoke the following built-in intents:
|
Each command either does something that makes sense for your skill (such as "la prossima" advancing to the next track in a playlist) or is handled gracefully. No unexpected errors occur. See Configuring Your Skill for the AudioPlayer Directives. |
| Test | Expected Results | |
|---|---|---|
1. |
Set the volume on the device to 3 and invoke an intent that streams the audio. Listen for several minutes. |
|
2. |
While listening to the audio playback, invoke the following built-in intents:
|
|
3. |
While listening to the audio playback, use a remote control or other hardware device to pause and resume the audio. |
Be sure your skill properly handles |
4. |
While listening to the audio playback, invoke the following built-in intents:
|
Each command either does something that makes sense for your skill (such as "次" advancing to the next track in a playlist) or is handled gracefully. No unexpected errors occur. See Configuring Your Skill for the AudioPlayer Directives. |
| Test | Expected Results | |
|---|---|---|
1. |
Set the volume on the device to 3 and invoke an intent that streams the audio. Listen for several minutes. |
|
2. |
While listening to the audio playback, invoke the following built-in intents:
|
|
3. |
While listening to the audio playback, use a remote control or other hardware device to pause and resume the audio. |
Be sure your skill properly handles |
4. |
While listening to the audio playback, invoke the following built-in intents:
|
Each command either does something that makes sense for your skill (such as "next" advancing to the next track in a playlist) or is handled gracefully. No unexpected errors occur. See Configuring Your Skill for the AudioPlayer Directives. |
| Test | Expected Results | |
|---|---|---|
1. |
Set the volume on the device to 3 and invoke an intent that streams the audio. Listen for several minutes. |
|
2. |
While listening to the audio playback, invoke the following built-in intents:
|
|
3. |
While listening to the audio playback, use a remote control or other hardware device to pause and resume the audio. |
Be sure your skill properly handles |
4. |
While listening to the audio playback, invoke the following built-in intents:
|
Each command either does something that makes sense for your skill (such as "next" advancing to the next track in a playlist) or is handled gracefully. No unexpected errors occur. See Configuring Your Skill for the AudioPlayer Directives. |
3.6 Screen Display and Navigation
Complete these tests if your skill uses the Display interface to display content on Alexa-enabled devices with screens. Pay particular attention to the interaction of voice, touch, audio, and video in your skill.
| Test | Expected Results | |
|---|---|---|
1. |
Open the skill using an Alexa-enabled device with a screen and interact with the skill for several minutes, paying particular attention to the user experience. |
|
2. |
For Echo Show or Fire TV Cube, place the device at a 7-foot or 2-meter distance from the viewer. Interact with the skill for several minutes. |
|
3. |
Exit the skill using "stop" or "cancel" while touch screen inputs are on the screen. |
|
3.7. Video Playback
In addition to the screen display requirements, some of which also apply to video, also complete these tests if your skill uses the VideoApp interface to stream video.
| Test | Expected Results | |
|---|---|---|
1. |
Set the volume on the device to 3 and invoke an intent that streams the video. Listen for several minutes. |
|
2. |
While the video plays, use the progress bar on the touchscreen to navigate forward and backward. |
Verify that the video resumes without error. |
3.8. Duplicates of Samples or Templates
Make sure your skill is not an exact duplicate of any of the Amazon-provided samples or templates. You can use the samples and templates as a starting point for your skill, but the finished skill must be different.
For example, if you use the Trivia sample to create a trivia skill, your skill must have:
- A different name
- A different set of questions
The samples and templates are available on GitHub:
- Javascript/Node.js: Alexa organization
- Java: amzn/alexa-skills-kit-java
3.9 Device Location API
Complete these tests if your skill uses the Device Address API to obtain the location of the user's device.
| Test | Expected Results | |
|---|---|---|
1. |
Enable the skill and grant the skill the requested permissions for the device location. Test the functionality that is dependent on the device location and note the responses. |
|
2. |
Enable the skill, but do not grant the device location permissions.
|
|
3. |
If applicable, change the device location to a location or address that is not supported by the skill, then enable the skill and grant permissions. Invoke the intents that require the device location. For example, if your skill provides food delivery in a particular city, set the device location to an address far outside your delivery area, then invoke the intents to place a delivery order. |
The skill informs the user that skill does not support their location. No errors are returned. |
3.10 Echo Buttons
Complete these tests if your skill works with Echo Buttons.
| Subject | Test | Expected Results |
|---|---|---|
Roll call |
Launch the skill with buttons paired, but without any buttons awake. |
When the skill first launches, it must begin with a roll call that prompts users to wake their paired buttons. Note: Skills can skip roll call if launched within ten minutes of a previous session of the same skill. |
Roll call timeout |
Launch the skill without any buttons awake. During roll call, do not press any buttons. |
If users do not press buttons in response to roll call, the skill must respond gracefully. The skill can do one of the following: 1) re-prompt users to press their buttons, 2) close the skill after informing users that buttons are required to use the skill, or 3) ask users to use the skill without buttons. The skill must not close or provide a hard error immediately following the timeout. |
Valid inputs |
Launch the skill with the required number of buttons per the skill's metadata. Following all customer-facing instructions in the skill's prompts and metadata, navigate the skill for up to ten minutes. Examine the skill's responses to valid button presses throughout navigation. |
The skill must return the expected responses when the user makes valid button presses. Note: This test applies equally to skills for which buttons are required and buttons are optional. |
Empty inputs |
Launch the skill with the required number of buttons per the skill's metadata. Navigate past roll call. When the skill prompts users to press buttons, do not press any buttons. |
For both button-required and button-optional skills, if users do not press buttons when the skill expects button presses, the skill must re-prompt users to press the buttons. The skill must not close or return a hard error. |
Player mode |
Launch the skill with the required number of buttons per the skill's metadata. Navigate past roll call and play the skill in single-player and multi-player modes, if applicable. |
In both modes, the skill must handle responses from all users in a mutually exclusive manner and must be able to maintain progress for all users independently. |
Dropping a player |
(Only applies if the skill supports dropping a player) |
After dropping a player in the middle of gameplay, the rest of the players must be able to continue playing the game. If a scorecard is available, the scorecard must not reset when the player is dropped. Note: When only two players play and a player drops, it depends upon the skill design whether the skill has to end or the skill can continue with a single player. |
LED colors |
Launch the skill with the required number of buttons per the skill's metadata. Navigate past roll call and observe the LED animations on the buttons during gameplay. |
The skill must use LED colors on the buttons to provide an interactive experience to the users. For example, in a Q&A skill, the skill might light up buttons with a color depending on whether the user provides a correct or incorrect answer. The color must be consistent when the answer is incorrect vs. correct. As another example, in a multi-player skill, the skill might light up Player 1's button to indicate that Player 1 must respond. |
Optional |
(Only applies to skills where buttons are optional) |
The user must be able to use the skill without buttons. The skill must also ask the user before roll call or during roll call whether the user wants to play with or without buttons. |
3.11 Test Skills
Alexa skills submitted for certification should have complete functionality and should not be test, trial, samples, or otherwise appear to be under development. The information updated in the distribution section of the developer console should not indicate that the skill is a trial/sample skill. To get started building skills, see Build Skills with the Alexa Skills Kit.
3.12 Customer Profile API
Complete these tests if your skill uses the Alexa Customer Profile API to obtain the customer contact information of the user.
If your skill does not use the Customer Profile API, make sure this permission is turned off:
- Edit your skill in the developer console.
- Navigate to the Build > Permissions page
- Turn off each of the following:
- Customer Email Address
- Customer Phone Number
- Customer Name
| Test | Expected Results | |
|---|---|---|
1. |
Enable the skill and grant skill the requested permissions like Customer Name, Customer Email Address, and/or Customer Phone Number. Test the functionality that is dependent on the customer contact Information and note the responses. |
All skill responses are appropriate with regards to the customer contact information permission. For example: the weather information requested has been sent to the correct email address. (Based on the permission the skill requires in the skill). |
2. |
Enable the skill, but do not grant the customer contact information permissions.
|
|
3. |
Change the customer contact information, then enable the skill and grant permissions. Invoke the intents that require the customer contact permission. |
All skill responses are appropriate with regards to the updated customer contact information and do not return any error message. |
