Functional Testing for a Custom Skill
During the certification process the certification team runs functional tests to verify that the core functionality of the skill works and the information displayed on the skill detail card in the Alexa app matches the functionality. You enter the skill description on the Distribution page for the skill on the developer console. For more details, see Define Skill Store Details for Publication.
Use the Alexa app for these tests. The Alexa app is available for Fire OS, Android, and iOS. The Alexa app displays your skills under development, as well as all published skills.
To return to the testing checklist, see Skill Certification Testing.
Review and test example phrases
To help users understand how to interact with the skill, you include example phrases in your skill metadata that display on the skill detail card in the Alexa app. You select these phrases from your sample utterances. Users usually try the example phrases the first time they interact with the skill. Therefore, make sure that these phrases work well and provide a good user experience.
For more details about defining example phrases, see Example phrases for custom skills.
The following image shows a skill detail card for the Dev Tips skill in the Alexa app
The example phrases you enter on the Distribution page in the Alexa developer console are automatically validated against several criteria. After you enter the phrases, correct any errors or warnings. Then, verify your phrases with the following tests because the console doesn't check for all possible problems.
Test | Expected Results | |
---|---|---|
1. |
Open the Alexa app, click Skills and then scroll or search to find your skill. Review the skill's detail page. Inspect all example phrases listed on the skill's detail card. |
|
2. |
Review your interaction model in the Alexa developer console. Verify that sample utterances are defined within each intent. Click the intent, review the utterances, and verify that your chosen example phrases match existing sample utterances. |
Each example phrase is explicitly included in the sample utterances. For instance, if you have this example phrase:
Then you must also have this utterance:
Any slots in the example phrase (such as Seattle in this example) must be filled with an example of a valid slot value. In this example, |
3. |
Launch the skill using each of the provided example phrases. |
Each example phrase renders a contextualized, non-error response. |
Skill description and detail page
The skill description teaches users about what your skill can do and how to use it. The description displays in the Alexa app when users view available skills. You define the description on the Distribution page in the developer console.
Test | Expected Results | |
---|---|---|
1. |
Open the Alexa app, click Skills and then scroll to your skill. Review the short description shown in the list of skills |
|
2. |
Open the detail card for the skill. Review the skill's full description shown on the skill's detail card. |
|
3. |
If either the short description or full description includes any URLs (such as directing users to more information about the skill), visit the URL in a browser and verify the page opens without error. |
|
4. |
If your skill's detail page includes links to a privacy policy or terms of use, open the links in a browser. |
|
5. |
Review the skill title and ensure it is relevant to the core functionality of the skill. |
|
6. |
Review the skill icon and ensure it is not cropped or blank. |
The icon is not cropped or blank. |
Test | Expected Results | |
---|---|---|
1. |
Open the Alexa app, click Skills and then scroll to your skill. Review the short description shown in the list of skills |
|
2. |
Open the detail card for the skill. Review the skill's full description shown on the skill's detail card. |
|
3. |
If either the short description or full description includes any URLs (such as directing users to more information about the skill), visit the URL in a browser and verify the page opens without error. |
|
4. |
If your skill's detail page includes links to a privacy policy or terms of use, open the links in a browser. |
|
5. |
Review the skill title and ensure it is relevant to the core functionality of the skill. |
|
6. |
Review the skill icon and ensure it is not cropped or blank. |
The icon is not cropped or blank. |
Test | Expected Results | |
---|---|---|
1. |
Open the Alexa app, click Skills and then scroll to your skill. Review the short description shown in the list of skills |
|
2. |
Open the detail card for the skill. Review the skill's full description shown on the skill's detail card. |
|
3. |
If either the short description or full description includes any URLs (such as directing users to more information about the skill), visit the URL in a browser and verify the page opens without error. |
|
4. |
If your skill's detail page includes links to a privacy policy or terms of use, open the links in a browser. |
|
5. |
Review the skill title and ensure it is relevant to the core functionality of the skill. |
|
6. |
Review the skill icon and ensure it is not cropped or blank. |
The icon is not cropped or blank. |
Test | Expected Results | |
---|---|---|
1. |
Open the Alexa app, click Skills and then scroll to your skill. Review the short description shown in the list of skills |
|
2. |
Open the detail card for the skill. Review the skill's full description shown on the skill's detail card. |
|
3. |
If either the short description or full description includes any URLs (such as directing users to more information about the skill), visit the URL in a browser and verify the page opens without error. |
|
4. |
If your skill's detail page includes links to a privacy policy or terms of use, open the links in a browser. |
|
5. |
Review the skill title and ensure it is relevant to the core functionality of the skill. |
|
6. |
Review the skill icon and ensure it is not cropped or blank. |
The icon is not cropped or blank. |
Test | Expected Results | |
---|---|---|
1. |
Open the Alexa app, click Skills and then scroll to your skill. Review the short description shown in the list of skills |
|
2. |
Open the detail card for the skill. Review the skill's full description shown on the skill's detail card. |
|
3. |
If either the short description or full description includes any URLs (such as directing users to more information about the skill), visit the URL in a browser and verify the page opens without error. |
|
4. |
If your skill's detail page includes links to a privacy policy or terms of use, open the links in a browser. |
|
5. |
Review the skill title and ensure it is relevant to the core functionality of the skill. |
|
6. |
Review the skill icon and ensure it is not cropped or blank. |
The icon is not cropped or blank. |
Test | Expected Results | |
---|---|---|
1. |
Open the Alexa app, click Skills and then scroll to your skill. Review the short description shown in the list of skills |
|
2. |
Open the detail card for the skill. Review the skill's full description shown on the skill's detail card. |
|
3. |
If either the short description or full description includes any URLs (such as directing users to more information about the skill), visit the URL in a browser and verify the page opens without error. |
|
4. |
If your skill's detail page includes links to a privacy policy or terms of use, open the links in a browser. |
|
5. |
Review the skill title and ensure it is relevant to the core functionality of the skill. |
|
6. |
Review the skill icon and ensure it is not cropped or blank. |
The icon is not cropped or blank. |
Test | Expected Results | |
---|---|---|
1. |
Open the Alexa app, click Skills and then scroll to your skill. Review the short description shown in the list of skills |
|
2. |
Open the detail card for the skill. Review the skill's full description shown on the skill's detail card. |
|
3. |
If either the short description or full description includes any URLs (such as directing users to more information about the skill), visit the URL in a browser and verify the page opens without error. |
|
4. |
If your skill's detail page includes links to a privacy policy or terms of use, open the links in a browser. |
|
5. |
Review the skill title and ensure it is relevant to the core functionality of the skill. |
|
6. |
Review the skill icon and ensure it is not cropped or blank. |
The icon is not cropped or blank. |
Test | Expected Results | |
---|---|---|
1. |
Open the Alexa app, click Skills and then scroll to your skill. Review the short description shown in the list of skills |
|
2. |
Open the detail card for the skill. Review the skill's full description shown on the skill's detail card. |
|
3. |
If either the short description or full description includes any URLs (such as directing users to more information about the skill), visit the URL in a browser and verify the page opens without error. |
|
4. |
If your skill's detail page includes links to a privacy policy or terms of use, open the links in a browser. |
|
5. |
Review the skill title and ensure it is relevant to the core functionality of the skill. |
|
6. |
Review the skill icon and ensure it is not cropped or blank. |
The icon is not cropped or blank. |
Test | Expected Results | |
---|---|---|
1. |
Open the Alexa app, click Skills and then scroll to your skill. Review the short description shown in the list of skills |
|
2. |
Open the detail card for the skill. Review the skill's full description shown on the skill's detail card. |
|
3. |
If either the short description or full description includes any URLs (such as directing users to more information about the skill), visit the URL in a browser and verify the page opens without error. |
|
4. |
If your skill's detail page includes links to a privacy policy or terms of use, open the links in a browser. |
|
5. |
Review the skill title and ensure it is relevant to the core functionality of the skill. |
|
6. |
Review the skill icon and ensure it is not cropped or blank. |
The icon is not cropped or blank. |
Test | Expected Results | |
---|---|---|
1. |
Open the Alexa app, click Skills and then scroll to your skill. Review the short description shown in the list of skills |
|
2. |
Open the detail card for the skill. Review the skill's full description shown on the skill's detail card. |
|
3. |
If either the short description or full description includes any URLs (such as directing users to more information about the skill), visit the URL in a browser and verify the page opens without error. |
|
4. |
If your skill's detail page includes links to a privacy policy or terms of use, open the links in a browser. |
|
5. |
Review the skill title and ensure it is relevant to the core functionality of the skill. |
|
6. |
Review the skill icon and ensure it is not cropped or blank. |
The icon is not cropped or blank. |
Cards
Cards are graphical cards displayed in the Alexa app to describe or enhance the voice interaction. Usually, skills return cards only when responding with information the user requested. Supportive prompts to ask the user for more information don't usually include cards.
Test | Expected Results | |
---|---|---|
1. |
Walk through the skill's functionality, invoking each intent. For each response that includes a card, review the card in the Alexa app. (Open the Alexa app and navigate to the Activity page.) |
For each card:
|
2. |
If any cards include URLs, visit each URL in a browser and verify the page opens without error. |
For each URL displayed in a card:
|
Account linking
Account linking enables your skill to connect the user's account in your system with their Amazon Alexa account. For more details, see Add Account Linking to Your Alexa Skill. If your skill doesn't use account linking, you can skip this test.
If your custom skill implements account linking, verify that your skill follows all the instructions in Account Linking for Custom Skills. If your smart home, video, or music skill implements account linking, verify that your skill follows all the instructions in Account Linking for Smart Home and Other Domains.
In addition, test the account linking flow from the user perspective:
- When the user invokes an intent that requires authorization, the skill successfully authenticates and completes the request.
- The skill returns the
LinkAccount
card if a user invokes an intent that requires authorization, but hasn't yet linked their account.
Link the skill with test credentials before you submit
When you submit your skill for certification, automated tests check for common certification issues. To verify that your skill passes these tests, use your Amazon developer account to link your skill with a test account in your other system before you submit:
- In your other system, make sure you have a test account to use with your skill. For example, if your skill connects with your "Ride Hailer" service, you need a Ride Hailer test account. Be sure to use a test account for this purpose, as you will need to share the credentials for this account with the certification team.
-
Log in to the Alexa app with the Amazon developer account you used to create the skill.
Important: You must use your Amazon developer account for this step. - Enable your skill in the Alexa app and start the account linking flow.
- Complete the account linking flow by using the test account from step 1. Continuing the earlier example, use your test credentials to link the skill to "Ride Hailer".
- Test that the skill is successfully linked to the other service. Your Amazon developer account should now be connected with the test account in the other service.
- When you submit your skill, provide the credentials (account and password) for the test account in the Testing Instructions field.
Test | Expected Results | |
---|---|---|
1. |
In the developer console, review the TOOLS > Account Linking page for the skill and verify that account linking is configured correctly. |
|
2. |
In the Alexa app, review the detail card for your skill. Verify that the skill description explains that you must log in to the other system to use the skill. |
|
3. |
Log in to the Alexa app as an Alexa user who hasn't yet enabled the skill, or log in and then disable the skill. Enable the skill, but don't complete the account linking process, leaving the skill enabled but not linked.
|
|
4. |
Disable the skill or log in to the Alexa app as an Alexa user who hasn't yet enabled the skill. Enable the skill. After you see the login page for your skill, click all links to verify that they navigate to new and correct pages. |
|
5. |
Disable the skill or log in to the Alexa app as an Alexa user who hasn't yet enabled the skill. Enable the skill. After you see the login page for your skill, log in and verify the account linking flow. |
|
Audio playback
Complete these tests if your skill uses the AudioPlayer interface to stream audio.
Test | Expected Results | |
---|---|---|
1. |
Set the volume on the device to 3 and invoke an intent that streams the audio. Listen for several minutes. |
|
2. |
While listening to the audio playback, invoke the following built-in intents:
|
|
3. |
While listening to the audio playback, use a remote control or other hardware device to pause and resume the audio. |
Be sure your skill properly handles |
4. |
While listening to the audio playback, invoke the following built-in intents:
|
Each command either does something that makes sense for your skill (such as "الجاية" advancing to the next track in a playlist) or is handled gracefully. No unexpected errors occur. See Configuring Your Skill for the AudioPlayer Directives. |
5. |
If your skill plays audio that contains a wake word, play that audio on an Alexa device. |
If your skill plays audio that contains a wake word, the wake word in the audio might wake up the device. Make sure that your skill's audio does not wake up the device. One way to do this is to make sure that there are no pauses after the wake word. |
Test | Expected Results | |
---|---|---|
1. |
Set the volume on the device to 3 and invoke an intent that streams the audio. Listen for several minutes. |
|
2. |
While listening to the audio playback, invoke the following built-in intents:
|
|
3. |
While listening to the audio playback, use a remote control or other hardware device to pause and resume the audio. |
Be sure your skill properly handles |
4. |
While listening to the audio playback, invoke the following built-in intents:
|
Each command either does something that makes sense for your skill (such as "volgende" advancing to the next track in a playlist) or is handled gracefully. No unexpected errors occur. See Configuring Your Skill for the AudioPlayer Directives. |
5. |
If your skill plays audio that contains a wake word, play that audio on an Alexa device. |
If your skill plays audio that contains a wake word, the wake word in the audio might wake up the device. Make sure that your skill's audio does not wake up the device. One way to do this is to make sure that there are no pauses after the wake word. |
Test | Expected Results | |
---|---|---|
1. |
Set the volume on the device to 3 and invoke an intent that streams the audio. Listen for several minutes. |
|
2. |
While listening to the audio playback, invoke the following built-in intents:
|
|
3. |
While listening to the audio playback, use a remote control or other hardware device to pause and resume the audio. |
Be sure your skill properly handles |
4. |
While listening to the audio playback, invoke the following built-in intents:
|
Each command either does something that makes sense for your skill (such as "next" advancing to the next track in a playlist) or is handled gracefully. No unexpected errors occur. See Configuring Your Skill for the AudioPlayer Directives. |
5. |
If your skill plays audio that contains a wake word, play that audio on an Alexa device. |
If your skill plays audio that contains a wake word, the wake word in the audio might wake up the device. Make sure that your skill's audio does not wake up the device. One way to do this is to make sure that there are no pauses after the wake word. |
Test | Expected Results | |
---|---|---|
1. |
Set the volume on the device to 3 and invoke an intent that streams the audio. Listen for several minutes. |
|
2. |
While listening to the audio playback, invoke the following built-in intents:
|
|
3. |
While listening to the audio playback, use a remote control or other hardware device to pause and resume the audio. |
Be sure your skill properly handles |
4. |
While listening to the audio playback, invoke the following built-in intents:
|
Each command either does something that makes sense for your skill (such as "next" advancing to the next track in a playlist) or is handled gracefully. No unexpected errors occur. See Configuring Your Skill for the AudioPlayer Directives. |
Test | Expected Results | |
---|---|---|
1. |
Set the volume on the device to 3 and invoke an intent that streams the audio. Listen for several minutes. |
|
2. |
While listening to the audio playback, invoke the following built-in intents:
|
|
3. |
While listening to the audio playback, use a remote control or other hardware device to pause and resume the audio. |
Be sure your skill properly handles |
4. |
While listening to the audio playback, invoke the following built-in intents:
|
Each command either does something that makes sense for your skill (such as "nächste" advancing to the next track in a playlist) or is handled gracefully. No unexpected errors occur. See Configuring Your Skill for the AudioPlayer Directives. |
Test | Expected Results | |
---|---|---|
1. |
Set the volume on the device to 3 and invoke an intent that streams the audio. Listen for several minutes. |
|
2. |
While listening to the audio playback, invoke the following built-in intents:
|
|
3. |
While listening to the audio playback, use a remote control or other hardware device to pause and resume the audio. |
Be sure your skill properly handles |
4. |
While listening to the audio playback, invoke the following built-in intents:
|
Each command either does something that makes sense for your skill (such as "अगला" advancing to the next track in a playlist) or is handled gracefully. No unexpected errors occur. See Configuring Your Skill for the AudioPlayer Directives. |
5. |
If your skill plays audio that contains a wake word, play that audio on an Alexa device. |
If your skill plays audio that contains a wake word, the wake word in the audio might wake up the device. Make sure that your skill's audio does not wake up the device. One way to do this is to make sure that there are no pauses after the wake word. |
Test | Expected Results | |
---|---|---|
1. |
Set the volume on the device to 3 and invoke an intent that streams the audio. Listen for several minutes. |
|
2. |
While listening to the audio playback, invoke the following built-in intents:
|
|
3. |
While listening to the audio playback, use a remote control or other hardware device to pause and resume the audio. |
Be sure your skill properly handles |
4. |
While listening to the audio playback, invoke the following built-in intents:
|
Each command either does something that makes sense for your skill (such as "la prossima" advancing to the next track in a playlist) or is handled gracefully. No unexpected errors occur. See Configuring Your Skill for the AudioPlayer Directives. |
Test | Expected Results | |
---|---|---|
1. |
Set the volume on the device to 3 and invoke an intent that streams the audio. Listen for several minutes. |
|
2. |
While listening to the audio playback, invoke the following built-in intents:
|
|
3. |
While listening to the audio playback, use a remote control or other hardware device to pause and resume the audio. |
Be sure your skill properly handles |
4. |
While listening to the audio playback, invoke the following built-in intents:
|
Each command either does something that makes sense for your skill (such as "次" advancing to the next track in a playlist) or is handled gracefully. No unexpected errors occur. See Configuring Your Skill for the AudioPlayer Directives. |
Test | Expected Results | |
---|---|---|
1. |
Set the volume on the device to 3 and invoke an intent that streams the audio. Listen for several minutes. |
|
2. |
While listening to the audio playback, invoke the following built-in intents:
|
|
3. |
While listening to the audio playback, use a remote control or other hardware device to pause and resume the audio. |
Be sure your skill properly handles |
4. |
While listening to the audio playback, invoke the following built-in intents:
|
Each command either does something that makes sense for your skill (such as "next" advancing to the next track in a playlist) or is handled gracefully. No unexpected errors occur. See Configuring Your Skill for the AudioPlayer Directives. |
Test | Expected Results | |
---|---|---|
1. |
Set the volume on the device to 3 and invoke an intent that streams the audio. Listen for several minutes. |
|
2. |
While listening to the audio playback, invoke the following built-in intents:
|
|
3. |
While listening to the audio playback, use a remote control or other hardware device to pause and resume the audio. |
Be sure your skill properly handles |
4. |
While listening to the audio playback, invoke the following built-in intents:
|
Each command either does something that makes sense for your skill (such as "next" advancing to the next track in a playlist) or is handled gracefully. No unexpected errors occur. See Configuring Your Skill for the AudioPlayer Directives. |
Screen display and navigation
Complete these tests if your skill uses Alexa Presentation Language or the deprecated Display interface to display content on Alexa-enabled devices with screens. Pay particular attention to the interaction of voice, tap, audio, and video in your skill.
Test | Expected Results | |
---|---|---|
1. |
Open the skill by using an Alexa-enabled device with a screen and interact with the skill for several minutes, paying particular attention to the user experience. |
|
2. |
For Echo Show or Fire TV Cube, place the device at a 7-foot or 2-meter distance from the viewer. Interact with the skill for several minutes. |
|
3. |
Exit the skill by using "stop" or "cancel" by voice with the same tap screen inputs on the screen. |
|
Video playback
In addition to the screen display requirements, some of which also apply to video, also complete these tests if your skill uses the VideoApp interface to stream video.
Test | Expected Results | |
---|---|---|
1. |
Set the volume on the device to 3 and invoke an intent that streams the video. Listen for several minutes. |
|
2. |
During video playback, use the progress bar on the screen to navigate forward and backward. |
Verify that the video resumes without error. |
Duplicates of samples or templates
Make sure your skill isn't an exact duplicate of any of the Amazon-provided samples or templates. You can use the samples and templates as a starting point for your skill, but the finished skill must be different.
For example, if you use the Trivia sample to create a trivia skill, your skill must have:
- A different name
- A different set of questions
Skill samples are available on GitHub in the alexa-samples organization.
Device location
Complete these tests if your skill uses the Device Settings REST API to obtain the location of the user's device.
Test | Expected Results | |
---|---|---|
1. |
Enable the skill and grant the skill the requested permissions for the device location. Test the functionality that's dependent on the device location and note the responses. |
|
2. |
Enable the skill, but don't grant the device location permissions.
|
|
3. |
If applicable, change the device location to a location or address that's not supported by the skill, and then enable the skill and grant permissions. Invoke the intents that require the device location. For example, if your skill provides food delivery in a particular city, set the device location to an address far outside your delivery area, and then invoke the intents to place a delivery order. |
The skill informs the user that the skill doesn't support their location. No errors are returned. |
Test skills
Alexa skills submitted for certification should have complete functionality and shouldn't be test, trial, sample, or otherwise appear to be under development. The information in the distribution section of the developer console shouldn't indicate that the skill is a trial or sample skill. For details about testing skills, see Test Your Skill.
Customer Profile API
Complete these tests if your skill uses the Customer Profile REST API to obtain the customer contact information of the user. For more details about how to use the customer profile information, see Request Customer Contact Information for Use in Your Skill.
If your skill doesn't use the Customer Profile API, disable this permission.
To turn off the customer contact permissions in the Alexa developer console
- Edit your skill in the developer console.
- Navigate to the Build > Permissions page
- Disable each of the following:
- Customer Email Address
- Customer Phone Number
- Customer Name
Test | Expected Results | |
---|---|---|
1. |
Enable the skill and grant the skill the requested permissions like customer name, customer email address, and customer phone number, as applicable for your skill. Test the functionality that's dependent on the customer contact information. |
Verify all skill responses are appropriate with regards to the customer contact information permission. For example, the weather information requested was sent to the correct email address, based on the permissions that the skill requires. |
2. |
Enable the skill, but don't grant the customer contact information permissions.
|
Verify the following items:
|
3. |
Change the customer contact information, and then enable the skill and grant permissions. Invoke the intents that require the customer contact permission. |
All skill responses are appropriate with regards to the updated customer contact information and don't return any error messages. |
Related topics
- Policy Requirements
- Security Testing
- Voice Interface and User Experience Testing
- Submit Skills for Certification in the Alexa Developer Console
Last updated: May 01, 2024