Your Alexa Consoles
?
Support

Functional Testing for a Custom Skill

The functional tests verify that:

  • The skill's basic functionality matches the information displayed on the skill's detail card in the Amazon Alexa app. You enter this data on the Distribution page for the skill on the developer console.

    See Define Skill Store Details and Availability for more details about defining this card.

  • The skill's core functionality works and provides useful home cards to the Amazon Alexa app.

Use the Amazon Alexa app for these tests. The Alexa app is available for Fire OS, Android, iOS, and web browsers. The Alexa app displays all of your skills currently under development, as well as all published skills.

To return to the high-level testing checklist, see Certification Requirements for Custom Skills.

3.1. Review and Test Example Phrases

Example phrases are shown on the skill's detail card in the Alexa app to help users understand how to interact with the skill. You select these phrases from your sample utterances to ensure accuracy.

These are the phrases users are most likely to try the first time they interact with the skill. Therefore, make sure that they work well and provide a good user experience.

Detail Card for a Skill in the Alexa App
Detail Card for a Skill in the Alexa App

For English-language skills, the example phrases you enter on the Distribution page are automatically validated against several criteria after you enter them. Correct any errors or warnings before your submit. Note that this does not check for all possible problems, so review the tests below as well.

Test Expected Results

1.

Open the Alexa app, click Skills and then scroll or search to find your skill.

Review the skill's detail page. Inspect all example phrases listed in the skill's detail card.

  • At least one example phrase has been provided.
  • The first phrase clearly indicates how to begin using the skill, and includes both the wake word and invocation name.
  • Example phrases containing the wake word and invocation name must use a supported format for invoking a skill. For a list of supported formats, see Understanding How Users Invoke Custom Skills.
  • The example phrases do not contain any additional information that is not intended to be spoken by users. The example phrases must function exactly as they are written.
  • The invocation name uses title case.
  • The example phrases do not contain any emoticons, symbols, or grammatical errors.
  • Each example phrase ends with punctuation.
  • The example phrases show how to launch the skill. They do not illustrate enabling the skill by voice. For example, the phrase must not be something like "Alexa, enable <skill name>."
  • The example phrases are all written in the same language used by the Alexa account. For instance, when using an account configured with German, the phrases are written in German.

2.

Edit your skill in the developer console. Your sample utterances are defined within each intent. Click the intent, review the utterances, and verify that your chosen example phrases match existing sample utterances.

For more about defining example phrases, see Define Skill Store Details and Availability: Example Phrases for Custom Skills.

If you are using the utterance for a built-in intent for an example phrase, the utterance does not need to be in your sample utterances. However, fully test the utterance and verify that your skill returns a valid response.

Each example phrase is explicitly included in the sample utterances.

For instance, if you have this example phrase:

  • Alexa, ask Tide Pooler when is high tide in Seattle.

Then you must also have this utterance:

  • OneshotTideIntent when is high tide in {City}

Any slots in the example phrase (such as Seattle in this example) must be filled with an example of a valid slot value. In this example, City is a custom slot type, and "Seattle" is a value explicitly defined for the City type.

3.

Launch the skill using each of the provided example phrases.

Each example phrase renders a contextualized, non-error response.

3.2. Skill Description and Detail Page

The skill description teaches users about what your skill can do and how to use it.

Test Expected Results

1.

Open the Alexa app, click Skills and then scroll to your skill. Review the short description shown in the list of skills

  • The short description describes the skill's core functionality.
  • The short description is written in the same language used by the Alexa account. For instance, when using an account configured with German, the skill short description is displayed in German.
  • (French) Any references to the term "skill" use "skill" and not the literal translation "compétence". Note that "skill" is feminine ("la skill" or "une skill").

2.

Open the detail card for the skill. Review the skill's full description shown on the skill's detail card.

  • The skill description captures any prerequisites needed to use the skill (such as additional hardware, software, or accounts).
  • The description describes the skill's core functionality.
  • The description does not misrepresent the skill's functionality.
  • The description is written in the same language used by the Alexa account. For instance, when using an account configured with German, the skill description is displayed in German.
  • (French) Any references to the term "skill" use "skill" and not the literal translation "compétence". Note that "skill" is feminine ("la skill" or "une skill").
  • If the skill integrates with Raspberry Pi or a similar device, the description must:
    1. Mention that a Raspberry Pi (or similar device) is required to access core functionality of the skill.
    2. Include links to documentation explaining how to configure the Raspberry Pi with the skill.
  • The description must not claim that the skill has any anti-virus features.

3.

If either the short description or full description includes any URLs (such as directing users to more information about the skill), visit the URL in a browser and verify the page opens without error.

  • The page referenced by the description opens without error.
  • The page is displayed in the same language used by the Alexa account. For instance, when using an account configured with German, the page should display in German.

4.

If your skill's detail page includes links to a privacy policy or terms of use, open the links in a browser.

  • The privacy policy / terms of use links shown on the skill detail page open without error.
  • The pages are displayed in the same language used by the Alexa account. For instance, when using an account configured with German, the privacy policy and terms of use must be in German.

5.

Review the skill title and ensure it is relevant to the core functionality of the skill.

The title is relevant to the skill description and is not misleading.

6.

Review the skill icon and ensure it is not cropped or blank.

The icon is not cropped or blank.

3.3. Home Cards

Home cards are graphical cards displayed in the Alexa app to describe or enhance the voice interaction.

Test Expected Results

1.

Walk through the skill's functionality, invoking each intent. For each response that includes a home card, review the card in the Alexa app.

For each card:

  • The card title and content are appropriate in the context of the skill's functionality.
  • The card title and content do not contain any code references or grammatical errors.
  • The card title and content are written in a language supported by Alexa. The current supported languages are English, French, and German.
  • If the card includes an image, the image is correctly displayed.

2.

If any home cards include URLs, visit each URL in a browser and verify the page opens without error.

For each URL displayed in a home card:

  • The page opens in a browser without error.
  • The page is displayed in a language supported by Alexa. The current supported languages are English, French, and German.

Note that skills normally only return home cards when responding with the information the user requested. Supportive prompts to ask the user for more information do not normally include home cards.

Home cards can be a useful tool when developing and debugging a skill, but be sure to remove any extraneous debug-type cards before submitting the skill.

3.4. Account Linking

If your skill needs to connect the identity of the end user with a user in another system (account linking), this test verifies that the skill implements the solution described in Account Linking for Custom Skills correctly. At a high level, this means:

  • When the user invokes an intent that requires authorization, the skill successfully authenticates and completes the request.
  • The skill returns the LinkAccount card if a user invokes an intent that requires authorization, but has not yet linked their account.

When submitting your skill, be sure to provide a valid set of account credentials with your testing instructions so our certification team can verify the account linking and functionality of your skill.

If your skill does not use account linking, you can skip this test.

Test Expected Results

1.

In the developer console, review the Account Linking page for the skill and verify that account linking has been configured correctly.

  • The Account Linking or Creation option is set to Yes.
  • The Authorization URI is set to the login page for your service.

2.

In the Alexa app, review the detail card for your skill. Verify that the skill description explains that you must log in to the other system to use the skill.

The description clearly explains the steps and information necessary to successfully link the accounts.

3.

Log in to the Alexa app as an Alexa user who has not yet enabled the skill.

Enable the skill, but do not complete the account linking process, leaving the skill enabled but not linked.

  1. Invoke any intents that do not require authentication and verify that they work.
  2. Invoke the intents that do require authentication and verify that the skill sends the LinkAccuont card and an appropriate prompt.
  • Each intent that does not require authentication works correctly even when accounts are not linked.
  • Each intent that does require authentication returns output speech telling the user what to do, and a LinkAccount card. Verify the card in the Alexa app.

4.

Disable the skill or log in to the Alexa app as an Alexa user who has not yet enabled the skill.

Enable the skill. Once the login page for your skill is displayed, click all links and verify that they navigate to new pages.

  • All clickable links on the login page navigate to new pages.
  • None of the clickable links open JavaScript alerts or popup windows.

5.

Disable the skill or log in to the Alexa app as an Alexa user who has not yet enabled the skill.

Enable the skill. Once the login page for your skill is displayed, log in and verify the account linking flow.

  • The login page for your skill is displayed correctly.
  • You can successfully log in using an account with the service.
  • After connecting the accounts, you can successfully invoke each intent that requires authentication.

3.5. Audio Playback

Complete these tests if your skill uses the AudioPlayer interface to stream audio.

Test Expected Results

1.

Set the volume on the device to 3 and invoke an intent that streams the audio. Listen for several minutes.

  • The audio playback is audible at volume level 3.
  • The volume of the audio playback does not vary significantly from normal Alexa text-to-speech. The volume remains consistent while playing for several minutes.
  • If the audio content includes words or speech, the content is spoken in a language supported by Alexa. The current supported languages are English, French, and German.

2.

While listening to the audio playback, invoke the following built-in intents:

  • AMAZON.PauseIntent (say "pause")
  • AMAZON.ResumeIntent (say "resume")
  • The "pause" utterance stops the audio playback.
  • The "resume" utterance resumes the audio playback.

3.

While listening to the audio playback, use a remote control or other hardware device to pause and resume the audio.

  • Pressing a pause button while audio is playing stops the playback.
  • Pressing a play button while the audio is stopped resumes the playback.

4.

While listening to the audio playback, invoke the following built-in intents:

  • AMAZON.CancelIntent (say "cancel")
  • AMAZON.LoopOffIntent (say "loop off")
  • AMAZON.LoopOnIntent (say "loop on")
  • AMAZON.NextIntent (say "next")
  • AMAZON.PreviousIntent (say "previous")
  • AMAZON.RepeatIntent (say "repeat that")
  • AMAZON.ShuffleOffIntent(say "shuffle off")
  • AMAZON.ShuffleOnIntent (say "shuffle")
  • AMAZON.StartOverIntent (say "start over")
  • AMAZON.StopIntent (say "stop")

Each command either does something that makes sense for your skill (such as "next" advancing to the next track in a playlist) or is handled gracefully. No unexpected errors occur. See Configuring Your Skill for the AudioPlayer Directives.

Test Expected Results

1.

Set the volume on the device to 3 and invoke an intent that streams the audio. Listen for several minutes.

  • The audio playback is audible at volume level 3.
  • The volume of the audio playback does not vary significantly from normal Alexa text-to-speech. The volume remains consistent while playing for several minutes.
  • If the audio content includes words or speech, the content is spoken in a language supported by Alexa. The current supported languages are English, French, and German.

2.

While listening to the audio playback, invoke the following built-in intents:

  • AMAZON.PauseIntent (say "pause" or "pause ça")
  • AMAZON.ResumeIntent (say "continue" or "reprends")
  • The "pause" utterance stops the audio playback.
  • The "resume" utterance resumes the audio playback.

3.

While listening to the audio playback, use a remote control or other hardware device to pause and resume the audio.

  • Pressing a pause button while audio is playing stops the playback.
  • Pressing a play button while the audio is stopped resumes the playback.

4.

While listening to the audio playback, invoke the following built-in intents:

  • AMAZON.CancelIntent (say "annule")
  • AMAZON.LoopOffIntent (say "arrête le mode boucle")
  • AMAZON.LoopOnIntent (say "boucle")
  • AMAZON.NextIntent (say "suivant")
  • AMAZON.PreviousIntent (say "précédent")
  • AMAZON.RepeatIntent (say "est-ce que tu peux répéter")
  • AMAZON.ShuffleOffIntent(say "arrête la lecture aléatoire")
  • AMAZON.ShuffleOnIntent (say "mélanger")
  • AMAZON.StartOverIntent (say "recommence")
  • AMAZON.StopIntent (say "arrête")

Each command either does something that makes sense for your skill (such as "next" advancing to the next track in a playlist) or is handled gracefully. No unexpected errors occur. See Configuring Your Skill for the AudioPlayer Directives.

3.6 Screen Display and Navigation

Complete these tests if your skill uses the Display interface to display content on Echo Show. Pay particular attention to the interaction of voice, touch, audio, and video in your skill.

Test Expected Results

1.

Open the skill using Echo Show and interact with the skill for several minutes, paying particular attention to the user experience.

  • All GUI content, whether in the form of static templates or video, must load within 10 seconds of the triggering voice or touch input.
  • GUI content (images and video) should not look stretched, grainy or pixelated.
  • The GUI content must not contradict the voice content (whether text-to-speech or audio) that is delivered alongside the GUI content. The GUI content does not need to match exactly, but must complement the voice content.
  • All GUI content should be relevant to the triggering voice input or touch input.
  • GUI content should not contain more than three actions per screen, if the screen is not displaying a list.
  • All actions present on the GUI should also be accessible via voice requests, meaning these actions are implemented as intents and have corresponding sample utterances in the language model.

2.

Place the Echo Show device at a 7-foot or 2-meter distance from the viewer. Interact with the skill for several minutes.

  • All GUI content should be readable from a 7-foot or 2-meter distance.

3.

Exit the skill using "stop" or "cancel" while touch screen inputs are on the screen.

  • The touch inputs should no longer be displayed once the skill exits.

3.7. Video Playback

In addition to the screen display requirements, some of which also apply to video, also complete these tests if your skill uses the VideoApp interface to stream video.

Test Expected Results

1.

Set the volume on the device to 3 and invoke an intent that streams the video. Listen for several minutes.

  • The video playback is audible at volume level 3.
  • The volume of the audio playback does not vary significantly from normal Alexa text-to-speech. The volume remains consistent while playing for several minutes.
  • If the audio content includes words or speech, the content is spoken in a language supported by Alexa. The current supported languages are English, French, and German.
  • Buffering during video playback should not exceed 10% of the total video playback time. For example, for every minute of video playback, buffering must not exceed 6 seconds.
  • During video playback, the audio should be in sync with the video.
  • The video is clearly visible and not grainy, pixelated, or distorted.

2.

While the video plays, use the progress bar on the touchscreen to navigate forward and backward.

Verify that the video resumes without error.

3.8. Duplicates of Samples or Templates

Make sure your skill is not an exact duplicate of any of the Amazon-provided samples or templates. You can use the samples and templates as a starting point for your skill, but the finished skill must be different.

For example, if you use the Trivia sample to create a trivia skill, your skill must have:

  • A different name
  • A different set of questions

The samples and templates are available on GitHub:

3.9 Device Location API

Complete these tests if your skill uses the Device Address API to obtain the location of the user's device.

Test Expected Results

1.

Enable the skill and grant the skill the requested permissions for the device location. Test the functionality that is dependent on the device location and note the responses.

  • All skill responses are appropriate with regards to the device location. For example, a weather skill that uses the device location is expected to give weather information for the city specified in the device location (unless the user provides a different location in an utterance).
  • If the skill communicates the device location back to the user, it matches the specified device location.

2.

Enable the skill, but do not grant the device location permissions.

  1. Invoke any intents that do not require the location and verify that they work.
  2. Invoke the intents that do require the location and verify the response.
  • Each intent that does not require the device location works correctly even the user has not granted the permissions.
  • Each intent that does require the device location returns output speech telling the user to grant the permissions, and a permissions card. Verify the card in the Alexa app.

3.

If applicable, change the device location to a location or address that is not supported by the skill, then enable the skill and grant permissions. Invoke the intents that require the device location.

For example, if your skill provides food delivery in a particular city, set the device location to an address far outside your delivery area, then invoke the intents to place a delivery order.

The skill informs the user that skill does not support their location. No errors are returned.

3.10 Echo Buttons

Complete these tests if your skill works with Echo Buttons.

Subject Test Expected Results

Roll call

Launch the skill with buttons paired, but without any buttons awake.

When the skill first launches, it must begin with a roll call that prompts users to wake their paired buttons.

Note: Skills can skip roll call if launched within ten minutes of a previous session of the same skill.

Roll call timeout

Launch the skill without any buttons awake. During roll call, do not press any buttons.

If users do not press buttons in response to roll call, the skill must respond gracefully. The skill can do one of the following: 1) re-prompt users to press their buttons, 2) close the skill after informing users that buttons are required to use the skill, or 3) ask users to use the skill without buttons.

The skill must not close or provide a hard error immediately following the timeout.

Valid inputs

Launch the skill with the required number of buttons per the skill's metadata. Following all customer-facing instructions in the skill's prompts and metadata, navigate the skill for up to ten minutes. Examine the skill's responses to valid button presses throughout navigation.

The skill must return the expected responses when the user makes valid button presses.

Note: This test applies equally to skills for which buttons are required and buttons are optional.

Empty inputs

Launch the skill with the required number of buttons per the skill's metadata. Navigate past roll call. When the skill prompts users to press buttons, do not press any buttons.

For both button-required and button-optional skills, if users do not press buttons when the skill expects button presses, the skill must re-prompt users to press the buttons.

The skill must not close or return a hard error.

Player mode

Launch the skill with the required number of buttons per the skill's metadata. Navigate past roll call and play the skill in single-player and multi-player modes, if applicable.

In both modes, the skill must handle responses from all users in a mutually exclusive manner and must be able to maintain progress for all users independently.

Dropping a player

(Only applies if the skill supports dropping a player)
Launch the skill with the required number of buttons per the skill's metadata. Navigate past roll call and in the middle of gameplay, drop a player using the applicable custom intent.

After dropping a player in the middle of gameplay, the rest of the players must be able to continue playing the game. If a scorecard is available, the scorecard must not reset when the player is dropped.

Note: When only two players play and a player drops, it depends upon the skill design whether the skill has to end or the skill can continue with a single player.

LED colors

Launch the skill with the required number of buttons per the skill's metadata. Navigate past roll call and observe the LED animations on the buttons during gameplay.

The skill must use LED colors on the buttons to provide an interactive experience to the users.

For example, in a Q&A skill, the skill might light up buttons with a color depending on whether the user provides a correct or incorrect answer. The color must be consistent when the answer is incorrect vs. correct. As another example, in a multi-player skill, the skill might light up Player 1's button to indicate that Player 1 must respond.

Optional

(Only applies to skills where buttons are optional)
Launch the skill with the required number of buttons per the skill's metadata. Examine whether the user is able to use the skill without buttons.

The user must be able to use the skill without buttons.

The skill must also ask the user before roll call or during roll call whether the user wants to play with or without buttons.