Music Skill Testing Guide

Before you submit your music skill for certification, you should test your skill. To test your skill, you can do any or all of the following:

  • Run validation tests
  • Perform functional tests
  • Run a beta test

We recommend that you test your skill in all of these ways before you submit it for certification.

Run validation tests

You can use the Alexa Skill Validation API to help determine if your skill's Lambda correctly supports the music skill APIs. The API runs several tests and produces an overall status of either SUCCESSFUL or FAILED. To pass certification, your skill must have an overall status of SUCCESSFUL. For more information, see supported validations.

You can use the validation API via the Alexa Skills Kit Command Line Interface (ASK CLI). For more information, see validate in the ASK CLI command reference.

When the validation is complete, you receive the overall status and the results for each test that was executed. The number of tests that are executed depends on which APIs your skill has implemented.

The following examples show the different results you might receive for each test when you use the validate command in the ASK CLI.

A required test passed

{
    "title": "<test name>",
    "status": "SUCCESSFUL",
    "importance": "REQUIRED"
}

A required test failed

A test can fail for many reasons. The description of a failed test explains the reasons for the failure. To pass certification, your skill must pass all of the required validation tests.

{
    "title": "<test name>",
    "status": "FAILED",
    "importance": "REQUIRED"
}

A recommended test passed

{
    "title": "<test name>",
    "status": "SUCCESSFUL",
    "importance": "RECOMMENDED"
}

A recommended test failed

Your skill can fail a recommended validation test and still pass certification. However, in this case some APIs might not work as expected and might negatively impact users using your skill. We recommend that you fix all failed validation tests before submitting your skill for certification.

{
    "title": "<test name>",
    "status": "FAILED",
    "importance": "RECOMMENDED"
}

Perform functional tests

Before you submit your music skill for certification, you should perform functional testing to make sure your skill works as you intend.

The following are our recommended functional tests for music skills. You should perform these tests manually to evaluate your skill from the end-user perspective. You should complete the test cases that correspond to features that your skill supports. Your music skills should pass these tests before you submit the skill for certification.

For the "play by …" tests (track, album, artist, and so on), make sure to test a variety of items for each catalog type to make sure the items play accurately. During certification, testers try a wide variety of items and expect each one to play accurately.

Test Utterance Expected result
Play music "Alexa, play music on skill name." Songs play without any additional input from the user.
Play by track "Alexa, play track name on skill name." The requested track plays.
Play by album "Alexa, play the album album name on skill name." The requested album plays.
Play by artist "Alexa, play artist name on skill name." A song by the requested artist plays.
Play by genre "Alexa, play genre on skill name." Songs from the requested genre play.
Play by playlist "Alexa, play the playlist playlist name on skill name." Songs from the requested playlist play.
Play by station "Alexa, play station name on skill name." The requested station plays.
Play popular "Alexa, play popular songs on skill name." A playlist of popular songs plays.
Play latest "Alexa, play the latest songs on skill name." A playlist of the latest songs plays.
Pause (While music is playing) "Alexa, pause." Music pauses.
Resume (After pausing music) "Alexa, resume." Music resumes playing.
Stop (While music is playing) "Alexa, stop." Music stops playing.
Next (While music is playing) "Alexa, next." The next song in the queue plays.
Previous (While music is playing) "Alexa, previous." The previous song in the queue plays.
Loop (While a queue is playing) "Alexa, loop." After the final track in the queue finishes, the queue plays again beginning with the first item from the queue.
Repeat (While music is playing) "Alexa, repeat." After the current songs finishes, the same song plays again.
Shuffle (While music is playing) "Alexa, shuffle." The order of songs in the queue changes randomly.
Accept positive feedback (While music is playing) "Alexa, I like this song." The skill accepts the positive feedback gracefully.
Accept negative feedback (While music is playing) "Alexa, I don't like this song." The skill accepts the negative feedback gracefully.
Play continuously for a long duration "Alexa, play [station name or playlist name or genre or artist name] on skill name." Music plays for a long duration (at least 15 minutes).
Explicit language filter (After enabling the explicit language filter) "Alexa, play the song name of a song with explicit lyrics on skill name." The clean (not explicit) version of the requested song plays, if one is available. If a clean version is not available, no content plays.

Run a beta test

You can set up and run a beta test to learn about your skill's user experience from real users while the skill is still in development, before you submit it for certification and make it available to the general public. Your beta testers can include your friends or family, social network contacts, existing users, or anybody for whom you have an email address and that is willing to test your skill and provide feedback. You can invite up to 500 beta testers per skill. At any time you can add testers, remove testers, or end a test. You can see tester feedback in the Alexa Skills Kit developer console, and can view feedback throughout the testing cycle.

For more information about setting up and running a beta test for your skill, see Skill Beta Testing for Alexa Skills.

Additional tests

In addition to the tests described in the preceding sections, you should perform some of the same tests that are applicable to custom skills. Following are links to functional tests for custom skills that are also relevant to music skills.

Policy guidelines

Make sure your skill's content meets Amazon's policy guidelines. See Policy Testing.

Security requirements

Test your skill code (Lambda function) to make sure it meets Amazon's security requirements. See Security Testing.

Skill description and detail page

Check the description that teaches users how to use your skill. See Skill Description and Detail Page in the functional testing guide for custom skills.

Account linking

Make sure you can successfully account link your skill. See Account Linking in the functional testing guide for custom skills.

Duplicate of samples or templates

Make sure your skill is not an exact duplicate of any of the Amazon-provided samples or templates. See Duplicates of Samples or Templates in the functional testing guide for custom skills.