Music Validation Certification Requirements and Troubleshooting
This page discusses how the Music Validation Tool (MVT) validates a test run and gives tips and troubleshooting advice to help your device pass its test runs for a given Music Service Provider (MSP). The MVT is a self-testing tool to help prepare your device for Alexa Music certification.
Coverage Validation
The most common cause of Coverage Validation failures is that a tester did not correctly run the test checklist for a given MSP. The causes of an incorrect test can vary. This section discusses possible coverage validation failure scenarios, including why a test failed, and steps to resolve those failures.
Coverage Validation tips
Keep the following general tips in mind when running tests for Coverage Validation:
- More utterances are better than fewer: When issuing utterance, always issue more utterances when you are unsure whether the previous utterance issued was processed correctly. Repeat the same utterance, if necessary. As long as one utterance was processed correctly, the test will pass Coverage Validation.
- Do not rush your utterances: Always wait longer than the specified duration in the MSP checklist. If an entity did not play long enough, or if it did not complete a playback, the test will fail Coverage Validation; however, the test might pass if playback runs for a longer duration than specified in the checklist.
- Follow the format of the example utterance: Make sure that you issue the entire utterance during the test run. For example, issuing "Play the song 'Sugar'" when the checklist specifies the utterance "Play the song 'Sugar' in my music library" will cause a failure. If you do not specify where to play the song from, the MVT will not recognize the utterance.
- Irregular device behavior: If a device behaves inconsistently during testing, fix the device before attempting further testing or certification. Even if you run the checklist correctly, if the device is not able to deduce the correct utterance, Coverage Validation will fail.
Avoid making the following common mistakes:
- Random utterances: Do not inject random utterances or any utterances that are not related to music playback when running the test.
- De-registering a device: Do not de-register your devices after a certification run, or Amazon will not be able to retrieve your utterances.
- Parallel tests: Do not run tests in parallel with the same account on the same MSPs across different devices.
Troubleshoot common Coverage Validation failures
The following table outlines some of the most common Coverage Validation failures:
MSP | Tests | Possible Failure Reason | Steps to Resolve |
---|---|---|---|
TuneIn | PlayTuneInLiveCoverageRule | Did not play enough live stations. | Play enough unique live stations to meet the required number (2) of different TuneIn stations, cannot repeat the same station name. |
Did not have enough wait time. | Ensure each station has played past the required wait times on the checklist. | ||
PlayTuneInProgramCoverageRule | Often, the user did not specify the phrase "the program" in the utterance. |
Say "Alexa, play the program 'Program Name' on TuneIn". |
|
Did not play enough custom stations |
Play at least 2 different custom stations. | ||
IHeartRadio | PlayLiveStationCoverageRule | Did not play enough live stations. | Play at least 5 unique live IHeartRadio stations. Do not repeat the same station name. |
Did not have enough wait time. | Ensure each station has played past the required wait times on the checklist. | ||
PlayCustomStationCoverageRule |
Did not play enough custom stations. |
Play at least 2 custom stations. | |
Amazon Music |
PlayCloudPlayerAlbumCoverageRule | Did not request an album. |
Say "Alexa, play the album 'Album Name' from my music library". |
Could not resolve to cloud player. | Say "Alexa, play the album 'Album Name' from my music library".</strong>". | ||
PlayCloudPlayerPlaylistCoverageRule | Did not request a playlist. | Say "Alexa, play the playlist 'Playlist Name' from my music library". | |
Could not resolve to cloud player. | Say "Alexa, play the playlist 'Playlist Name' from my music library". | ||
PlayCloudPlayerSongCoverageRule | Did not request a song. | Say "Alexa, play the song 'Song Name' by 'Artist Name' from my music library." | |
Could not resolve to cloud player | Say "Alexa, play the song 'Song Name' by 'Artist Name' from my music library." | ||
PlayPrimePlaylistCoverageRule |
Did not request a playlist. | Ensure you say "Alexa, play the playlist Top Prime Songs from Prime" or whichever playlist you wish to test. | |
Could not resolve to Prime. | Say "Alexa, play the playlist 'Playlist Name' from from Prime." | ||
PlayPrimeSongCoverageRule |
Did not request a song. |
Say "Alexa, play the song 'Song Name' by 'Artist Name' on Prime." |
|
Could not resolve to Prime. | Say "Alexa, play the song 'Song Name' by 'Artist Name' on Prime." | ||
PlayPrimeStationCoverageRule |
Did not request a station. | Say "Alexa, play the 'Station Name' station on Prime." | |
Could not resolve to Prime. | Say "Alexa, play the 'Station Name' station on Prime." |
If you are sure that you followed the checklist exactly, and Coverage Validation still fails, submit a support request through the Contact Amazon form.
AVS AudioPlayer API Validation
The goal of a validation run is to test the the device firmware in quiet, stable test environment. For the most accurate firmware testing results, test the device in a quiet environment with a stable network connection and do not proceed with the run if you notice functional issues, such as high latency, stutters, or errors. Although Amazon provides a lenience to account for these kind of issues, we still ask for sufficient evidence to conclude why an issue is not a firmware bug if you disagree with the test results. Use your device logs to help you determine the root cause of your test run failure. Alternately, if you are confident that an error is not a firmware bug and could be related to the test environment, re-run the certification test.
Validation rules
The following table explains the rules that are validated during AVS AudioPlayer API validation and shows the order of the rules as they are validated:
Rule | Validation description | Blocker? | Reason for blocking further validation |
---|---|---|---|
Event Sequencing | Validates the event sequence and event responses to directives based on the AudioPlayer State machine model. Directives would determine which events should be sent. | Yes |
Because the MVT has no visibility into what events actually occur on a device, it relies on the device reporting events in the correct order to determine the changes in the playback state, such as Start, Finish, Stop, or Error etc. For example, if a device fails to send a PlaybackStarted rule, the MVT cannot determine when the song started and therefore cannot validate the offset. Because all other rules include offset validation, they would also fail, so the MVT blocks further validation. |
PlaybackNearlyFinished |
|
No | N/A |
PlaybackStopped |
|
Yes | Prevents validation of ProgressReportDelayElapsed and ProgressReportIntervalElapsed rules because the offset calculation is inaccurate. |
PlaybackFinished |
|
Yes | Prevents validation of ProgressReportDelayElapsed and ProgressReportIntervalElapsed rules because the offset calculation is inaccurate. |
PlaybackStarted |
|
Yes | Prevents validation of ProgressReportDelayElapsed and ProgressReportIntervalElapsed rules because the offset calculation is inaccurate. |
ProgressReportDelayElapsed |
|
No | N/A |
ProgressReportIntervalElapsed |
|
No | N/A |
Time range validation
The MVT validates the time range for a certification rule if the device sends an event within the acceptable time range (after adjusting for pauses and delays from the utterance and TTS where applicable).
The following causes of pauses/delays are considered legitimate and do not count against the validation:
- Cloud side processing delay = 500ms per utterance
- Utterance audio stream duration
- Spoken prelude
- Speech Synthesis Markup Language (SSML) playback
Different events use different references for calculating the acceptable time range. The MVT applies a lenience function to the expected value and only flags excessive latency.
The following table shows the reference used and ideal expected delay for each event, not accounting for device-side latency:
Event | Reference | Ideal expected delay |
---|---|---|
Playback Started when starting from stopped state or replacing currently playing stream. | Play directive timestamp. | TTS duration + SSML playback duration. |
Playback Started when continuing to the enqueued next stream after finishing the previous one. | PlaybackFinished timestamp. | TTS duration + SSML playback duration. |
PlaybackFinished | No reference | (No validation) |
PlaybackStopped when handling a stop directive. | Stop directive timestamp | 0 |
PlaybackStopped when replacing current stream. | Play directive timestamp. | 0 to (TTS duration + SSML playback duration) |
ProgressReportDelayElapsed | PlaybackStarted timestamp. | Progress delay value + TTS duration + SSML playback duration |
ProgressReportIntervalElapsed | PlaybackStarted timestamp | Cumulative pause duration + ProgressInterval * N (for the nth ProgressReportIntervalElapsedEvent) |
Lenience
The MVT calculates Lenience as 5s + pause duration < 10% of the expected delay from reference < 8s + pause duration.
For example, consider an utterance where the time between the play directive and the PlaybackStarted event was 8 seconds, which is above the maximum 3 seconds lenience for this event. This case would be tagged as a DELAYED failure by the MVT.
The MVT calculates two different types of Lenience:
- PlayPeriod Lenience: Used for offsets that depend on the duration of how long a stream played.
- Progress Report Lenience: Used for events that have a offsets which are pre-determined by the play directive.
Offset validation
While a stream is playing, interruptions can pause a device and affect the offset value; however interruptions do not affect the offset if the device attenuates and continues playing. Because the MVT does not have access to the PlaybackPaused and PlaybackResumed events, the MVT estimates a best guess at a reasonable offset. To compensate, the MVT uses a lower bound and an upper bound to determine an acceptable range for the offset.
- Upper bound for offset = Timestamp of event - Timestamp of PlaybackStarted event + offset of PlaybackStartedEvent
- Lower bound for offset = Upper bound for offset - Estimated pause duration (utterance speak duration + TTS render duration)
- Acceptable range = Lower bound - Lenience value <= Reported offset <= Upper bound + Lenience value
The lenience allowed for the offset differs by scenario:
- Playback Started: No lenience. The PlaybackStarted event must report the exact offset specified in the play directive.
- PlaybackFinished: PlayPeriod lenience.
- PlaybackStopped: PlayPeriod lenience.
- PlaybackNearlyFinished: PlayPeriod lenience.
- ProgressReportDelayElapsed (PRDE): ProgressReport lenience.
- ProgressReportIntervalElapsed (PRIE): ProgressReport lenience.
Failure Tags
To help with troubleshooting, each validation failure will be tagged with one or more "failure tags". The following table explains the meaning behind each of these failure tags:
Tag | Description |
---|---|
MISSING | The event was not sent at the expected interval. |
INVALID | If a device stops (PlaybackStopped) unexpectedly or there is an PlaybackStarted without a Play/Stop directive |
EXTRA_EVENT | The MVT detected a duplicate event. Remove the extra event. |
UNEXPECTED_EVENT | The sequence of events is incorrect, and the MVT cannot determine the cause. Investigate your AudioPlayer Interface implementation to help troubleshoot issue. |
TOO_EARLY | Event sent too early. Applicable only to PRDE and PRIE. |
DELAYED | Event sent too late. Typical acceptable latency is 2s (to reach AlexaService) after the actual event happening on the device. |
INCORRECT_OFFSET_TOO_LOW | Offset below acceptable lower bound. |
INCORRECT_OFFSET_TOO_HIGH | Offset above acceptable upper bound. |
UNEXPECTED_PLAYBACK |
The device played a URL audio resource with no spoken prelude with no enqueued play directive. |
Incorrect Offset Too Low/Too High
When an event is sent with an offset that is not accurately calculated, it will be flagged as INCORRECT_OFFSET_TOO_LOW or INCORRECT OFFSET TOO HIGH by the MVT. For example, if the offset expected for PlaybackStopped is 50753 milliseconds but is instead reported as 42941 ms, the event does not meet accepted threshold for offset reporting and will fail.
Extra Events
When a device erroneously sends a duplicate event, the duplicate will be flagged as EXTRA_EVENT by the MVT.
Unexpected Playback
When a PlaybackStarted event is sent without an Enqueued directive, it will be flagged as UNEXPECTED_PLAYBACK by the MVT. Consider an example where the PlaybackStarted event occurred because of a Play directive that had been correctly sent. However, a separate PlaybackStarted event does not have a Play Directive preceding it; therefore this PlaybackStarted event is flagged as Unexpected Playback.
Other causes of failure
Sometimes failures can be caused by circumstances not related to the device itself:
- Network issues
- Alexa services latency
- MSP/CDN issues causing stutter
- Wrong URL sent by MSP, causing Playback error
If you think that your certification failure was caused by one of the above factors, search your device logs to confirm, then follow the two next steps:
-
Re-run the test. Non-device-related issues are typically difficult to consistently reproduce.
If the tests are causing cross region network calls, for example, calling a US-only MSP from India, then network latencies are unavoidable. Consider running the tests from same region.
-
Contact Amazon if you are confident that you followed all of the test guidelines but are still seeing unexpected validation errors.