Music Validation Certification Requirements and Troubleshooting

This page discusses how the Music Validation Tool (MVT) validates a test run and gives tips and troubleshooting advice to help your device pass its test runs for a given Music Service Provider (MSP). The MVT is a self-testing tool to help prepare your device for Alexa Music certification.

Coverage Validation

The most common cause of Coverage Validation failures is that a tester did not correctly run the test checklist for a given MSP. The causes of an incorrect test can vary. This section discusses possible coverage validation failure scenarios, including why a test failed, and steps to resolve those failures.

Coverage Validation tips

Keep the following general tips in mind when running tests for Coverage Validation:

More utterances are better than fewer: When issuing utterance, always issue more utterances when you are unsure whether the previous utterance issued was processed correctly. Repeat the same utterance, if necessary. As long as one utterance was processed correctly, the test will pass Coverage Validation.
Do not rush your utterances: Always wait longer than the specified duration in the MSP checklist. If an entity did not play long enough, or if it did not complete a playback, the test will fail Coverage Validation; however, the test might pass if playback runs for a longer duration than specified in the checklist.
Follow the format of the example utterance: Make sure that you issue the entire utterance during the test run. For example, issuing "Play the song 'Sugar'" when the checklist specifies the utterance "Play the song 'Sugar' in my music library" will cause a failure. If you do not specify where to play the song from, the MVT will not recognize the utterance.
Irregular device behavior: If a device behaves inconsistently during testing, fix the device before attempting further testing or certification. Even if you run the checklist correctly, if the device is not able to deduce the correct utterance, Coverage Validation will fail.

Avoid making the following common mistakes:

Random utterances: Do not inject random utterances or any utterances that are not related to music playback when running the test.
De-registering a device: Do not de-register your devices after a certification run, or Amazon will not be able to retrieve your utterances.
Parallel tests: Do not run tests in parallel with the same account on the same MSPs across different devices.

Troubleshoot common Coverage Validation failures

The following table outlines some of the most common Coverage Validation failures:

MSP	Tests	Possible Failure Reason	Steps to Resolve
TuneIn	PlayTuneInLiveCoverageRule	Did not play enough live stations.	Play enough unique live stations to meet the required number (2) of different TuneIn stations, cannot repeat the same station name.
	PlayTuneInLiveCoverageRule	Did not have enough wait time.	Ensure each station has played past the required wait times on the checklist.
	PlayTuneInProgramCoverageRule	Often, the user did not specify the phrase "the program" in the utterance.	Say "Alexa, play the program 'Program Name' on TuneIn".
	PlayTuneInProgramCoverageRule	Did not play enough custom stations	Play at least 2 different custom stations.
IHeartRadio	PlayLiveStationCoverageRule	Did not play enough live stations.	Play at least 5 unique live IHeartRadio stations. Do not repeat the same station name.
	PlayLiveStationCoverageRule	Did not have enough wait time.	Ensure each station has played past the required wait times on the checklist.
	PlayCustomStationCoverageRule	Did not play enough custom stations.	Play at least 2 custom stations.
Amazon Music	PlayCloudPlayerAlbumCoverageRule	Did not request an album.	Say "Alexa, play the album 'Album Name' from my music library".
	PlayCloudPlayerAlbumCoverageRule	Could not resolve to cloud player.	Say "Alexa, play the album 'Album Name' from my music library".</strong>".
	PlayCloudPlayerPlaylistCoverageRule	Did not request a playlist.	Say "Alexa, play the playlist 'Playlist Name' from my music library".
	PlayCloudPlayerPlaylistCoverageRule	Could not resolve to cloud player.	Say "Alexa, play the playlist 'Playlist Name' from my music library".
	PlayCloudPlayerSongCoverageRule	Did not request a song.	Say "Alexa, play the song 'Song Name' by 'Artist Name' from my music library."
	PlayCloudPlayerSongCoverageRule	Could not resolve to cloud player	Say "Alexa, play the song 'Song Name' by 'Artist Name' from my music library."
	PlayPrimePlaylistCoverageRule	Did not request a playlist.	Ensure you say "Alexa, play the playlist Top Prime Songs from Prime" or whichever playlist you wish to test.
	PlayPrimePlaylistCoverageRule	Could not resolve to Prime.	Say "Alexa, play the playlist 'Playlist Name' from from Prime."
	PlayPrimeSongCoverageRule	Did not request a song.	Say "Alexa, play the song 'Song Name' by 'Artist Name' on Prime."
	PlayPrimeSongCoverageRule	Could not resolve to Prime.	Say "Alexa, play the song 'Song Name' by 'Artist Name' on Prime."
	PlayPrimeStationCoverageRule	Did not request a station.	Say "Alexa, play the 'Station Name' station on Prime."
	PlayPrimeStationCoverageRule	Could not resolve to Prime.	Say "Alexa, play the 'Station Name' station on Prime."

If you are sure that you followed the checklist exactly, and Coverage Validation still fails, submit a support request through the Contact Amazon form.

AVS AudioPlayer API Validation

The goal of a validation run is to test the the device firmware in quiet, stable test environment. For the most accurate firmware testing results, test the device in a quiet environment with a stable network connection and do not proceed with the run if you notice functional issues, such as high latency, stutters, or errors. Although Amazon provides a lenience to account for these kind of issues, we still ask for sufficient evidence to conclude why an issue is not a firmware bug if you disagree with the test results. Use your device logs to help you determine the root cause of your test run failure. Alternately, if you are confident that an error is not a firmware bug and could be related to the test environment, re-run the certification test.

Validation rules

The following table explains the rules that are validated during AVS AudioPlayer API validation and shows the order of the rules as they are validated:

Rule	Validation description	Blocker?	Reason for blocking further validation
Event Sequencing	Validates the event sequence and event responses to directives based on the AudioPlayer State machine model. Directives would determine which events should be sent.	Yes	Because the MVT has no visibility into what events actually occur on a device, it relies on the device reporting events in the correct order to determine the changes in the playback state, such as Start, Finish, Stop, or Error etc. For example, if a device fails to send a PlaybackStarted rule, the MVT cannot determine when the song started and therefore cannot validate the offset. Because all other rules include offset validation, they would also fail, so the MVT blocks further validation.
PlaybackNearlyFinished	Ensures that a device only sends one PlaybackNearlyFinished before a song finishes and sends no more than one PlaybackNearlyFinished per stream. Validates that the offset is within the acceptable range (Exception:Offset issues for PlaybackNearlyFinished are only set to WARNING, and are not certification blockers.)	No	N/A
PlaybackStopped	Sent within the expected time range. Validates that the offset is within the acceptable range.	Yes	Prevents validation of ProgressReportDelayElapsed and ProgressReportIntervalElapsed rules because the offset calculation is inaccurate.
PlaybackFinished	Sent within the expected time range. Validates that the offset is within the acceptable range.	Yes	Prevents validation of ProgressReportDelayElapsed and ProgressReportIntervalElapsed rules because the offset calculation is inaccurate.
PlaybackStarted	Sent within the expected time range. Validates if offset exactly matches the play directive.	Yes	Prevents validation of ProgressReportDelayElapsed and ProgressReportIntervalElapsed rules because the offset calculation is inaccurate.
ProgressReportDelayElapsed	Sent ONLY if it needs to be sent and if required to be sent. Sent within the expected time range. Validates if offset is within the acceptable range (based on the value of Progress Delay in Play directive for Amazon Music and IHRCustom).	No	N/A
ProgressReportIntervalElapsed	The number of events sent exactly matches the number expected. Sent within the expected time range. >Validates if offset is within acceptable range (based on the value of Progress Interval in Play directive for TuneIn, IHRCustom and IHRLive).	No	N/A

Time range validation

The MVT validates the time range for a certification rule if the device sends an event within the acceptable time range (after adjusting for pauses and delays from the utterance and TTS where applicable).

The following causes of pauses/delays are considered legitimate and do not count against the validation:

Cloud side processing delay = 500ms per utterance
Utterance audio stream duration
Spoken prelude
Speech Synthesis Markup Language (SSML) playback

Different events use different references for calculating the acceptable time range. The MVT applies a lenience function to the expected value and only flags excessive latency.

The following table shows the reference used and ideal expected delay for each event, not accounting for device-side latency:

Event	Reference	Ideal expected delay
Playback Started when starting from stopped state or replacing currently playing stream.	Play directive timestamp.	TTS duration + SSML playback duration.
Playback Started when continuing to the enqueued next stream after finishing the previous one.	PlaybackFinished timestamp.	TTS duration + SSML playback duration.
PlaybackFinished	No reference	(No validation)
PlaybackStopped when handling a stop directive.	Stop directive timestamp	0
PlaybackStopped when replacing current stream.	Play directive timestamp.	0 to (TTS duration + SSML playback duration)
ProgressReportDelayElapsed	PlaybackStarted timestamp.	Progress delay value + TTS duration + SSML playback duration
ProgressReportIntervalElapsed	PlaybackStarted timestamp	Cumulative pause duration + ProgressInterval * N (for the nth ProgressReportIntervalElapsedEvent)

Note: Cumulative pause duration = Audio stream + cloud size latency + TTS render duration + SSML playback after the most recent PlaybackStarted timestamp.

Lenience

The MVT calculates Lenience as 5s + pause duration < 10% of the expected delay from reference < 8s + pause duration.

For example, consider an utterance where the time between the play directive and the PlaybackStarted event was 8 seconds, which is above the maximum 3 seconds lenience for this event. This case would be tagged as a DELAYED failure by the MVT.

The MVT calculates two different types of Lenience:

PlayPeriod Lenience: Used for offsets that depend on the duration of how long a stream played.
Progress Report Lenience: Used for events that have a offsets which are pre-determined by the play directive.

Offset validation

While a stream is playing, interruptions can pause a device and affect the offset value; however interruptions do not affect the offset if the device attenuates and continues playing. Because the MVT does not have access to the PlaybackPaused and PlaybackResumed events, the MVT estimates a best guess at a reasonable offset. To compensate, the MVT uses a lower bound and an upper bound to determine an acceptable range for the offset.

Upper bound for offset = Timestamp of event - Timestamp of PlaybackStarted event + offset of PlaybackStartedEvent
Lower bound for offset = Upper bound for offset - Estimated pause duration (utterance speak duration + TTS render duration)
Acceptable range = Lower bound - Lenience value <= Reported offset <= Upper bound + Lenience value

Note: The offset is validated only if the event is sent within the expected time range.

The lenience allowed for the offset differs by scenario:

Playback Started: No lenience. The PlaybackStarted event must report the exact offset specified in the play directive.
PlaybackFinished: PlayPeriod lenience.
PlaybackStopped: PlayPeriod lenience.
PlaybackNearlyFinished: PlayPeriod lenience.
ProgressReportDelayElapsed (PRDE): ProgressReport lenience.
ProgressReportIntervalElapsed (PRIE): ProgressReport lenience.

Failure Tags

To help with troubleshooting, each validation failure will be tagged with one or more "failure tags". The following table explains the meaning behind each of these failure tags:

Tag	Description
MISSING	The event was not sent at the expected interval.
INVALID	If a device stops (PlaybackStopped) unexpectedly or there is an PlaybackStarted without a Play/Stop directive
EXTRA_EVENT	The MVT detected a duplicate event. Remove the extra event.
UNEXPECTED_EVENT	The sequence of events is incorrect, and the MVT cannot determine the cause. Investigate your AudioPlayer Interface implementation to help troubleshoot issue.
TOO_EARLY	Event sent too early. Applicable only to PRDE and PRIE.
DELAYED	Event sent too late. Typical acceptable latency is 2s (to reach AlexaService) after the actual event happening on the device.
INCORRECT_OFFSET_TOO_LOW	Offset below acceptable lower bound.
INCORRECT_OFFSET_TOO_HIGH	Offset above acceptable upper bound.
UNEXPECTED_PLAYBACK	The device played a URL audio resource with no spoken prelude with no enqueued play directive.

Incorrect Offset Too Low/Too High

When an event is sent with an offset that is not accurately calculated, it will be flagged as INCORRECT_OFFSET_TOO_LOW or INCORRECT OFFSET TOO HIGH by the MVT. For example, if the offset expected for PlaybackStopped is 50753 milliseconds but is instead reported as 42941 ms, the event does not meet accepted threshold for offset reporting and will fail.

Extra Events

When a device erroneously sends a duplicate event, the duplicate will be flagged as EXTRA_EVENT by the MVT.

Unexpected Playback

When a PlaybackStarted event is sent without an Enqueued directive, it will be flagged as UNEXPECTED_PLAYBACK by the MVT. Consider an example where the PlaybackStarted event occurred because of a Play directive that had been correctly sent. However, a separate PlaybackStarted event does not have a Play Directive preceding it; therefore this PlaybackStarted event is flagged as Unexpected Playback.

Other causes of failure

Sometimes failures can be caused by circumstances not related to the device itself:

Network issues
Alexa services latency
MSP/CDN issues causing stutter
Wrong URL sent by MSP, causing Playback error

If you think that your certification failure was caused by one of the above factors, search your device logs to confirm, then follow the two next steps:

Re-run the test. Non-device-related issues are typically difficult to consistently reproduce.

If the tests are causing cross region network calls, for example, calling a US-only MSP from India, then network latencies are unavoidable. Consider running the tests from same region.
Contact Amazon if you are confident that you followed all of the test guidelines but are still seeing unexpected validation errors.