Test Wake Word Detection Delay (WWDD)
This page describes how to perform Wake Word Detection Delay (WWDD) testing for potential Alexa Built-in devices. WWDD is the minimum time that a particular device requires a user to pause between saying "Alexa" and their request for AVS to reliably receive the request. For on-device clients with microphones (Alexa Built-in devices), Amazon requires that the WWDD is no more than one second. Ideally, you should ideally implement the device audio input in such a way that the WWDD is zero.
Consider this example: "Alexa, 2 plus 7 plus 1."
The device must hear the entire request: "2 plus 7 plus 1." If the device hears any less than that, AVS can't provide an accurate response for the request. If the device requires significant delay to accurately capture the whole request, saying the wake word creates a poor experience for the customer.
For more guidance, see Enable Cloud-Based Wake Word Verification and Requirements for Cloud-Based Wake Word Verification.
Test room configuration
For WWDD testing, place the Speech Speaker in a single location: 0.9 m from the device along the 90-degree path. To make sure that the DUT is able to respond, say something like "Alexa, what time is it?"
For full instructions, learn how to Set up your Test Environment.
To help test a device for WWDD, Amazon provides utterance audio files with the same request, but with increasing amounts of delay between "Alexa" and the start of the request. For example: "Alexa, (increasing delay here) 2 plus 7 plus 1."
When testing, open the audio file with a particular delay built into the utterance, and then play that audio file 10 times. Each time, use the scoresheet to characterize the waking and the response of the device.
How to use the utterance files
As you play through the audio files, note which file first consistently yields the expected response. The filename of the audio file indicates the time delay, giving you the minimum time delay, or the WWDD, for the device being tested.
How to use the scoresheet
Open the WWDD tab of the scoresheet. The row headings on the left are varying amounts of wake word delay. These numbers appear in the file names found in the WWDD_utterances folder on the Speech Laptop defined in Equipment and materials.
Here is how you characterize the waking and response:
- 1: AVS responds with the expected answer, for example, "The answer to your calculation is 10."
- p: The response shows that the start of the request was missed, but the domain is correct, for example, "The answer to your calculation is 8." Another typical response for a device failing WWDD would be "7 plus 1 is 8" or "plus 1 is 1."
- w: The device woke up, but it had no response. Or, the response was both wrong and not in the expected domain.
- 0: The device failed to wake up.
In the following example, for the audio file with the "0.00" delay in the utterance, the DUT frequently missed the start of the request and returned an incorrect response within the correct domain. The DUT first achieved all scores of 1 for the 0.05 second delay. As support for the prior result, testing was repeated with 0.10 second delay. Therefore, the official WWDD value for this device is 0.05 seconds.
In these steps, assess the DUT for WWDD by playing each audio file 10 times, and then annotate the results in the scoresheet provided.
Prepare the test files
To play the WWDD utterance files: On the Speech Laptop, open the WWDD_utterances folder. To make sure that the device responds to an audio file, repeat the following steps as needed:
- Load and play the audio file that has 0.2 seconds of delay.
- Adjust the volume so that an SPL reading at the microphone array of the DUT is roughly 75 dBC. (The intended result is that the device has no difficulty hearing the request under ideal conditions.)
- Open the scoresheet.
- In the Information tab of the scoresheet, enter a name for your device in place of Name DUT.
- Select the WWDD tab. (The name you typed in the Information tab now appears in several other places throughout the scoresheet, such as the WWDD tab.)
- In the Information tab, fill in all remaining fields except for "Wake Word Detection Delay"). To find the Serial Number, you can use either of these approaches:
- Log in to the Amazon.com account associated with the DUT. Visit Manage Your Account and Devices, then Your Devices, then Actions. There, you should find the serial number.
- In the Alexa app account that the device is registered to, visit Settings, then Devices, and select the DUT from the list. At the bottom of the page, you'll find the serial number.
Test for each audio file
- Load and play the WWDD utterance audio file with the shortest delay (0.00 seconds).
- In the scoresheet, characterize the response (1, p, w, 0).
- For this same audio file, repeat this process, playing the file and characterizing the response, nine more times.
- Using the next (longer delay) audio file, repeat these steps until your scoresheet contains a row with all "1s."
- Repeat these steps with one more audio file. This test is complete when you have two rows with all "1s" in your scoresheet.
- In the Information tab of the scoresheet, in the WWDD row, note the delay associated with the first row of all "1s."
View the results
For your device to pass, the recorded wake word detection delay must be no more than 1 second. Ideally, the device audio input is implemented in such a way that the WWDD is effectively zero.
|WWDD - Wake Word Detection Delay||No more than 1 second|
See Amazon's passing criteria for all acoustic tests.