Alexa Acoustic Testing Guide
To keep the mutual trust of our customers for the expected Alexa experience, the Alexa Voice Service (AVS) testing team verifies the performance of acoustic systems for Alexa Built-in devices. Acoustic performance is critical to make sure that wake word recognition performs as expected and that a device handles user requests appropriately.
The following sections describe how to conduct your own acoustic testing in a way that emulates how Amazon tests devices, including the following information:
- The room configurations that Amazon uses for testing
- The equipment Amazon recommends to recreate Amazon test conditions
- Test steps
Amazon also provides scoresheets and audio files to create the speech and noise conditions for executing the acoustic tests.
Common terms
The following image breaks down the structure of a typical Alexa utterance:
Definitions
| Term | Definition |
|---|---|
| Utterance | An expression spoken to an Alexa Built-in device |
| Wake Word | The first word of the utterance, usually "Alexa", which signals an Alexa Built-in device to listen for the rest of the utterance. |
| Request | The segment of the utterance after the wake word. |
| Response Accuracy Rate (RAR) | The number of correct responses per corresponding requests. |
| Wake Word Detection Delay (WWDD) | The minimum time that a particular device requires a user to pause between saying "Alexa" and the remaining request for AVS to reliably receive the entire request. |
| Wake word False Alarm Rate (FAR) | The number of unwanted wakes per a given period. |
| Wake word False Rejection Rate (FRR) | The number of missed wake words per wake words spoken. |
| Imposter Acceptance Rate (IAR) | The number of times a voice is incorrectly recognized as belonging to the device owner per a given set of attempts. |
Abbreviations
| Acronym | Definition |
|---|---|
| DUT | Device Under Test |
| FAR | Wake word False Alarm Rate |
| FRR | Wake word False Rejection Rate |
| RAR | Response Accuracy Rate |
| WWDD | Wake word Detection Delay |
| IAR | Imposter Acceptance Rate |
Passing criteria
Before submitting the DUT to Amazon for evaluation, your own testing must verify that the DUT passes the following criteria:
| Metric | Passing criteria |
|---|---|
| FAR - Wake Word False Alarm Rate | Three or fewer wakes in a 24-hour period |
| FRR - Silence Condition | No more than 10% |
| FRR - Stationary Noise Condition | No more than 20% |
| FRR - External Music Condition | No more than 20% |
| FRR - Device Playback/Barge-in Condition | No more than 33% |
| RAR - Silence Condition | Greater than 85% |
| RAR - Stationary Noise Condition | Greater than 75% |
| RAR - External Music Condition | Greater than 75% |
| RAR - Device Playback/Barge-in Condition | Greater than 75% |
| WWDD - Wake Word Detection Delay | No more than 1 second (Ideally, the WWDD should effectively be zero.) |
| IAR – Imposter Acceptance Rate; applies to Mobile devices only | No more than 10 percent in silence condition |
Automatic Speech Recognition (ASR) profiles
For a given test case, the physical Device Under Test (DUT) placement depends on the typical distance from the mouth of a user to the device microphone. This placement is also referred to as the Automatic Speech Recognition (ASR) profile of the device. The applicable sound or speech sources for a test also depend on the ASR.
Amazon tests three ASR profiles:
- Close-talk: 0 to 1 ft (0.3 m)
- Near-field: 0 to 3 ft (0.9 m)
- Far-field: 0 to 9 ft (2.75 m)
For general discussion on ASR, see Audio Hardware Configurations.
Room requirements
To perform acoustic self-testing for your device, use a semi-anechoic chamber for testing. If a semi-anechoic chamber is unavailable, these testing techniques are also applicable in typical, well-defined, office rooms. To prepare your acoustic environment, or playback room, meet the following requirements.
Dimensions
Room size: At least 2.2 m high.
Room treatment:
- Reflection damping – Equip any office-like rooms with some acoustical damping to meet the target reverb times
- Reverberation time – Less than 0.7 s and higher than 0.2 s; between 125 Hz and 8 kHz
Noise floor:
- To reduce the influence of external noise, the noise floor measured in a room must be less than 35 dBSPL (A).
- The room should meet the ETSI standard and be less than 30 dBSPL (A).
Space around equipment:
- There must be at least a 0.5 m clearance between the walls and the test speakers
- There must be at least a 0.5 m clearance between the walls and the DUT
For an aerial view of this configuration, see Set up your Test Environment.
About the Amazon acoustic test chamber
To help you plan a sustainable testing environment, these are the characteristics of a test room under development by Amazon for our acoustic testing.
Amazon room size:
| Current | Future | |
|---|---|---|
| External dimensions | 4.6 m x 6.1 m x 2.7 m | 6.1 m x 7.6 m x 2.7 m |
| Internal dimensions | 4.4 m x 5.9 m x 2.4 m | 5.9 m x 7.4 m x 2.4 m |
Amazon room treatment:
- Ceiling track with acoustic curtains on three walls
- Carpet tiles
- Drop acoustic ceiling and ceiling panels
- Corner bass trap panels
- 10 Hz neoprene isolators
- Reverberation time: Less than 0.7 s and higher than 0.2 s; between 125 Hz and 8 kHz
Amazon noise floor: Less than 35 dBSPL (A) given an outside noise of 70 dBSPL (A).
Testing process
Use the instructions in the following sections to conduct each test described. Your DUT must pass all required tests before submitting your test result form to Amazon.
Acoustic testing setup
For successful testing, review our guidance and emulate the conditions described at each step.
- Review the recommended Equipment and Materials
- Learn how to Set up your Test Environment
- Learn to Calibrate your Setup
How to test
As appropriate for your device, conduct the four acoustic tests before you submit for certification. Each of the following links describes the ideal test room configuration, test files, test steps, and how to capture your metrics:
- How to test Wake Word Detection Delay.
- How to test False Rejection Rate and Response Accuracy Rate.
- How to test False Alarm Rate.
- How to test Voice Recognition and Imposter Acceptance Rate (Mobile only)
Submit your results
When your metrics illustrate that the DUT meets all criteria for passing, email the results to AVS-Certification-Labs@amazon.com. After Amazon reviews and verifies your results, we'll instruct you to ship devices to Amazon to conduct our own testing of your DUT. For more details, review our full Testing and Certification Process.
