Acoustic Testing Guide
Acoustic performance is critical to ensure the wake word is performing as expected and that customer requests are handled appropriately. To ensure we keep the mutual trust of our customers for the expected Alexa experience, the Alexa Voice Service checks the performance of acoustic systems.
In this guide, you'll learn to conduct your own acoustic testing in a way that emulates how Amazon will test your device. This includes the type of rooms we use for testing, the equipment we recommend to recreate our test conditions, and test steps. AVS also provides scoresheets and audio files representing the various speech and noise conditions you'll use to run these tests.
Common terms
Definitions
| Term | Definition |
|---|---|
| Utterance | An expression spoken to an Alexa Built-in device |
| Wake word | The first word of the utterance, usually "Alexa", which signals an Alexa Built-in device to listen for the rest of the utterance |
| Request | The portion of the utterance after the wake word |
| Response Accuracy Rate (RAR) | The number of correct responses per corresponding requests |
| Wake Word Detection Delay (WWDD) | The minimum time that a particular device requires a user to pause between saying "Alexa" and the remaining request in order for the Alexa Voice Service to reliably receive the entire request |
| Wake Word False Alarm Rate (FAR) | The number of unwanted wakes per a given period |
| Wake Word False Rejection Rate (FRR) | The number of missed wake words per wake words spoken |
Abbreviations
| Acronym | Definition |
|---|---|
| DUT | Device Under Test |
| FAR | Wake Word False Alarm Rate |
| FRR | Wake Word False Rejection Rate |
| RAR | Response Accuracy Rate |
| WWDD | Wake Word Detection Delay |
Passing criteria
Before you can submit the DUT to Amazon for evaluation, your own testing must demonstrate that the DUT passes the following criteria:
| Metric | Passing criteria |
|---|---|
| FAR - Wake Word False Alarm Rate | 3 or fewer wakes in a 24-hour period |
| FRR - Silence Condition | No more than 10% |
| FRR - Stationary Noise Condition | No more than 20% |
| FRR - External Music Condition | No more than 20% |
| FRR - Device Playback / Barge-in Condition | No more than 33% |
| RAR - Silence Condition | Greater than 85% |
| RAR - Stationary Noise Condition | No more than 75% |
| RAR - External Music Condition | No more than 75% |
| RAR - Device Playback / Barge-in Condition | No more than 75% |
| WWDD - Wake Word Detection Delay | No more than 1 second (Ideally, the device audio input is implemented in such a way that the WWDD is effectively zero.) |
Automatic Speech Recognition (ASR) profiles
During testing, where to place the Device Under Test (DUT) depends on the use case for the device, which is defined by the typical distance from a customer's mouth to the microphone(s) of the device. This is also referred to as the Automatic Speech Recognition (ASR) profile of the device. The various sound or speech sources described in this document also depend on the ASR.
Amazon tests three ASR profiles, and this guide will teach you to test all three:
- Close-talk: 0 to 1 ft (0.3 m)
- Near-field: 0 to 3 ft (0.9 m)
- Far-field: 0 to 9 ft (2.75 m)
For general discussion on ASR, see Audio Hardware Configurations.
Room requirements
To run the tests described in this guide, you should use a semi-anechoic chamber for testing, but these techniques are also applicable in typical (well-defined) office rooms. To prepare your acoustic environment, or playback room, meet the following requirements.
Dimensions
Room size: At least 2.2 m high.
Room treatment:
- Reflection damping: Office-like rooms should be equipped with some acoustical damping to meet the target reverb times
- Reverberation time: Less than 0.7 s and higher than 0.2 s; between 125 Hz and 8 kHz
Noise floor:
- To reduce the influence of external noise, the noise floor measured in a room shall be less than 35 dBSPL (A).
- The room should meet the ETSI standard and be less than 30 dBSPL (A).
Space around equipment:
- There must be at least a 0.5 m clearance between the walls and the test speakers
- There must be at least a 0.5 m clearance between the walls and the DUT
For an aerial view of this configuration, see Set up your Test Environment.
About Amazon's acoustic test chamber
To help you plan a future-proof environment, these are the characteristics of a test room under development by Amazon for our acoustic testing.
Amazon's room size:
| Current | Future | |
|---|---|---|
| External dimensions | 4.6 m x 6.1 m x 2.7 m | 6.1 m x 7.6 m x 2.7 m |
| Internal dimensions | 4.4 m x 5.9 m x 2.4 m | 5.9 m x 7.4 m x 2.4 m |
Amazon's room treatment:
- Ceiling track with acoustic curtains on three walls
- Carpet tiles
- Drop acoustic ceiling and ceiling panels
- Corner bass trap panels
- 10 Hz neoprene isolators
- Reverberation time: Less than 0.7 s and higher than 0.2 s; between 125 Hz and 8 kHz
Amazon's noise floor: Less than 35 dBSPL (A) given an outside noise of 70 dBSPL (A).
Testing process
Use this guide to conduct each test described. Your device - the device under test (DUT) - must pass all of the required tests before you can submit your test result form to Amazon.
Acoustic testing setup
For successful testing, review our guidance and emulate the conditions described at each step.
- Review the recommended Equipment and Materials
- Learn how to Set up your Test Environment
- Learn to Calibrate your Setup
How to test
You'll conduct three types of acoustic testing before you submit for certification. Each "How to test" page of this guide describes the ideal test room configuration, test files, test steps, and how to capture your metrics.
- How to test Wake Word Detection Delay
- How to test False Rejection Rate and Response Accuracy Rate
- How to test False Alarm Rate
Submit your results
When your metrics illustrate that the DUT meets all of these passing criteria, email the results to AVS-Certification-Labs@amazon.com, and Amazon will get back to you. Once we've reviewed your results and confirmed they're successful, we'll instruct you to ship devices to Amazon to conduct our own testing of your DUT. For more details, review our full Testing and Certification Process.
