Acoustic Testing Guide

Acoustic performance is critical to ensure the wake word is performing as expected and that customer requests are handled appropriately. To ensure we keep the mutual trust of our customers for the expected Alexa experience, the Alexa Voice Service checks the performance of acoustic systems.

In this guide, you'll learn to conduct your own acoustic testing in a way that emulates how Amazon will test your device. This includes the type of rooms we use for testing, the equipment we recommend to recreate our test conditions, and test steps. AVS also provides scoresheets and audio files representing the various speech and noise conditions you'll use to run these tests.

Common terms

Acoustic Testing Utterance for Wake Word Detection

Definitions

Term Definition
Utterance An expression spoken to an Alexa Built-in device
Wake word The first word of the utterance, usually "Alexa", which signals an Alexa Built-in device to listen for the rest of the utterance
Request The portion of the utterance after the wake word
Response Accuracy Rate (RAR) The number of correct responses per corresponding requests
Wake Word Detection Delay (WWDD) The minimum time that a particular device requires a user to pause between saying "Alexa" and the remaining request in order for the Alexa Voice Service to reliably receive the entire request
Wake Word False Alarm Rate (FAR) The number of unwanted wakes per a given period
Wake Word False Rejection Rate (FRR) The number of missed wake words per wake words spoken

Abbreviations

Acronym Definition
DUT Device Under Test
FAR Wake Word False Alarm Rate
FRR Wake Word False Rejection Rate
RAR Response Accuracy Rate
WWDD Wake Word Detection Delay

Passing criteria

Before you can submit the DUT to Amazon for evaluation, your own testing must demonstrate that the DUT passes the following criteria:

Metric Passing criteria
FAR - Wake Word False Alarm Rate 3 or fewer wakes in a 24-hour period
FRR - Silence Condition No more than 10%
FRR - Stationary Noise Condition No more than 20%
FRR - External Music Condition No more than 20%
FRR - Device Playback / Barge-in Condition No more than 33%
RAR - Silence Condition Greater than 85%
RAR - Stationary Noise Condition No more than 75%
RAR - External Music Condition No more than 75%
RAR - Device Playback / Barge-in Condition No more than 75%
WWDD - Wake Word Detection Delay No more than 1 second (Ideally, the device audio input is implemented in such a way that the WWDD is effectively zero.)

Automatic Speech Recognition (ASR) profiles

During testing, where to place the Device Under Test (DUT) depends on the use case for the device, which is defined by the typical distance from a customer's mouth to the microphone(s) of the device. This is also referred to as the Automatic Speech Recognition (ASR) profile of the device. The various sound or speech sources described in this document also depend on the ASR.

Amazon tests three ASR profiles, and this guide will teach you to test all three:

  • Close-talk: 0 to 1 ft (0.3 m)
  • Near-field: 0 to 3 ft (0.9 m)
  • Far-field: 0 to 9 ft (2.75 m)

For general discussion on ASR, see Audio Hardware Configurations.

Room requirements

To run the tests described in this guide, you should use a semi-anechoic chamber for testing, but these techniques are also applicable in typical (well-defined) office rooms. To prepare your acoustic environment, or playback room, meet the following requirements.

Dimensions

Room size: At least 2.2 m high.

Room treatment:

  • Reflection damping: Office-like rooms should be equipped with some acoustical damping to meet the target reverb times
  • Reverberation time: Less than 0.7 s and higher than 0.2 s; between 125 Hz and 8 kHz

Noise floor:

  • To reduce the influence of external noise, the noise floor measured in a room shall be less than 35 dBSPL (A).
  • The room should meet the ETSI standard and be less than 30 dBSPL (A).

Space around equipment:

  • There must be at least a 0.5 m clearance between the walls and the test speakers
  • There must be at least a 0.5 m clearance between the walls and the DUT

For an aerial view of this configuration, see Set up your Test Environment.

About Amazon's acoustic test chamber

To help you plan a future-proof environment, these are the characteristics of a test room under development by Amazon for our acoustic testing.

Amazon's room size:

  Current Future
External dimensions 4.6 m x 6.1 m x 2.7 m 6.1 m x 7.6 m x 2.7 m
Internal dimensions 4.4 m x 5.9 m x 2.4 m 5.9 m x 7.4 m x 2.4 m

Amazon's room treatment:

  • Ceiling track with acoustic curtains on three walls
  • Carpet tiles
  • Drop acoustic ceiling and ceiling panels
  • Corner bass trap panels
  • 10 Hz neoprene isolators
  • Reverberation time: Less than 0.7 s and higher than 0.2 s; between 125 Hz and 8 kHz

Amazon's noise floor: Less than 35 dBSPL (A) given an outside noise of 70 dBSPL (A).

Testing process

Use this guide to conduct each test described. Your device - the device under test (DUT) - must pass all of the required tests before you can submit your test result form to Amazon.

Acoustic testing setup

For successful testing, review our guidance and emulate the conditions described at each step.

How to test

You'll conduct three types of acoustic testing before you submit for certification. Each "How to test" page of this guide describes the ideal test room configuration, test files, test steps, and how to capture your metrics.

Submit your results

When your metrics illustrate that the DUT meets all of these passing criteria, email the results to mailto:AVS-Certification-Labs@amazon.com, and Amazon will get back to you. Once we've reviewed your results and confirmed they're successful, we'll instruct you to ship devices to Amazon to conduct our own testing of your DUT. For more details, review our full Testing and Certification Process.