Test for False Rejection Rate and Response Accuracy Rate (FRR/RAR)
When performing acoustic self-testing for an Alexa Built-in device, your device must be able to recognize when a user speaks the Alexa wake word and accurately respond to user requests. To quantify wake word recognition and response accuracy for a device, execute the the False Rejection Rate (FRR) and Response Accuracy Rate (RAR) tests.
- FRR – The FRR represents the rate of missed wake words per wake word utterances attempted.
- RAR – The RAR represents the rate of correct responses for Alexa requests.
This page describes the testing procedures and passing criteria for the FRR and RAR for acoustic self-testing for Alexa Built-in devices.
- About the FRR test
- About the RAR test
- Test room configuration
- Test files
- Test steps
About the FRR test
The FRR test calculates the number of missed wake words per wake word utterances attempted. For example, if a user says "Alexa" ten times, and the device wakes up nine out of ten times, the FRR is one missed wake word per 10 wake words spoken, or 0.1 (10 percent).
A device passes FRR testing if it meets the following standards:
- No more than 10 percent in the silence condition.
- No more than 20 percent in the stationary noise condition.
- No more than 20 percent in the external music condition.
- No more than 33 percent when the device is playing music.
About the RAR test
RAR calculates number of correct responses per corresponding requests. For example, if a user makes ten requests, and Alexa gives a correct response nine out of ten times, the RAR is nine correct responses per 10 requests, or 0.9 (90%).
Test room configuration
For each of the silence and noise conditions, position the Speech Speaker and prepare the volume levels before playing the utterance audio files and assessing response.
For full instructions, see Setting up your Test Environment.
In the FRR_RAR_utterances folder on the Speech Laptop, 30 audio files represent a mix of male and female voices saying Alexa utterances. For each combination of Speech and Noise Speaker locations and the specified noise conditions, play all 30 audio files.
Use the scoresheet to note how the device responded. The scoresheet calculates your complete set of notes for a particular Speaker/Noise condition combination and forms a Wake word FRR and a RAR for the device for that Speaker/Noise condition combination.
How to use the utterance files
Each utterance file follows a consistent naming convention to help with debugging.
The FRR_RAR_utterances folder on the Speech Laptop has 30 audio files with recordings of six people (a mix of male and female voices), where each person has spoken five utterances. The filename for the utterance audio, for example, DE_P04_02_M.wav, hints at who expressed what. Each person (P01 through P06) has five utterances (01 through 05) and is of gender M or F.
The filename convention is language_personNumber_utteranceNumber_gender.wav. Referring to the naming convention as you test might make it easier to document helpful insights if your device has issues responding to a particular person or gender.
How to use the scoresheet
The scoresheet provides separate tables for the Echo Dot and the Device Under Test (DUT).
Review the layout of the FRR RAR tab of the scoresheet:
- The left side has six tables, one for each of the six voices in the audio files. To the right are six more tables.
- The upper left corner of each table identifies whether a table corresponds to the Echo Dot or the DUT.
- The column headings identify the noise conditions for the tests.
- Under each noise condition are five columns, one for each utterance from a single person.
False Rejection Rate, Far-field scoresheet:
The far-field example shows an Echo Dot being tested, where the tester had played three of the five utterances for "Person 1" for the Silence condition when the Speech Speaker was at the 0.9 m/90° location.
- For the first utterance file, the device awoke ("1") and gave the expected response ("1").
- For the second utterance file, the device awoke ("1") but gave no response ("0").
- For the third utterance file, the device didn't wake up ("0"), so it couldn't give a response ("0").
In the following steps, assess both the DUT and the Echo Dot for FRR and RAR. For both the DUT and the Echo Dot, perform the test steps for each of the noise conditions:
- Stationary noise
- External music
- Playback of music through the device
Test procedure overview
Open the FRR RAR tab in the scoresheet.
For the DUT:
- Position the DUT for test.
- Make sure that no other device in your test environment responds to "Alexa."
- Test for FRR and RAR as described in Test at each speech speaker location.
For the Echo Dot:
- Put the Echo Dot where the DUT had been for its tests.
- Make sure that no device other than the Echo Dot in your test environment responds to "Alexa."
- Test for FRR and RAR as described in Test at each speech speaker location.
Prepare the test environment
- Silence condition: For the silence condition, set the volume level of the Speech Laptop according to your Sound Pressure Level (SPL) notes for "Speech Speaker for silence." The Noise Laptop plays no sound in the silence condition.
- For either the stationary noise or external music condition:
- Set the volume level of the Speech Laptop according to your SPL notes for "Speech Speaker for noise."
- Set the volume level of the Noise Laptop according to your SPL notes for the current noise condition stationary noise or external music.
- Load and play the corresponding noise audio.
- For the noise condition of the device playing music:
- Set the volume level of the Speech Laptop according to your SPL notes for "Speech Speaker for noise." The Noise Laptop plays no sound in the silence condition.
- Say to the device: "Alexa, play my 'Test' playlist," referring to the "Test" playlist that you created in Set up your Amazon Music account.
Test at each speech speaker location
- Place and orient the Speech Speaker accordingly.
- Any noise condition should already be playing; if the noise audio has stopped, restart it.
- Sort the audio files by file name to group the utterances by person.
- For each of the 30 utterance audio files for the language to be tested:
- Load the utterance file into your media player.
- Play the utterance file.
- In the FRR row, note whether the device awoke ("1" for yes, "0" for no). If the FRR value is 0, write "0" for RAR.
- In the RAR row, note whether the response was correct ("1" for yes, "0" for no).
If you have Speaker Locations or Noise Conditions to test for the current noise condition, repeat the testing steps for each location or condition.
View the results
View the results table in the right-hand column of the FRR RAR tab. After you record the values for all 30 utterances for a noise condition and the four speech speaker locations, the scoresheet calculates the FRR and RAR for that noise condition.
|FRR - Silence Condition||No more than 10%|
|FRR - Stationary Noise Condition||No more than 20%|
|FRR - External Music Condition||No more than 20%|
|FRR - Device Playback/Barge-In Condition||No more than 33%|
|RAR - Silence Condition||Greater than 85%|
|RAR - Stationary Noise Condition||Greater than 75%|
|RAR - External Music Condition||Greater than 75%|
|RAR - Device Playback/Barge-In Condition||Greater than 75%|
See the Amazon passing criteria for all acoustic tests.
In the bottom right-hand corner of the sheet, there is a table that distills results per Speech Speaker location. Amazon doesn't use this data; use this table for your own debugging if you observe issues with your device for a particular location or sound source.