Test for False Rejection Rate and Response Accuracy Rate (FRR / RAR)
The "False Rejection Rate" (FRR) of a device is the number of missed wake words per wake words spoken. For example, if ten requests are given and the device wakes up nine out of ten times, the FRR is "1 missed wake word / 10 wake words spoken", or 0.1 (10%). A device passes this aspect of acoustic testing if the result is no more than 10% in the silence condition, no more than 20% when there is stationary noise, no more than 20% when there is external music, and no more than 33% when music is being played back from the device.
The Response Accuracy Rate (RAR) of a device is the number of correct responses per corresponding requests. For example, if ten requests are given and the response from Alexa is good nine out of ten times, the RAR is "9 correct responses / 10 requests", or 0.9 (90%).
- Test room configuration
- Test files
- Test steps
Test room configuration
For each of the silence and noise conditions, position the Speech Speaker and prepare the volume levels before playing the utterance audio files and assessing response.
For full instructions, see Setting up your Test Environment.
In the FRR_RAR_utterances folder added to the Speech Laptop, there are 30 audio files representing a mix of male and female voices saying Alexa utterances. For each combination of Speech and Noise Speaker locations and various noise conditions, you'll play all 30 audio files.
Use the scoresheet to note how the device responded. Your complete set of notes for a particular Speaker/Noise condition combination are calculated in the scoresheet. These calculations form a Wake Word False Rejection Rate (FRR) and a Response Accuracy Rate (RAR) for the device for that Speaker/Noise condition combination.
How to use the utterance files
Each utterance file follows a consistent naming convention that may prove useful when debugging.
In the FRR_RAR_utterances folder on the Speech Laptop, there are 30 audio files. There are recordings of six people (a mix of male and female), where each person has spoken five utterances. An utterance audio file's name (i.e. DE_P04_02_M.wav) hints at who expressed what. Each person (P01 through P06) has five utterances (01 through 05) and is of gender M or F.
The filename convention is language_personNumber_utteranceNumber_gender.wav. Referring to this convention as you test may make it easier to document helpful insights if your device has issues responding to a particular person or gender.
How to use the scoresheet
The scoresheet provides separate tables for the Echo Dot and the Device Under Test (DUT).
Open the "FRR RAR" tab of the scoresheet and observe the following:
- There are six tables along the left: one for each of the six persons mentioned above. To the right are six more tables.
- The upper left corner of each table identifies whether it is for the Echo Dot or for the DUT.
- The column headings identify the various noise conditions for these tests.
- Under each noise condition are five columns.
There is a subtle yet significant aspect to the last point: these five columns are for the five utterances of a single person. Therefore, do not mix the results for different persons within one table. To make this easier, sort your utterance files by name before playing through all 30 utterances. Naturally, as you fill in all five columns, you'll need to jump to the next table to fill in the equivalent five columns for the same noise condition (such as "silence"), yet the next person (with his or her five utterances).
False Rejection Rate, Far-field scoresheet:
Take the example above, which was captured from the far-field version of the scoresheet where an Echo Dot was being tested. At this point in the testing, three of the five utterances for "Person 1" have been played for the Silence condition while the Speech Speaker was at the 0.9 m / 90° location.
For the first utterance file, the device awoke ("1") and gave the expected response ("1"). For the second utterance file, the device awoke ("1"), but it did not respond ("0"). For the third utterance file, the device did not awake ("0"), and therefore, it could not have a response ("0").
In these steps, assess both the DUT and the Echo Dot for FRR and RAR. For both the DUT and the Echo Dot, you'll perform these steps for each of the noise conditions:
- Stationary noise
- External music
- Playback of music through the device
Open the "FRR RAR" tab in the scoresheet.
For the DUT:
- Position the DUT for test.
- Ensure that this device is the only device in your test environment that responds to "Alexa."
- Test for FRR and RAR as described in the steps below.
For the Echo Dot:
- Put the Echo Dot where the DUT had been for its tests.
- Ensure the Echo Dot is the only device in your test environment that responds to "Alexa."
- Test for FRR and RAR as described in the steps below.
Prepare the test environment
For the silence condition: Set the volume level of the Speech Laptop according to your Sound Pressure Level (SPL) notes for "Speech Speaker for silence." (The Noise Laptop plays no sound in this condition.)
For either the stationary noise or external music condition:
- Set the volume level of the Speech Laptop according to your SPL notes for "Speech Speaker for noise."
- Set the volume level of the Noise Laptop according to your SPL notes for the current noise condition stationary noise or external music.
- Load and play the corresponding noise audio.
For the noise condition of music being played back through the device:
- Set the volume level of the Speech Laptop according to your SPL notes for "Speech Speaker for noise". (The Noise Laptop plays no sound in this condition.)
- Say to the device: "Alexa, play my Test playlist." (This playlist was established in the section Audio Sources.)
Test at each speech speaker location
- Place and orient the Speech Speaker accordingly.
- Any noise condition should already be playing; if the noise audio has stopped, restart it.
- Sort the audio files by file name. This groups the utterances by person.
- For each of the 30 utterance audio files for the language to be tested:
- Load the utterance file into your media player.
- Play the utterance file.
- In the FRR row, note whether the device awoke ("1" for yes, "0" for no). If the FRR value is 0, write "0" for RAR.
- In the RAR row, note whether the response was correct ("1" for yes, "0" for no).
If there are more Speaker Locations to test for the current noise condition, repeat the steps above; if there are more noise conditions to test, repeat.
View the results
View the results table in the right-hand column of the FRR RAR tab. Once you've entered values for all 30 utterances for a particular noise condition and all four Speech Speaker locations, two numbers appear for that noise condition: a number for FRR, and a number for RAR.
|FRR - Silence Condition||No more than 10%|
|FRR - Stationary Noise Condition||No more than 20%|
|FRR - External Music Condition||No more than 20%|
|FRR - Device Playback / Barge-In Condition||No more than 33%|
|RAR - Silence Condition||Greater than 85%|
|RAR - Stationary Noise Condition||Greater than 75%|
|RAR - External Music Condition||Greater than 75%|
|RAR - Device Playback / Barge-In Condition||Greater than 75%|
See Amazon's passing criteria for all acoustic tests.
In the bottom right-hand corner of the sheet, there is a table that distills results per Speech Speaker location. This information is not used by Amazon. Rather, it is provided for your own debugging if you observe issues with your device for a particular location or sound source.