Test for Imposter Acceptance Rate (IAR)
Perform Voice Training on your devices so that Alexa recognizes the voice of a registered device owner. An Alexa Built-in mobile device that implements voice recognition shouldn't respond unless Alexa recognizes the user voice speaking the wake word. To verify that the Voice Training for an Alexa Built-in mobile device meets Amazon standards, Amazon tests applicable devices for Imposter Acceptance Rate (IAR).
The IAR of a device is the number of times that the device incorrectly recognizes a voice as belonging to the device owner per a given number of attempts. For example, if a non-registered user makes ten attempts, and the device wakes one out of ten times, the IAR is "one invalid wake in 10 attempts", or 0.1 (10%). A device passes the IAR aspect of acoustic testing if the result is no more than 10 percent in the silence condition.
Because of the unique nature of a voice-trained device, Amazon provides mobile voice training wake word utterances and corresponding FRR/RAR utterances spoken by a standard set of user voices. Use the Amazon utterance files when assessing FRR/RAR for the device. See Test steps for details.
Test room configuration
Position the Speech Speaker and prepare the volume levels before playing the utterance audio files and assessing response.
For full instructions, see Setting up your Test Environment.
Voice train the Device Under Test (DUT)
Previously, you copied the MobileTest folder to the Speech Laptop in setting up the Audio source files. The MobileTest folder has 10 subfolders representing 10 different test users (five male, five female). Each folder contains the following files for each user:
- A Voice Training .wav file that guides you through the voice training process for the associated user voice.
- A set of FRR/RAR audio files as spoken by that user. Use these files to perform FRR/RAR testing on a voice trained device.
- An IAR folder containing 39 audio files representing a mix of male and female “imposters” saying the wake word. When executing the IAR test, play all 39 audio files for each user voice.
Use the scoresheet to note how the device responded. The scoresheet uses your complete set of notes to calculate an IAR for the device. Execute the IAR test in the silence condition only.
About the utterance files
Each utterance file in the MobileTest folder structure follows a consistent naming convention to help with debugging.
In the MobileTest folder on the Speech Laptop, locate the 10 subfolders (one for each user voice). Each user has separate utterances for FRR/RAR testing and for IAR testing.
The FRR/RAR utterance files use the following naming convention:
Each user has five utterances designated for FRR/RAR for that particular voice. To complete FRR/RAR testing, retrain the device to the individual voices as you progress through the test.
The IAR utterances for each speaker use the following naming conventions:
language_Mobile_SpeakerName_personNumber_utteranceNumber_gender.wav and language_personNumber_utteranceNumber_gender.wav
The IAR utterances consist of a mix of male and female “imposters” saying the wake word.
Refer to the naming conventions as you test to help document insights if your device has issues corresponding to a particular person or gender.
Use the scoresheet for FRR/RAR and IAR
Learn to use the scoresheet for testing Alexa Built-in mobile devices with voice recognition. Use this scoresheet for FRR/RAR and IAR testing.
Use the scoresheet for FRR/RAR testing
Use the scoresheet as described in the FRR/RAR Test Steps documentation when evaluating FRR/RAR on a Voice Trained device.
Use the scoresheet for IAR testing
For IAR assessment, open the IAR tab of the scoresheet, and view the layout:
- The IAR tab one table for each of the ten user voices.
- The 39 columns assigned to each person are to record test results from utterances from 39 different people, and shouldn't include any wake word utterance by the person whose voice trained the device.
- Make sure that your results don't include a wake word utterance by a person whose voice trained the device.
For the IAR test, the DUT shouldn't respond to any recorded wake word utterance from any speaker other than the speaker whose voice trained the device.
The following image shows a sample scoresheet where the DUT was trained to recognize Person 1 (“Andrew”), so that the device shouldn't wake unless it hears Person 1 (“Andrew”) say the wake word. When five of the 39 impersonator test wake word utterances were played in the “silence” condition with the Speech Speaker was at the distance 0.9 m/90 degree location, for the first and last two utterances, the device didn't wake, giving a score of “0”. However, for the third utterance, the device did wake and giving a score of “1”.
In the following steps, assess the DUT for IAR. Perform the IAR test in the silence condition only.
Prepare the test environment
Run the IAR test in the silence condition only. Set the volume level of the Speech Laptop according to your Sound Pressure Level (SPL) notes for "Speech Speaker for silence." (The Noise Laptop plays no sound in the silence condition.)
- Open the IAR tab in the scoresheet.
- Position the DUT for test.
- Make sure that no device in your test environment other than the DUT responds to "Alexa."
- Test for IAR as described in the next section.
Test at the 0.9 m, 90 degree speech speaker location (Location 4)
- Place and orient the Speech Speaker accordingly.
- Train the device for the first voice.
- Use the audio file labeled “Training” for the current voice. This file consists of the named user saying the wake word 10 times to lead you through the process for training the device.
- Prepare the DUT for voice training (per your device’s specification).
- Load the training utterance audio file onto your media player.
- Play the training utterance file for that person.
- Follow the prompts on the device as the training utterance file plays, until the device has been trained.
- Open the IAR subfolder for the first voice, which contains the “imposter” wake word utterance files for the first voice test. For each of the 39 utterance files in the IAR subfolder:
- Load the utterance file into your media player.
- Play the utterance file.
- Note in the scoresheet whether the device woke ("1" for yes, "0" for no).
- When you’ve completed IAR scoring for the first voice, delete that voice profile from the device to prepare for training for the next voice.
Repeat the IAR test steps for all 10 voices in the MobileTest folder.
View the results
View the results table in the right-hand column of the IAR tab. After you've entered values for all 10 voices, the IAR results appear on the scoresheet.
|IAR - Silence Condition||No more than 10%|
See the Amazon passing criteria for all acoustic tests.
Submit your results
If you've completed all testing, and your metrics illustrate passing criteria, email your results to AVS-Certification-Labs@amazon.com for Amazon to review your scoresheets and confirm your results. After Amazon confirms your results, ship the requested devices to Amazon for testing. For more details, review our full Testing and Certification Process.