Calibrating Your Setup

If your acoustic testing setup is correctly calibrated, your test environment and Amazon's test environment will yield very similar results. Even though your speakers, media players, and laptop are likely different from Amazon's, assessments of the DUT's ability to hear utterances and play back correct responses can be consistent.

To account for the difference in environments, Amazon specifies a sound pressure level (SPL) for each test condition. You are expected to adjust the volume of playback in your test environment to achieve the same sound pressure level. To measure sound pressure level, you use a sound level meter (SLM). In this document, you'll learn to configure your setup so that you can emulate the SPL values of Amazon's test lab.

Before testing, follow the steps below to gather all facts about your laptop volume levels. Then, when you are ready for each full acoustic test, you'll set your laptop volume levels according to the facts you've gathered from your effort in this section. The details that you gather are primarily for you, serving as an aid in repeatability of your tests and in any debugging of your audio setup.

Measuring the ambient sound / noise floor of the test environment

The sound level meter (SLM) has a setting of "Slow" or "Fast." You will be instructed which setting to use depending on the test. The SLM also has a choice of "A" or "C" weighting for dB measurements. For all Alexa acoustic tests, you will use the "C" setting to gather dBC measurements.

To assess the ambient sound:

  1. Open the "SPL" tab of the Scoresheet.
  2. Prepare the sound level meter:
    • Choose the "Slow" setting
    • Choose the "C" setting to be able to measure in dBC
  3. With the test environment as quiet as possible - and with the Speech Speaker in location 1 and the Noise Speaker in place - hold the SPL wand next to the center of the microphone array of the DUT until the SPL value has stabilized (over 10 seconds).
  4. Record the SPL value in the cell at the intersection of "ambient noise" and "SPL dBC".

Control volume from each laptop

For each of your Speech Speaker/Laptop and Noise Speaker/Laptop setups, there are three volume controls you should care about:

  • The volume control of the speaker
  • The volume control of the media player with which you play an audio file
  • The volume control of the laptop

Make the laptop volume control your main volume control. The following steps offer a sequence with which to prepare each speaker, media player, and laptop:

  1. With the speaker power on, turn the volume of the speaker to its minimum.
  2. Load an audio file in a media player on the laptop.
  3. If the audio file is playing, stop it.
  4. Without muting the laptop volume, reduce the laptop volume to its minimum value or position. (This is to avoid the situation of someone using mute/unmute while the volume is mistakenly at a high setting.)
  5. Set the speaker's volume control to about 75% of maximum.
  6. Set the volume of the media player (again, with the file not playing) to maximum.
  7. With the laptop volume still minimized, start playing the audio file.
  8. Raise the laptop volume until you hear the audio file.

Your laptop volume is now your control of overall volume.

Establishing volume levels for each test

Speech speaker for silence

For testing device response to utterances in a silent setting, the Speech Speaker should be delivering 63 dBC.

  1. Set the sound level meter (SLM) to "Slow."
  2. Place the SLM on a stand 1 m away from the Speech Speaker. To do this, you may need to temporarily move the speaker from its final setup. Remember to mark the position with masking tape.
  3. Play the stationary noise file through the Speech Speaker; adjust the Speech Laptop volume level until the SPL value is 63 dBC.
  4. In the Scoresheet for "Speech Speaker for silence", document the Speech Laptop volume level in the "Volume setting value" column of the "SPL" table.

Speech speaker for noise

For testing device response to utterances in a noise condition, the Speech Speaker should be delivering 70 dBC (independent of the volume level for the Noise Speaker).

  1. Ensure nothing is being played through the Noise Speaker.
  2. Set the sound level meter to "Slow."
  3. Place the SLM on a stand 1 m away from the Speech Speaker.
  4. Play the calibration audio file through the Speech Speaker; adjust the Speech Laptop volume level until the SPL value is 70 dBC.
  5. In the Scoresheet for "Speech Speaker for noise", document the Speech Laptop volume level in the "Volume setting value" column of the "SPL" table.

Noise speaker for (stationary) noise

For testing under the (stationary) noise condition, the Noise Speaker should be delivering 64 dBC.

  1. Ensure nothing is being played through the Speech Speaker.
  2. Set the sound level meter to "Slow."
  3. Hold the SLM near the microphone array of the DUT, pointed at the noise speaker. For the Echo Dot, this would be 50 mm above the device in the center.
  4. Play the calibration audio file through the Noise Speaker; adjust the Noise Laptop volume level until the SPL value is 64 dBC.
  5. In the "Volume setting value" column for "Noise Speaker for (stationary) noise", document the Noise Laptop volume level.

Noise speaker for external music

For testing under the external music condition, the Noise Speaker should be delivering 64 dBC.

  1. Ensure nothing is being played through the Speech Speaker.
  2. Set the sound level meter to "Slow."
  3. Hold the SLM near the microphone array of the DUT, pointed at the noise speaker.
  4. Play the calibration audio file through the Noise Speaker; adjust the Noise Laptop volume level until the SPL value is 64 dBC (+/- 1 dB).
  5. In the "Volume setting value" column for "Noise Speaker for external music", document the Noise Laptop volume level.

Device playback: the DUT

For testing under the condition of music playback through the DUT, the DUT should be delivering 64 dBC.

  1. Ensure nothing is being played through the Speech Speaker or Noise Speaker.
  2. Set the sound level meter to "Fast."
  3. Setup the SLM in a stand (or hold it 1 m from the DUT), pointed at the DUT. If the Echo Dot is active, turn off its microphones.
  4. To the DUT, say, "Alexa, play my Test playlist." (For how to create this playlist, see Music (audio) account setup.)
  5. To the DUT, say, "Alexa, Skip forward 1 minute and 45 seconds"; you should calibrate after the background singers say "Because I'm" at the 2:02 mark of the song.
  6. Increase or decrease the volume of device playback until the SPL value is as close to 64 dBC as the DUT volume settings permit. You can say, "Alexa, volume 3" or "Alexa, volume 5" or "Alexa, volume 7", and so on, until you find a suitable volume for music playback. Due to these volume step sizes, 64 dBC may not be achievable. Note the value in the scorecard.
  7. In the "Volume setting value" column for "device playback - DUT", document the volume level of the device.

Device playback: the Echo Dot

For testing under the condition of music playback through the DUT, the DUT should be delivering 64 dBC.

  1. Ensure nothing is being played through the Speech Speaker or Noise Speaker.
  2. Set the sound level meter to "Fast."
  3. Setup the SLM in a stand (or hold it 1 m from the DUT), pointed at the DUT.
  4. To the DUT, say, "Alexa, play my Test playlist." (For how to create this playlist, see Music (audio) account setup.)
  5. To the DUT, say, "Alexa, Skip forward 1 minute and 45 seconds"; you should calibrate after the background singers say "Because I'm" at the 2:02 mark of the song.
  6. Increase or decrease the volume of device playback until the SPL value is as close to 64 dBC as the DUT volume settings permit. You can say, "Alexa, volume 3" or "Alexa, volume 5" or "Alexa, volume 7", and so on, until you find a suitable volume for music playback. Due to these volume step sizes, 64 dBC may not be achievable. Note the value in the scorecard.
  7. In the "Volume setting value" column for "device playback - Echo Dot," document the volume level of the device.

Now that you've acquired the recommended Equipment and Materials, finished Setting up your Test Environment, and calibrated your setup, you're ready to test for Wake Word Detection Delay, False Rejection Rate and Response Accuracy Rate, and False Alarm Rate. If you still have questions on your setup, contact your Amazon representative for guidance.

Amazon Alexa
Amazon Appstore
Amazon Web Services
Amazon Dash Replenishment