About the Automatic Speech Recognition Evaluation tool

The Automatic Speech Recognition (ASR) Evaluation tool allows you to batch test audio files to measure the ASR accuracy of the skills that you've developed. With the ASR Evaluation tool, you can test your sample audio utterances against ASR models and compare expected transcriptions with the actual transcriptions. The tool generates an evaluation report with accuracy metrics and pass/fail results for each test utterance, which you can use to resolve accuracy issues.

To learn more about ASR, see What Is Automatic Speech Recognition?.

The ASR Evaluation tool is available in the Alexa developer console. If you want to run ASR evaluations programmatically, see the Use the Automatic Speech Recognition API.

Benefits of ASR evaluation

If the users of your skill aren't getting their expected responses from Alexa, ASR Evaluation can help you troubleshoot speech recognition issues and improve skill performance. ASR Evaluation can help pinpoint commonly misrecognized words for your skill. You can then try to improve recognition accuracy for those words by mapping them back to the skill model as sample utterances and slot values.

For example, if you have a coffee-related skill where you expect users to ask Alexa to "order a mocha," ASR evaluation results might show you that sometimes Alexa misunderstands the word "mocha" as "milk." To mitigate this issue, you can map an utterance directly to an Alexa intent to help improve Alexa's understanding within your skill.

Overview of the ASR evaluation process

Complete the following procedure to run the ASR Evaluation tool and improve the speech recognition accuracy and interaction model for your skill.

To run the ASR Evaluation tool

  1. Create an annotation set of recorded utterances to use for testing.
  2. Run the ASR evaluation tool.
  3. Use the results from an ASR evaluation to improve your skill's accuracy and interaction model.