Run an ASR Evaluation

The Automatic Speech Recognition (ASR) Evaluation tool allows you to run an ASR evaluation on an existing annotation set. To create an annotation set for ASR, see Create an Annotation Set for Automatic Speech Recognition (ASR).

Prerequisites

You need the following items to run an ASR evaluation:

• An Amazon developer account.
  To create an account, see Create Your Amazon Developer Account.
• An existing annotation set of audio utterances.
  For instructions about creating an annotation set, Create an Annotation Set for Automatic Speech Recognition (ASR).

Run an ASR evaluation on an annotation set

To run an ASR evaluation on an annotation set

1. Navigate to your annotation set:
   1. With your Amazon developer credentials, log in to the Alexa developer console.
   2. From the developer console, navigate to the Build tab.
   3. Under the Custom left nav tab, click Annotation Sets to display the NLU Evaluation tab, and then click the ASR Evaluation tab to display your list of existing annotation sets.

2. In the top-right corner of the page, click the Evaluate Model button:

   

   An ASR Evaluation window opens.

   

3. On the ASR Evaluation window, from the Annotation Source drop-down list, select the annotation set that you want to evaluate.

4. Click the Run an Evaluation button.

   The ASR Evaluation tool runs its test. When the test has completed, the ASR Evaluation window displays the Evaluation Id for the run with a link to the test report.

5. To view the Evaluation Report for the run, click the hyperlink with the Evaluation Id.

   The Evaluation Report displays the Expected vs. Actual Transcription, the Pass/Fail rate for each utterance, and the overall pass percentage for the annotation set. The overall pass percentage is a weighted average of all the utterances in the annotation set:

   

Export ASR results to the Natural Language Understanding (NLU) evaluation tool

Although the ASR Evaluation tool measures speech recognition and transcription accuracy for Alexa, the NLU Evaluation Tool measures natural language understanding accuracy. After you have your transcriptions from the ASR Evaluation tool, you can use the NLU evaluation tool to see how accurately those transcriptions map to intents and slots. Using these two tools together gives you an end-to-end holistic view of the accuracy of your interaction model.

To export ASR results to the NLU evaluation tool

1. On the ASR Evaluation Report page, from the Select export location menu, choose the NLU annotation set that corresponds to the ASR results that you want to evaluate.

   

2. Click the Export and go button.

   The NLU evaluation tool appends the actual transcriptions of the ASR evaluation to a new or existing NLU annotation set and displays a "Success" notification.

   

   Next, use the NLU evaluation tool to compare the actual transcriptions from the ASR evaluation to the skill model's expected mapping to intents and slots. See Batch Test Your Natural Language Understanding (NLU) Model.

Next steps

If your Evaluation Report identifies areas where you would like to improve your skill's performance, see Improve your Automatic Speech Recognition (ASR) Test Results.

• Previous: Create an Annotation Set for ASR
• Next: Improve your ASR Test Results