SpeechRecognizer 2.3

The SpeechRecognizer interface is the core interface of the Alexa Voice Service (AVS) and exposes directives and events for capturing and interacting with user speech.

Version changes

• Support for multiple active wake words:
  • ADDED wakeWords configuration parameter
  • ADDED SetWakeWords directive
  • ADDED WakeWordsReport event
  • ADDED WakeWordsChanged event
  • CHANGED Recognize event to include a new wakeWord field
  • REMOVED RecognizerState context entry
• New setting for enabling and disabling wake word confirmation tone
  • ADDED SetWakeWordConfirmation directive
  • ADDED WakeWordConfirmationReport event
  • ADDED WakeWordConfirmationChanged event
• New setting for enabling and disabling end of listening tone
  • ADDED SetSpeechConfirmation directive
  • ADDED SpeechConfirmationReport event
  • ADDED SpeechConfirmationChanged event
• Support for calculation of user-perceived latency
  • ADDED SetEndOfSpeechOffset directive
  • CHANGED Recognize event to include a new startOfSpeechTimestamp field

State Diagram

The following diagram illustrates state changes driven by SpeechRecognizer components. Boxes represent SpeechRecognizer states and the connectors indicate state transitions.

SpeechRecognizer has the following states:

IDLE: Prior to capturing user speech, SpeechRecognizer should be in an idle state. SpeechRecognizer should also return to an idle state after a speech interaction with AVS has concluded. This can occur when a speech request has been successfully processed or when an ExpectSpeechTimedOut event has elapsed.

Additionally, SpeechRecognizer may return to an idle state during a multiturn interaction, at which point, if additional speech is required from the user, it should transition from the idle state to the expecting speech state without a user starting a new interaction.

RECOGNIZING: When a user begins interacting with the device, specifically when captured audio is streamed to AVS, SpeechRecognizer should transition from the idle state to the recognizing state. It should remain in the recognizing state until the client stops recording speech (or streaming is complete), at which point your SpeechRecognizer component should transition from the recognizing state to the busy state.

BUSY: While processing the speech request, SpeechRecognizer should be in the busy state. You cannot start another speech request until the component transitions out of the busy state. From the busy state, SpeechRecognizer will transition to the idle state if the request is successfully processed (completed) or to the expecting speech state if Alexa requires additional speech input from the user.

EXPECTING SPEECH: SpeechRecognizer should be in the expecting speech state when additional audio input is required from a user. From expecting speech, SpeechRecognizer should transition to the recognizing state when a user interaction occurs or the interaction is automatically started on the user's behalf. It should transition to the idle state if no user interaction is detected within the specified timeout window.

Capability assertion

A device can implement SpeechRecognizer 2.3 on its own behalf, but not on behalf of any connected endpoints.

New AVS integrations must assert support through Alexa.Discovery, but AVS continues support for legacy integrations through the Capabilities API.

Note that SpeechRecognizer 2.3 depends on version 2.0 or higher of the System interface.

Sample Object

If your device doesn't support wake words, for example, push-to-talk devices, omit the wakeWords configuration parameter from your payload.

{
  "type": "AlexaInterface",
  "interface": "SpeechRecognizer",
  "version": "2.3",
  "configurations": {
    "wakeWords": [
      {
        "scopes": ["DEFAULT"],
        "values": [
          ["ALEXA"]
        ]
      }
    ]
  }
}

wakeWords

This list informs Alexa of the wake words that the device can be set to listen for through the SetWakeWords, WakeWordsReport, and WakeWordsChanged messages.

Currently, the only wake word available for AVS devices is ALEXA, which is applicable to every locale the device might be set to. Therefore, specify ALEXA in the global DEFAULT scope, as shown above.

Directives

StopCapture

The StopCapture directive instructs the device to stop capturing user speech after AVS has identified the user intent or when detected the end of speech. When a device receives the StopCapture directive, the device closes the microphone and stops listening for user speech.

Sample message

{
    "directive": {
        "header": {
            "namespace": "SpeechRecognizer",
            "name": "StopCapture",
            "messageId": "{{STRING}}",
            "dialogRequestId": "{{STRING}}"
        },
        "payload": {
        }
    }
}

Header parameters

ExpectSpeech

Alexa sends an ExpectSpeech to a device when Alexa requires more information to fulfill a user request. ExpectSpeech instructs the device to open the microphone and begin streaming user speech. If the microphone isn't opened within the specified timeout window, the device must send an ExpectSpeechTimedOut event to AVS.

During a multi-turn interaction with Alexa, a device receives at least one ExpectSpeech directive instructing the device to start listening for user speech. If present, pass the initiator object included in the payload of the ExpectSpeech directive back to Alexa as the initiator object in the following Recognize event. If initiator is absent from the payload, the following Recognize event should not include initiator.

See Interaction Model.

Sample message

{
    "directive": {
        "header": {
            "namespace": "SpeechRecognizer",
            "name": "ExpectSpeech",
            "messageId": "{{STRING}}",
            "dialogRequestId": "{{STRING}}"
        },
        "payload": {
            "timeoutInMilliseconds": {{LONG}},
            "initiator": {
                "type": "{{STRING}}",
                "payload": {
                    "token": "{{STRING}}"
                }
            }
        }
    }
}

Header parameters

Payload parameters

SetEndOfSpeechOffset

The SetEndOfSpeechOffset provides the device with the values necessary to calculate user-perceived latency (UPL) associated with a user utterance.

UPL is the duration between the end of user speech and the beginning of the resulting activity, such as audio output from a Speak directive.

The device stops streaming audio to Alexa as part of the Recognize event after receiving the StopCapture directive. As a result, the device doesn't detect the duration of the user speech in the audio stream.

The SetEndOfSpeechOffset directive contains the duration of the user speech in the endOfSpeechOffsetInMilliseconds field. By correlating with the timestamp that the speech started, the device determines the timestamp at the end of speech. Finally, the device calculates the duration between the end of speech and the beginning of the resulting activity.

Sample message

{
  "directive": {
    "header": {
      "namespace": "SpeechRecognizer",
      "name": "SetEndOfSpeechOffset",
      "messageId": "{{STRING}}",
      "dialogRequestId": "{{STRING}}"
    },
    "payload": {
      "endOfSpeechOffsetInMilliseconds": {{LONG}},
      "startOfSpeechTimestamp": "{{STRING}}"
    }
  }
}

Header parameters

Payload parameters

SetWakeWords

Alexa sends the SetWakeWords directive to the device to instruct the device to set one or more wake words. For example, a user could set a wake word through the Alexa companion app. The device must send the WakeWordsReport event in response to the SetWakeWords directive.

Sample message

{
  "directive": {
    "header": {
      "namespace": "SpeechRecognizer",
      "name": "SetWakeWords",
      "messageId": "{{STRING}}"
    },
    "payload": {
      "wakeWords": ["{{STRING}}", ...]
    }
  }
}

Header parameters

Payload parameters

SetWakeWordConfirmation

Alexa sends the SetWakeWordConfirmation directive to the device to instruct whether to play a tone when it detects a user speaking a wake word, such as when a user updates the appropriate setting in the Alexa companion app. The device must send the WakeWordConfirmationReport event in response to the SetWakeWordConfirmation directive.

Sample message

{
  "directive": {
    "header": {
      "namespace": "SpeechRecognizer",
      "name": "SetWakeWordConfirmation",
      "messageId": "{{STRING}}"
    },
    "payload": {
      "wakeWordConfirmation": "{{STRING}}"
    }
  }
}

Header parameters

Payload parameters

SetSpeechConfirmation

Alexa sends the SetSpeechConfirmation directive to the device to communicate whether to play a tone when the device stops capturing user audio input, such as when a user updates the appropriate setting in the Alexa companion app. The device must send the SpeechConfirmationReport event in response to the SetSpeechConfirmation directive.

Sample message

{
  "directive": {
    "header": {
      "namespace": "SpeechRecognizer",
      "name": "SetSpeechConfirmation",
      "messageId": "{{STRING}}"
    },
    "payload": {
      "speechConfirmation": "{{STRING}}"
    }
  }
}

Header parameters

Payload parameters

Events

Recognize

Use the Recognize event to send user speech to AVS and translate that speech into one or more directives. Send the Recognize event as a multipart message, consisting of two parts:

• A JSON-formatted object.
• The binary audio captured by the device microphone.

After a user initiates an interaction with Alexa, the microphone must remain open until:

• The device receives a StopCapture directive.
• AVS closes the stream.
• The user manually closes the microphone. For example, a press and hold implementation.

The profile parameter and initiator object tell Alexa which ASR profile to use to best understand the captured audio, and how the user initiated the interaction.

To reduce latency, separate captured audio into chunks before streaming the audio to AVS. For requirements on sending captured audio, see the Codec specifications table.

Codec specifications

Send all captured audio to AVS in either PCM or Opus, and adhere to the following specifications:

For a protocol-specific example, see Structuring an HTTP/2 Request.

Sample message

{
  "context": [
    // Use an array of context objects to communicate the
    // state of all device components to Alexa. See Context for details.      
  ],
  "event": {
    "header": {
      "namespace": "SpeechRecognizer",
      "name": "Recognize",
      "messageId": "{{STRING}}",
      "dialogRequestId": "{{STRING}}"
    },
    "payload": {
      "profile": "{{STRING}}",
      "format": "{{STRING}}",
      "initiator": {
        "type": "{{STRING}}",
        "payload": {
          "wakeWordIndices": {
            "startIndexInSamples": {{LONG}},
            "endIndexInSamples": {{LONG}}
          },
          "wakeWord": "{{STRING}}",
          "token": "{{STRING}}"
        }
      }
      "startOfSpeechTimestamp": "{{STRING}}"
    }
  }
}

Binary audio attachment

Each Recognize event requires a corresponding binary audio attachment as one part of the multipart message. Each binary audio attachment requires the following headers:

Content-Disposition: form-data; name="audio"
Content-Type: application/octet-stream

{{BINARY AUDIO ATTACHMENT}}

Context

This event requires your product to report the status of all client component states to Alexa in the context object.

Header parameters

Payload parameters

ASR profiles

ASR profiles vary by different products, form factors, acoustic environments and use cases. Use the following table to learn more about accepted values for the profile parameter.

Initiator

The initiator parameter tells AVS how a user triggered an interaction with Alexa and determines the answers to two questions:

• When Alexa detects the end of speech, does AVS send a StopCapture to the device?
• Should AVS perform cloud-based wake word verification on the stream?

Include initiator in the payload of each SpeechRecognizer.Recognize event. The following values are accepted:

ExpectSpeechTimedOut

Send an ExpectSpeechTimedOut event to AVS if an ExpectSpeech directive was received, but was not satisfied within the specified timeout window.

Sample message

{
    "event": {
        "header": {
            "namespace": "SpeechRecognizer",
            "name": "ExpectSpeechTimedOut",
            "messageId": "{{STRING}}",
        },
        "payload": {
        }
    }
}

Header parameters

Payload parameters

Send an empty payload.

WakeWordsReport

The device sends the WakeWordsReport event in response to a SetWakeWords directive sent by Alexa. (For wake word changes initiated by the device, including when the change is first triggered by a peripheral such as a companion app, the WakeWordsChanged event must be sent instead.)

Send the WakeWordsReport event both in cases of success and failure, reporting the wake words actually set on the device after processing the SetWakeWords directive.

Include the event object, without the messageId in its header, in the StateReport event states list when responding to the ReportState directive.

Sample message

{
  "event": {
    "header": {
      "namespace": "SpeechRecognizer",
      "name": "WakeWordsReport",
      "messageId": "{{STRING}}"
    },
    "payload": {
      "wakeWords": ["{{STRING}}", ...]
    }
  }
}

Header parameters

Payload parameters

WakeWordsChanged

The device sends the WakeWordsChanged event when the device initiates a wake word change. Such changes include those triggered by peripherals, such as third-party companion apps that instruct the device to change its wake words without otherwise informing Alexa directly. (For wake word changes initiated by Alexa through the SetWakeWords directive, send the WakeWordsReport event, instead.)

Sample message

{
  "event": {
    "header": {
      "namespace": "SpeechRecognizer",
      "name": "WakeWordsChanged",
      "messageId": "{{STRING}}"
    },
    "payload": {
      "wakeWords": ["{{STRING}}", ...]
    }
  }
}

Header parameters

Payload parameters

WakeWordConfirmationReport

The device sends the WakeWordConfirmationReport event in response to a SetWakeWordConfirmation directive sent by Alexa. (For setting changes initiated by the device, including when the change is first triggered by a peripheral such as a companion app, the wakewordschanged event must be sent instead.)

Send the WakeWordConfirmationReport event in cases of both success and failure, reporting the value actually set on the device after processing the SetWakeWordConfirmation directive.

Include the event object, without the messageId in its header, in the StateReport event states list when responding to the ReportState directive.

Sample message

{
  "event": {
    "header": {
      "namespace": "SpeechRecognizer",
      "name": "WakeWordConfirmationReport",
      "messageId": "{{STRING}}"
    },
    "payload": {
      "wakeWordConfirmation": "{{STRING}}"
    }
  }
}

Header parameters

Payload parameters

WakeWordConfirmationChanged

The device sends the WakeWordConfirmationChanged event when the device initiates a setting change. Such changes include those triggered by peripherals, such as third-party companion apps that instruct the device to change the setting without otherwise informing Alexa directly. (For changes initiated by Alexa through the SetWakeWordConfirmation directive, send the WakeWordConfirmationReport event instead.)

Sample message

{
  "event": {
    "header": {
      "namespace": "SpeechRecognizer",
      "name": "WakeWordsChanged",
      "messageId": "{{STRING}}"
    },
    "payload": {
      "wakeWordConfirmation": "{{STRING}}"
    }
  }
}

Header parameters

Payload parameters

SpeechConfirmationReport

The device sends the SpeechConfirmationReport event in response to a SetSpeechConfirmation directive sent by Alexa. (For setting changes initiated by the device, including when the change is first triggered by a peripheral such as a companion app, the SpeechConfirmationChanged event must be sent instead.)

Send the SpeechConfirmationReport event in cases of both success and failure, reporting the value actually set on the device after processing the SetSpeechConfirmation directive.

Include the event object, without the messageId in its header, in the StateReport event states list when responding to the ReportState directive.

Sample message

{
  "event": {
    "header": {
      "namespace": "SpeechRecognizer",
      "name": "SpeechConfirmationReport",
      "messageId": "{{STRING}}"
    },
    "payload": {
      "speechConfirmation": "{{STRING}}"
    }
  }
}

Header parameters

Payload parameters

SpeechConfirmationChanged

The device sends the SpeechConfirmationChanged event when the device initiates a setting change. Such changes include those triggered by peripherals, such as third-party companion apps that instruct the device to change the setting without otherwise informing Alexa directly. (For changes initiated by Alexa through the SetSpeechConfirmation directive, send the SpeechConfirmationReport event instead.)

Sample message

{
  "event": {
    "header": {
      "namespace": "SpeechRecognizer",
      "name": "SpeechConfirmationChanged",
      "messageId": "{{STRING}}"
    },
    "payload": {
      "speechConfirmation": "{{STRING}}"
    }
  }
}

Header parameters

Payload parameters