Create Custom Test Plans

Important: Amazon offers this feature as beta. The feature might change as Amazon receives feedback and iterates on the functionality.

Use the Alexa Automated Testing Service (ATS) to create test plans that define voice commands and expected outcomes for your smart home device tests. Review test plan best practices and learn how to create custom test plans.

Best practices for test plan creation

Well-structured test plans improve test reliability and make results easier to interpret. Follow these best practices to create effective test plans:

Use descriptive names for test cases and steps.
Group test steps by capability or common user interactions.
Limit the number of capabilities in one test case.
Include setup steps before testing other capabilities. For example, turn on a device before testing its dimming or color capabilities.
Include both success and error scenarios.
Use realistic utterances that reflect how users naturally speak. Avoid overly complex utterances.
Validate both Alexa responses and device states.

Tip: Before you create custom test plans, review the pre-built templates to see examples of effective test steps and expected outcomes.

Test plan composition

A test plan consists of three levels:

Test Plan – The top-level container for all your tests.
Test Cases – Logical groupings of related test steps.
Test Steps – Individual test steps with utterances to start the test and validations for the response.

An example test plan to test lights might consist of the following test cases:

Test Plan: Bedroom-Lights-Test
- Test Case – Power control
  - Test Step – Turn on light
  - Test Step – Turn light off
- Test Case – Brightness control
  - Test Step – Turn on light
  - Test Step – Set brightness to 50%
  - Test Step – Dim the light

Organize your test cases into logical groups that test one Alexa capability at a time. Order test steps from simple to complex.

Use the following common test patterns.

Test pattern	Description
Setup → Action → Verify	Turn on device, change setting, verify state.
Boundary testing	Test minimum, maximum, and mid-range values.
State transitions	Test moving between different states in test steps.
Error scenarios	Test invalid commands or unavailable features.

ATS displays the JSON formatted test plan as you add test cases. The following example shows the test plan structure.

{
    "testPlanName": "name",
    "testCases": [{
        "testCaseName": "test case name",
        "description": "description of the test case",
        "testSteps": [{
            "utterance": "voice command with wake word",
            "waittime": 8,
            "validation": [{
                    "type": "TTS_FUZZY_MATCH",
                    "expectedOutput": "Alexa response"
                },
                {
                    "type": "STATE_MATCH",
                    "target": {
                        "namespace": "Alexa interface",
                        "name": "reported property name"
                    },
                    "expectedOutput": "exact match value as string or object"
                },
                {
                    "type": "LLM_STATE_MATCH",
                    "validationCriteria": "semantic description of expected response"
                }
            ]
        }]
    }]
}

Create a custom test plan

Follow these steps to create and run a custom test plan. The testing service displays the Echo devices and devices under test registered to your Alexa app.

Note: At this time, you can't modify a test plan after you save it.

To create a custom test plan in the Alexa developer console

Sign in to the Alexa developer console.
On the Skills tab, under SKILL NAME, scroll to find your smart home skill.
Under ACTIONS, expand the dropdown list in your skill's row, and then select Test.
On the banner that says, Try the Alexa Automated Testing Service (Beta) …., choose the ATS tool.
For Login Required, choose Log in with Amazon, and then enter your Amazon developer credentials to sign in to ATS.
For Click 'Allow' to Sign-In to Open Automation Kit, choose Allow.
To enable auto-suggestions for your device type, under Select Device, select the target device for the test plan.
Under Create a New Test Plan, leave Select Template blank and choose Create Blank Test Plan.
On Create Test Plan, under Test Plan Name, enter a descriptive name for your test plan.
Valid values: 1–50 characters including letters, numbers, underscores, and hyphens.
For each test case, enter the following information:
1. For Test Case Name, enter a unique test case name that describes the test case.
  Valid values: 1–50 characters including letters, numbers, underscores, hyphens, periods, and spaces.
2. (Optional) For Description, enter a brief summary of the purpose of the test case.
  Valid values: 0–200 characters including letters, numbers, punctuation, and spaces.
3. For Test Steps, enter the following information:
  1. For Utterance, enter the voice command to send to Alexa.
    Valid values: Letters, numbers, spaces, and these special characters: _ $ { } , . ? ! '.
    Choose + deviceName to insert a device name placeholder. When the test runs, the testing service inserts the actual device name into the utterance.
  2. For Wait Time (seconds), enter how long the testing service waits for a response before proceeding. For more details, see Wait time guidance.
  3. For Validations, select the validation type. For more details, see Add validations.
  4. To add another validation, choose + Add Validation.
4. To add another test step, choose + Add Test Step.
To add another test case, choose + Add Test Case.
Review your test plan carefully because you can't modify the test plan after you save it.
To save the test plan, choose Create Test Plan.
To run the test plan, see Run the test plan

Wait time guidance

Set the Wait Time (seconds) to the number of seconds you expect your device to respond to the utterance. Response times vary based on the device type. If your device responds quickly, set a short wait time, such as 3–5 seconds for a light bulb or switch. If your device requires more time to complete the control request, use a longer duration, such as 6–8 seconds for thermostats and locks. Other devices, such as appliances, might require even longer wait times.

For most interfaces, Alexa waits 8 seconds for the response before timing out. Some interfaces require a shorter response time. When your device doesn't respond within the timeout value, Alexa tells the user, "Sorry, <device name placeholder> is not responding." For more details, see Alexa.Response.

Add validations

ATS supports three types of validations: Alexa voice response, exact state match, and natural language state match. Always include at least one validation per test step. Make sure to test both success and error paths.

Alexa voice response validation

Use Alexa voice response validation to verify the spoken response. To validate that the Alexa voice response matches the expected output, select Alexa Voice Response (TTS Fuzzy Match) as the validation type. In the Expected Alexa Voice Output field, enter the response you expect Alexa to return. Use short, specific phrases to take advantage of how fuzzy matching works.

ATS uses fuzzy matching, which ignores case, extra words, and punctuation. The test passes if the expected output appears anywhere in the response.

The following examples show how fuzzy matching works.

Expected output	Matches
OK	OK, Okay, ok.
bedroom light	I turned on the bedroom light.
not responding	Sorry, bedroom light is not responding.

Tip: To make sure your test plan covers all expected behaviors, test for both success and error responses.

Device state validation

Use device state validations for exact value matches. To validate that a property value exactly matches your expected value for a specified interface, select Device State Change (State Match) as the validation type. Then, enter the following information.

For Namespace, enter a valid Alexa interface for the capability supported by your device. For interface names, see List of Alexa Interfaces.
For Property Name, enter a reportable property defined in the Alexa interface. For property names, see the documentation for the specific interface.
For Expected Output, enter the exact property value you expect the test to return. For complex values, use JSON format.

Tip: To make sure your device responds correctly, always validate state changes after control commands.

Natural language state validation

Use natural language state validations for expected states that are complex or don't have exact value matches. To validate that the device state closely matches your expected value, select Device State Change (Natural Language State Match) as the validation type.

For Validation Criteria (Optional), enter a clear, specific description of the expected state using natural language. For example:

The light should be dimmer than 50 percent brightness.
The thermostat should be in cooling mode with temperature around 70 degrees Fahrenheit.

Note: If you leave the criteria blank, Alexa infers the expected response, which might produce less precise validation results. For the best test quality, Amazon recommends that you enter the criteria.

Customize a pre-built test plan

Follow these steps to modify a test plan that you create from a pre-built template.

Note: At this time, you can't modify a test plan after you save it.

To customize a pre-built test plan in the Alexa developer console

Sign in to the Alexa developer console.
On the Skills tab, under SKILL NAME, scroll to find your smart home skill.
Under ACTIONS, expand the dropdown list in your skill's row, and then select Test.
On the banner that says, Try the Alexa Automated Testing Service (Beta) …., choose the ATS tool.
For Login Required, choose Log in with Amazon, and then enter your Amazon developer credentials to sign in to ATS.
For Click 'Allow' to Sign-In to Open Automation Kit, choose Allow.
Under Create a New Test Plan, for Select Template, select a template that matches a capability of your device, and then choose Create from Template.
On Create Test Plan, under Test Plan Name, enter a new name for your test plan, if needed.
Valid values: 1–50 characters including letters, numbers, underscores, and hyphens.
For each test case, update the following information as applicable:
1. For Test Case Name, enter a unique test case name that describes the test case.
  Valid values: 1–50 characters including letters, numbers, underscores, hyphens, periods, and spaces.
2. (Optional) For Description, enter a brief summary of the purpose of the test case.
  Valid values: 0–200 characters including letters, numbers, punctuation, and spaces.
3. For Test Steps, update the following information as applicable:
  1. For Utterance, enter the voice command to send to Alexa.
    Valid values: Letters, numbers, spaces, and these special characters: _ $ { } , . ? ! '.
    Choose + deviceName to insert a device name placeholder. When the test runs, the testing service inserts the actual device name into the utterance.
  2. For Wait Time (seconds), enter how long the testing service waits for a response before proceeding. For more details, see Wait time guidance.
  3. For Validations, select the validation type. For more details, see Add validations.
  4. To add another validation, choose + Add Validation.
4. To add another test step, choose + Add Test Step.
To add another test case, choose + Add Test Case.
Review your test plan carefully because you can't modify the test plan after you save it.
To save the test plan, choose Create Test Plan.
To run the test plan, see Run the test plan

Was this page helpful?

Provide feedback

Last updated: May 01, 2026