Use the Alexa Automated Testing Service (Beta)

Use the Alexa Automated Testing Service (ATS) in the Alexa developer console to test device control for your Alexa-connected smart home devices. ATS supports both smart home skills and Matter protocol implementations. The testing service lets you create test plans by using natural language instead of complex JSON definitions. Choose from pre-built templates or build custom tests that meet your specific requirements.

For details about other test tools, see Test and Debug Your Smart Home Skill.

How it works

ATS provides pre-defined templates, or you can create custom test plans. To build a test, you describe what you want to test as natural-language utterances and ATS creates a test plan. For example, to test a light bulb with color control, you might add the utterance, "Alexa, set the ${deviceName} to blue."

When you run the test plan, ATS sends voice commands to your device through Alexa and validates the device response and state change reports. After the test completes, you receive a detailed test report with pass and fail results and error descriptions for any failures.

ATS provides pre-defined templates for the following Alexa interfaces:

You can use the testing service throughout the development lifecycle to perform the following tasks:

  • Test your smart home skill or Matter integration during development.
  • Debug device integration issues.
  • Validate device behavior before you submit your product for Works with Alexa (WWA) certification.
  • Create regression tests for your skill-enabled devices.

Prerequisites

Before you start, make sure that you have the following items:

  • An Amazon developer account.
  • An Alexa-enabled device, such as an Amazon Echo. Make sure that you sign in to the Echo device with the same credentials as your Amazon developer account.
  • For skill-enabled devices, you must meet the following prerequisites:
  • For Matter-enabled devices, you must meet the following prerequisites:
    • Commission your device on the same Alexa fabric as your Echo device.
    • Have your device local identifier available.
  • At least one test device connected to your developer account. Make sure your device is discoverable and responds to Alexa commands.
  • Register the Echo device and the device under test to the same account. Verify both devices in the Alexa app.

Steps to run a pre-built test

Follow these steps to create and run a test plan that uses a pre-built template. The testing service displays the Echo devices and devices under test registered to your Alexa app.

  1. Create a test plan from a template.
  2. Run the test plan.
  3. View the test results.

Step 1: Create a test plan from a template

To run automated tests, you first define a test plan. If your device supports one of the capabilities defined by a pre-built template, you can create a test plan that uses the template. If your device supports additional capabilities or test scenarios, you can customize the test plan. For more details, follow the instructions in Customize a pre-built test plan. If your device has capabilities not supported by a template, see Create a custom test plan.

To create a test plan from a template in the Alexa developer console

  1. Sign in to the Alexa developer console.
  2. On the Skills tab, under SKILL NAME, scroll to find your smart home skill.
  3. Under ACTIONS, expand the dropdown list in your skill's row, and then select Test.
  4. On the banner that says, Try the Alexa Automated Testing Service (Beta) …., choose the ATS tool.
  5. For Login Required, choose Log in with Amazon, and then enter your Amazon developer credentials to sign in to ATS.
  6. For Click 'Allow' to Sign-In to Open Automation Kit, choose Allow.
  7. Under Create a New Test Plan, for Select Template, select a template that matches a capability of your device, such as Power Controller, and then choose Create from Template.
  8. On Create Test Plan, under Test Plan Name, enter a descriptive name for your test plan.
    Valid values: 1–50 characters including letters, numbers, underscores, and hyphens.
  9. To review the pre-built tests in the plan, expand each test case.
    You can also see a preview of the test plan in the JSON Preview panel.
  10. To save the test plan, choose Create Test Plan.

Step 2: Run the test plan

Select and run your test plan on ATS. You can run test plans based on pre-built or custom test cases.

To run a test in the Alexa developer console

  1. Under Run a Test, choose the following items:
    1. For Test Plan Name, select the test plan that you created.
    2. For Device Name, select the name of your device under test.
    3. For Alexa Device, select the name of your Alexa-enabled device.
  2. To run the test, choose Run Test.
  3. Wait for the tests to complete.
  4. If an error appears at the top of the page, correct the error, and then rerun the test plan.

Step 3: View test results

After the tests in the selected test plan complete, the testing service generates a test report that shows the overall test results, individual test case results, and error messages for any failures. The report also includes detailed step-by-step execution and device state information.

When a test case fails, use the error messages to help debug the issues. For more help, see Alexa Automated Testing Service Troubleshooting Guide.

To view the test report for a specific test run

  1. In the test run history table, navigate to the most recent entry of your test plan with the status COMPLETED.
  2. Under ACTIONS in your test plan's row, choose View Report.
  3. Under Test Report, view the test run ID, test plan name, and overall test status.
  4. Under Device Information, view the device name.
  5. Under Test Summary, view the total number of test cases run, number of test cases that succeeded, and the number that failed.
  6. Under Test Cases, you can view the status of each test case: Success or Failed. If the test case failed, the report includes an error message.
  7. To expand the test case results, choose the arrow icon ().
    You can view the results of each step in the test case.
  8. To return to the Alexa Automated Testing Service page, choose Back.
  9. For failed test cases, update your skill or device, and then rerun the test plan.

Test status values

During your test run, you might see one of the following statuses in the test run history table.

Status Description

IN PROGRESS

The test cases in the specified test plan are running.

COMPLETED

Test run finished. The results are ready to view.

TERMINATED

An internal error occurred during test execution.

Download test reports

The testing service deletes test reports after 30 days. Follow these steps to download your test report.

To download your test report

  1. In the test run history table, navigate to the most recent entry of your test plan with the status COMPLETED.
  2. Under ACTIONS in your test plan's row, choose Download Report.
  3. The JSON-formatted report downloads to your default download location.

View test plans

Before you run a test, you can view the test plan.

To view your test plan

  1. Under Run a Test, for Test Plan Name, select the test plan that you want to review.
  2. To view the test plan, choose Preview Test Plan.

Save your test plan

You might want to include your test plan in your CI/CD pipeline. Follow these steps to download your test plan. If your test plan expires, you can create a new test plan based on the saved version.

To save your test plan

  1. In the test run history table, navigate to the most recent entry of your test plan with the status COMPLETED.
  2. Under ACTIONS in your test plan's row, choose View Test Plan.
    The JSON-formatted test plan opens in a separate browser tab or window.
  3. To save your test plan to your PC or Mac, choose Save.
Was this page helpful?

Last updated: May 01, 2026