Create and Manage A/B Tests in the Alexa Developer Console

You can use the developer console to create and manage A/B tests for your skills.

For a step-by-step tutorial on how to create an A/B test using these instructions, see Set up an A/B Test using the Developer Console.

Create A/B test

You can use the developer console to create an A/B test with the test metrics you selected.

To create an experiment

  1. Log in to the Alexa developer console.
  2. Select the Certification tab.
  3. In the left-hand navigation menu, select the A/B Testing tab.
  4. On the A/B testing page click Create.
  5. Fill in the required information to create your A/B test.

  6. Click Create.

    You're redirected back to the A/B testing page. Your test details appear in a table row on this page.

Update A/B test

To update your A/B test

  1. Log in to the Alexa developer console.
  2. Select the Certification tab.
  3. In the left-hand navigation menu, select the A/B Testing tab.
  4. In the A/B testing table, find the experiment you want to edit, click the drop down box and then select Edit.
  5. Edit the necessary information required to update your test, as described in Create test. You can edit all fields expect the control.
  6. Click Save.

(Optional) Evaluate A/B test

Before you launch your A/B test live to real customers, you should test and verify that it's working as intended. You do this by setting your own customer treatment override value to T1. After you set this value, you receive your treatment experience when you launch your skill.

To evaluate your A/B test

  1. Log in to the Alexa developer console.
  2. Select the Certification tab.
  3. In the left-hand navigation menu, select the A/B Testing tab.
  4. In the A/B testing table, find the experiment you want to edit, click the drop down box and then select Start.
  5. In the pop-up box, choose Pilot this test.

  6. Click Start.

    When you launch your test, your skill launches with the settings you chose in Create A/B Test.

  7. Select Complete.

Start A/B test

Starting your A/B test pushes your T1 experience live to your customers.

To start your A/B test

  1. Log in to the Alexa developer console.
  2. Select the Certification tab.
  3. In the left-hand navigation menu, select the A/B Testing tab.
  4. In the A/B testing table, find the experiment you want to edit, click the drop down box and then select Start.

  5. In the pop-up box, choose Start this test.

    When you launch your experiment, your experiment starts with the settings you set in create your experiment.

  6. Select Complete.

View your A/B test analytics

You can analyze your A/B test results in two ways, while your test is running or after it finishes. Note that some metrics can take from two to five days to be available for review.

To view your A/B testing analytics

  1. Log in to the Alexa developer console.
  2. Select the Analytics tab.
  3. In the left-hand navigation menu, select the A/B Testing tab.

  4. In the Test and Select Date Range drop downs, choose the test name and date that you want to view analytics for.

    Your data displays in a table below.

  5. (Optional) Select Edit or Complete to modify the current test.

Finish A/B test

If your A/B test results are satisfactory, change your test state to COMPLETED After you do this, your test no longer collects metrics.

To finish your A/B test

  1. Log in to the Alexa developer console.
  2. Select the Certification tab.
  3. In the left-hand navigation menu, select the A/B Testing Tab.
  4. In the A/B testing table, find the test you want to edit, click the drop down box and then select Complete.
  5. In the pop-up box, choose:
    • Stop this A/B test – Stops the test. The treatment version of your skill isn't published live.
  6. Select Complete.
Was this page helpful?

Last updated: Jul 17, 2024