Set up an A/B Test using the Developer Console

You can create an endpoint-based test using the ASK developer console. After you set up and enable your test, your skill customers randomly receive one of the two versions when they launch your skill. After the test is over, you can analyze the metrics that you collected and decide which skill version you want to continue using.

About this tutorial

This tutorial walks you through how to set up an A/B test using the developer console.

Prerequisites

A/B testing eligibility criteria

To run an A/B test, you must meet the following eligibility criteria.

Tutorial criteria

The specific changes you must make between your skill versions aren't covered in this tutorial. Before you start this tutorial, you must decide what changes are appropriate for your skill and decide the purpose of your A/B test.

Procedure overview

The following task list outlines the procedure that you must follow to complete this tutorial.

Step 1: Create A/B test

You can use the developer console to create an A/B test with the test metrics you selected.

To create an A/B test using the developer console

  1. Sign in to the Alexa developer console.
  2. Select the Certification tab.
  3. In the left-hand navigation menu, select the A/B Testing Tab.
  4. On the A/B testing page click Create.
  5. Fill in the information required to create your A/B test.
    • Name – A unique name for your A/B test. You can't use an identical name for two tests. However, you can use the same name if you delete one of the experiments. Maximum 255 characters.
    • Hypothesis – A description of the hypothesis you're evaluating in your test. Only used for reference purposes and doesn't impact the experiment outcome.
    • Experiment type
      • Endpoint-based experiment – Run an endpoint-based test.
    • Control – These fields auto populate depending on what type of experiment type you choose.
    • Metrics A list of accepted test metrics that you're testing. You must designate each metric with either a KEY or GUARDRAIL value. Each metric must also contain an expected metric direction of either INCREASE or DECREASE.
      • Type – Choose Guardrail or Key.
      • Direction – Choose INCREASE or DECREASE.

  6. Click Create.

    You're redirected back to the A/B testing page. Your test details appear in a table row on this page.

Step 2: Update your A/B test

Use the following instructions to configure an override for your developer account in the A/B test.

  • Use Experiment Overrides – Toggle ON to assign your developer account to a specific control group in your test. Otherwise, your account is randomly assigned one of the two groups when you start your test. Overrides can't be removed after you start your test.
    • Control Group – Assign your account to the control group of your test.
    • Treatment Group – Assign your account to the treatment group of your test.

Step 3: Separate T1 and C behavior

Use the following instructions to separate the T1 and C behavior of your skill.

Endpoint-based

When you run an endpoint-based A/B test, you must branch your skill code into your T1 and C versions. The following code example illustrates how to do this.

Node.js example

const test = handlerInput.requestEnvelope.context.Experimentation.activeExperiments[0];
     if (test) {

       const treatment = test.treatmentId;

       if (treatment == interfaces.alexa.experimentation.TreatmentId.T1) {
         return handlerInput.responseBuilder.speak("treatment response")
               .getResponse();
       } else {
         return handlerInput.responseBuilder.speak("control response")
               .getResponse();
       }
     } else  {
       return handlerInput.responseBuilder.speak("not exposed to treatment")
             .getResponse();
     }

Step 4: (Optional) Evaluate A/B test

Before you launch your A/B test live to real customers, you should test and verify that it's working as intended. You do this by setting the customer treatment override value of your developer account to T1. After you set this value, you receive your treatment experience when you launch your skill.

To evaluate your A/B test

  1. Sign in to the Alexa developer console.
  2. Select the Certification tab.
  3. In the left-hand navigation menu, select the A/B Testing Tab.
  4. In the A/B testing table, find the test you want to edit, click the drop down box and then select Start.
  5. In the pop-up box, choose Pilot this test.

  6. Click Start.

    When you start your test, your skill launches with the settings you choose in step one, Step 1: Create an A/B test.

    When you're done testing your test, see Finish Test.

Step 5: Start A/B test

Starting your A/B test pushes your T1 experience live to your customers.

To start your A/B test

  1. Sign in to the Alexa developer console.
  2. Select the Certification tab.
  3. In the left-hand navigation menu, select the A/B Testing Tab.
  4. In the A/B testing table, find the test you want to edit, click the drop down box and then select Start.

  5. In the pop-up box, choose Start this test.

    When you launch your test, it starts with the settings you set in Step 2: Create A/B test.

    When your done running your test, see Finish A/B test.

Step 6: View A/B test analytics

You can analyze your A/B test results in two ways, while your test is running or after it finishes. Note that some metrics can take from two to five days to be available for review.

To analyze your A/B test

  1. Sign in to the Alexa developer console.
  2. Select the Analytics tab.
  3. In the left-hand navigation menu, select the A/B Testing tab.

  4. In the Test and Select Date Range drop downs, choose the test name and date that you want to view analytics for.

    Your data displays in a table on the page.

  5. (Optional) Select Edit or Complete to modify the current test.

Step 7: (Optional) Increase exposure percentage

After you analyze your data, you might find that it doesn't lead to any conclusions. If you want to collect more data, you can increase your exposure percentage to include more customers in your T1 group. After you do this, you can continue to run your test.

To increase your exposure percentage

  1. Sign in to the Alexa developer console.
  2. Select the Analytics tab.
  3. In the left-hand navigation menu, select the A/B Testing Tab.
  4. In the A/B testing table, find the test you want to update, click the drop down box and then select Edit.
  5. Update your Traffic Exposure.
  6. Click Save.

Step 8: Finish A/B test

If your A/B test results are satisfactory, change your test state to COMPLETED After you do this, your test no longer collects any metrics.

To finish your A/B test

  1. Sign in to the Alexa developer console.
  2. Select the Certification tab.
  3. In the left-hand navigation menu, select the A/B Testing Tab.
  4. In the A/B testing table, find the test you want to edit, click the drop down box and then select Complete.
  5. In the pop-up box, choose:
    • Stop this A/B test – Stops the test. The treatment version of your skill isn't published live.
  6. Select Complete.
Was this page helpful?

Last updated: Oct 13, 2023