Set up an Endpoint-based A/B Test

You control an endpoint-based A/B test by adding conditional statements to the skill code of your live skill. These statements branch your skill into two versions, a control version (C) and a treatment (T1) version.

After you set up this logic and enable the test, your skill customers randomly receive one of the two versions when they launch your skill. After the test is over, you can analyze the metrics that you collected and decide which skill version you want to continue using.

About this tutorial

This tutorial walks you through how to set up an endpoint-based A/B test from start to finish, using the ASK Command Line Interface (CLI).

In the following steps, you create a test that measures the efficacy of two different ISP Upsell statements by tracking the following test metrics:

• Customer Friction
• ISP : Sales
• Skill Next Day Retention

To complete this task, you assign one Upsell statement to the C version of your skill and assign the other Upsell statement to the T1 version of your skill.

Tutorial criteria

To complete this tutorial exactly as-is, you must meet the following prerequisites:

• Your skill uses In-Skill purchasing (ISP) and has an ISP status of published.
• You skill has configured an ISP Upsell message.

If you don't use ISP in your skill

You can still follow this tutorial if your skill doesn't use ISP. To do this, use the Generic Code Example instead of the ISP Code Example, provided in Step 2: Update your skill code. Configure the generic code example to accommodate the skill code logic that you want to test.

Endpoint-based A/B test eligibility criteria

To run an endpoint-based A/B test, you must meet the following eligibility criteria.

Procedure overview

The following task list outlines the procedure that you must follow to complete this tutorial.

• Step 1: Create A/B test
• Step 2: Update skill code
• Step 3: (Optional) Evaluate A/B test
• Step 4: Start A/B test
• Step 5: Analyze A/B test results
• Step 6: (Optional) Increase exposure percentage
• Step 7: Stop A/B test

Step 1: Create A/B test

You use the ASK CLI to create an A/B test with your test metrics.

To create an A/B test

1. Create an A/B test with the following command.

   If you aren't testing ISP Upsell messages, modify the following create-experiment parameters accordingly.

   ask smapi create-experiment --skill-id <skill-id> --experiment-name "my_experiment" --experiment-description "My first A/B test" --experiment-type ENDPOINT_BASED --experiment-planned-duration "P3W" --experiment-exposure-percentage 10 --experiment-metric-configurations "[{\"name\":\"Customer Friction\",\"metricTypes\":[\"KEY\"],\"expectedChange\":\"DECREASE\"},{\"name\":\"ISP : Sales\",\"metricTypes\":[\"KEY\"],\"expectedChange\":\"INCREASE\"},{\"name\":\"Skill Next Day Retention\",\"metricTypes\":[\"GUARDRAIL\"],\"expectedChange\":\"INCREASE\"}]" --debug                  
   

   Note: If you want to test your experiment before you expose your new experience to your customers, you must use an experiment-exposure-percentage value of 0 in your create-experiment command

2. Find your experiment ID.

   After you create your test, you get the following output that contains your experiment ID, as shown below in red. You need this value in the next step.

    {
      "key": "location",
      "value": "/v0/skills/amzn1.ask.skill.66424560-4gh4-4358-94b1-741ff424e423/experiments/a556651f-f432-406c-a6a6-c5e74fd654fd"
    },

   Note: For the remainder of this tutorial, use the same experiment ID that you generated in this step. Also, make sure that you continue using the same skill ID associated with your test.

Step 2: Update skill code

Now that you have created an A/B test, branch your skill code into conditional statements to handle your C and T1 logic.

To branch your skill code

1. Add the following code example to your skill code.
   • If you're testing your ISP Upsell message, choose ISP Code Example.
   • If you're running a custom test, choose Generic Code Example.
   • ISP Code Example
   • Generic Code Example
   • Request Example
   ISP Example

   NodeJS example

    const UpsellRequestHandler = {
      canHandle(handlerInput) {
          return handlerInput.requestEnvelope.request.type === 'UpsellRequest';
      },
      handle(handlerInput) {
   
        const test = handlerInput.requestEnvelope.context.Experimentation.activeExperiments[0];
        if (test) {
   
          const treatment = test.treatmentId;
   
          if (treatment == interfaces.alexa.experimentation.TreatmentId.T1) {
            return handlerInput.responseBuilder.addDirective({
                type: "Connections.SendRequest",
                name: "Upsell",
                payload: {
                    InSkillProduct: {
                        productId: "myISP_productId_1"                       
                    },
                upsellMessage: requestAttributes.t("TREATMENT_UPSELL_MSG")                        
                }})
              .shouldEndSession(false)
              .getResponse();
          } else {
            return handlerInput.responseBuilder.addDirective({
                type: "Connections.SendRequest",
                name: "Upsell",
                payload: {
                    InSkillProduct: {
                        productId: "myISP_productId_2"                       
                    },
                upsellMessage: requestAttributes.t("CONTROL_UPSELL_MSG")                        
                }})
              .shouldEndSession(false)
              .getResponse();
          }
        } else { // Customer is not exposed to experiment, defaults to control behavior
          return handlerInput.responseBuilder.addDirective({
            type: "Connections.SendRequest",
            name: "Upsell",
            payload: {
                InSkillProduct: {
                    productId: "myISP_productId_1"                       
                },
            upsellMessage: requestAttributes.t("CONTROL_UPSELL_MSG")                        
            }})
          .shouldEndSession(false)
          .getResponse();
        }                    
      }
     };
   

   Generic Example

   NodeJS example

    const test = handlerInput.requestEnvelope.context.Experimentation.activeExperiments[0];
          if (test) {
   
            const treatment = test.treatmentId;
   
            if (treatment == interfaces.alexa.experimentation.TreatmentId.T1) {
              return handlerInput.responseBuilder.speak("treatment response")
                    .getResponse();
            } else {
              return handlerInput.responseBuilder.speak("control response")
                    .getResponse();
            }
          } else  {
            return handlerInput.responseBuilder.speak("not exposed to treatment")
                  .getResponse();
          }
   

   Request Example

   Your request payload contains an activeExperiments object with the experiment ID that generated when you created your A/B test. The treatmentId object contains the treatmentId. This value is either C or T1.

    {
      "version": "1.0",
      "session": {},
      "context": {
        "experimentation": {
          "activeExperiments": [
            {
              "id": "Experiment ID",
              "treatmentId": "T1"
            }
          ]
        }
      },
      "request": {}
     }
   

2. Modify the following sections in the copied code.

   • ISP Code Example
   • Generic Code Example
   ISP Example

   • Experiment ID – Find and replace any instances of <<experiment_id>> with the experiment ID that you generated earlier.
   • Upsell message – Assign different Upsell messages to both the TREATMENT_UPSELL_MSG and the CONTROL_UPSELL_MSG.
   • ISP Product ID – Provide a valid ISP Product ID in the C and T1 InSkillProduct object. Make sure your product IDs Live, meaning your product is certified and published. For more details about ISP product IDs, see Add ISP Support to Your Skill Code.

   Generic Example

   • Experiment ID – Find and replace any instances of <<experiment_id>> with the experiment ID that you generated in step 1.
   • Custom code – Modify your custom code as needed.
3. Save your skill code.

Step 3: (Optional) Evaluate A/B test

Before you launch your A/B test live to real customers, make sure it's working as intended. You do this by setting the customer treatment override value of your developer account to T1. After you set this value, you receive your treatment experience when you launch your skill.

To evaluate your A/B test

1. Change your customer treatment override value to T1 with the following command.

   ask smapi set-customer-treatment-override --skill-id <skill-id> --experiment-id <experiment-id> --treatment-id T1

   Note: You can't change your treatment after the test starts. If you don't set your own customer treatment override before you start your test, your account is randomly assigned the T1 treatment or C treatment, similar to any other customer.

2. Start your test with the following command.

   A test can take up to 10 minutes before it starts.

   ask smapi manage-experiment-state --skill-id <skill-id> --experiment-id <experiment-id> --target-state ENABLED

3. Open your skill in the ASK simulator and verify the test is working for you.

   When you run your skill, you receive the T1 experience of your test. For example, in context of this tutorial, you see the TREATMENT_UPSELL_MSG displayed in your skill instead of the CONTROL_UPSELL_MSG.

Step 4: Start A/B test

Starting your A/B test pushes your T1 experience live to your customers.

• Start A/B test after evaluating a test
• Start A/B test without evaluating a test

ask smapi update-exposure --skill-id <skill-id> --experiment-id <experiment-id> --exposure-percentage 10
ask smapi manage-experiment-state --skill-id <skill-id> --experiment-id <experiment-id> --target-state RUNNING

Step 5: Analyze A/B test results

You can analyze your A/B test results in two ways, while your test is running or after it finishes. Note that some metrics can take from two to five days to be available for review.

To analyze your A/B test results

1. Check if your metrics are available with the following command.

   ask smapi list-experiment-metric-snapshots --skill-id <skill-id> --experiment-id <experiment-id>

   After running this command, you get a response that returns your experiment ID with an associated metricSnapshotId.

2. Use the returned metricSnapshotId to pull your detailed analytics report with the following command.

   ask smapi get-experiment-metric-snapshot --skill-id <skill-id> --experiment-id <experiment-id> --metric-snapshot-id <metric-snapshot-id>

Step 6: (Optional) Increase exposure percentage

After you analyze your data, you might find that it doesn't lead to any conclusions. If you want to collect more data, you can increase your exposure percentage to include more customers in your T1 group. After you do this, you can continue to run your A/B test.

To increase your exposure percentage

Increase your exposure percentage with the following command.

ask smapi update-exposure --skill-id <skill-id> --experiment-id <experiment-id> --exposure-percentage 20

Step 7: Stop A/B test

If your A/B test results are satisfactory, change your experiment state to STOPPED. You can also choose to stop your test if you noticed a problem with it.

After you stop a test, it no longer collects metrics.

To stop your A/B test

Change your test status to STOPPED with the following command.

ask smapi manage-experiment-state --skill-id <skill-id> --experiment-id <experiment-id> --target-state STOPPED

To rerun your test, you must create an entirely new test, starting from step 1.

Related topics

• About A/B testing
• Configuration Attributes and Metrics
• Troubleshoot A/B Tests

Last updated: Oct 13, 2023