Troubleshoot A/B Tests

The following page provides possible resolutions for common issues when running A/B tests.

Create A/B test

Issue: I don't see my experiment ID after I create an A/B test

Symptoms

When you create an A/B test using the ASK CLI or SMAPI APIs, you don't see your experiment ID in the response.

Try this

Your experiment ID returns in the response header instead of the response body. To see your experiment ID in the header response, create your test using the --debug flag.

For example

ask smapi create-experiment --skill-id <skill-id> --experiment-name "my_experiment" --debug

To retrieve the ID of an existing test, run the list-experiments command instead.

Issue: experiment-planned-duration throws an error when I create an A/B test

Symptoms

When you create an A/B test with the --experiment-planned-duration flag, you get an error stating that your test length isn't valid.

Try this

The --experiment-planned-duration flag requires that you represent your date in the ISO-8601 format. For example, you represent a value of four weeks with the ISO-8601 value of P4W. Make sure that your test length doesn't exceed 4 weeks.

For example

ask smapi create-experiment --skill-id <skill-id> --experiment-name "my_experiment" --experiment-planned-duration "P3W"

Issue: My skill passed certification, but I can't create an A/B test

Symptoms

Your skill satisfies the A/B testing eligibility criteria, but you still can't create a test.

Try this

Make sure your skill must passes Amazon's certification criteria, including adhering to Amazon's content guidelines. For more details on these content guidelines, see Policy Requirements.

Customer treatment overrides

Issue: I can't add a customer treatment override for other users

Symptoms

You get an error when you try to add another developer account to the customer treatment override of your A/B test.

Try this

You can't add external users to your customer treatment override, meaning you can only set the customer treatment override value of a skill that you own. To satisfy this requirement, your Customer ID and Vendor ID must match the information configured in the skill that you're running yours tests on. For example, you can set the override value of a skill you developed or add the override value of another developer that shares the same Vendor ID as your skill.

If you have verified that your Customer ID or Vendor ID matches your skill and you're still getting an error, make sure you're logged into the correct developer account.

To add a customer treatment override for a skill you own

  1. Authenticate your A/B test using the Login with Amazon (LWA) credentials that contain the correct Customer ID or Vendor ID. For example, by logging into the developer console with the relevant account.

  2. Run the set-customer-treatment-override CLI command and pass in the appropriate customer treatment override value for your test.

    You can only add 10 customer treatment overrides per test.

Issue: I can't change my customer treatment override during an active A/B test

Symptoms

You get an error when you try to use the set-customer-treatment-override CLI command to change your own customer override value.

Try this

You can't change your customer treatment override after an A/B test has started and is in the ENABLED state. To run a test with a new customer treatment override value, you must create a new test.

Analytics

Issue: My A/B test analytics data is incorrect

Symptoms

When you view your A/B test analytics, the data is incorrect or outdated.

Try this

Some metrics can take from two to five days to be available for review. If your data seems completely wrong, double check that you're passing in the correct skill ID and experiment ID in your list-experiment-metric-snapshots CLI command or when using the Get all metric snapshots API.

Other

Issue: I received a Treatment Allocation Alarm (TAA) during my A/B test

Symptoms

When you're running an A/B test, you might receive a Treatment Allocation Alarm (TAA) notification. A TAA is a warning that indicates there is a problem with the current test you're running.

For example, if you're running a test with 10M visitors, you should expect about 5M visitors in your control (C) and about 5M visitors in your treatment 1 (T1) for the entire analysis duration of that test. Here, you might receive a TAA alarm if the visitor balance between your C and T1 changes significantly from the expected allocation of control and treatment groups.

Try this

If you receive a TAA alarm, you should end your test and examine your trigger logic for any errors. If you continue to receive a TAA after adjusting your trigger logic, reach out to Alexa developer support.

Issue: I can't decrease my exposure percentage during an active A/B test

Symptoms

You get an error when trying to decrease your exposure percentage using the update-exposure CLI command.

Try this

You can't decrease your exposure percentage while an A/B test is running. To decrease your exposure percentage, you must create a new test.

Was this page helpful?

Last updated: Oct 13, 2023