A/B Testing CLI Commands
Use the following ASK CLI commands to create and manage A/B tests for your skills.
create-experiment
Creates a new A/B test for a skill.
Calls the Create A/B test SMAPI API.
Syntax:
ask smapi create-experiment [-s, --skill-id <skill-id>]
[--experiment-name <experiment-name>]
[--experiment-description <experiment-description>]
[--experiment-type <experiment-type>]
[--experiment-planned-duration <experiment-planned-duration>]
[--experiment-exposure-percentage <experiment-exposure-percentage>]
[--experiment-metric-configurations <experiment-metric-configurations>]
[--profile | -p <profile>]
[--debug]
Options:
- -s, --skill-id
- Required. The skill ID associated with your A/B test.
- --experiment-name
- Required. A unique name for your A/B test. You can't use an identical name for two tests. However, you can use the same name if you delete one of the tests. Maximum 255 characters.
- --experiment-description
- Required. A description of the hypothesis you set for your test. Only used for reference purposes and doesn't impact the test outcome.
- --experiment-type
- Required. The type of A/B test you want to run for your skill.
Accepted values:ENDPOINT_BASED
- Sets your tests T1 value to your skill version in theLIVE
stage. To separate your C and T1 behaviors, you must add conditional logic in your skill code. For more details about separating endpoint-based tests, see Set up an Endpoint-based A/B Experiment.
- --experiment-planned-duration
- Required. The number of weeks you're running an A/B test. Only used for reference purposes and doesn't impact the actual test length.
Accepted values: Any value between 1 and 4 weeks represented in the ISO-8601 format. For example, you represent a value of four weeks with the ISO-8601 value ofP4W
. - --experiment-exposure-percentage
- Required. The percentage of unique customers that you're including in your A/B test. This number doesn't have to match your total users. For example, if you have 100 total customers and you set your
exposurePercentage
to 40 percent, you're including 40 customers in your test. In this case, your test includes 20 customers in C and 20 customers in T1. The remaining 60 customers aren't included in the test and receive the default behavior equivalent to C, meaning they don't contribute to the test metrics.
Accepted values: Any value between0
to100
. - --experiment-metric-configurations
- Required. A list of accepted metrics that you're testing in your A/B test. You must designate each metric with either a
KEY
orGUARDRAIL
value. Each metric must also contain an expected metric direction of eitherINCREASE
orDECREASE
.
You can specify the JSON as a string or in a file. If you specify a file, you can specify it using either an absolute or relative path. For example, "$(cat {filepath})" or "file:{filepath}". - --profile, -p
- The ASK CLI profile to use. When you don't include this option, ASK CLI uses the default profile.
- --debug
- When you include this option, debug messages appear in the output of the command.
list-experiments
Gets a list of tests associated with a skill ID. Returns both completed and in-progress tests.
Calls the Get A/B test SMAPI API.
Syntax:
ask smapi list-experiments [-s,--skill-id <skill-id>]
[--next-token <next-token>]
[--max-results <max-results>]
[--profile | -p <profile>]
[--debug]
Options:
- -s, --skill-id
- Required. The skill ID associated with your A/B test.
- --next-token
- Optional. Token used to retrieve the next page of results.
- --max-results
- Optional. Sets the maximum number of results returned in the response body.
maxResults
shouldn't exceed the upper limit. The response might contain fewer results than maxResults, but it never contains more. - --profile, -p
- The ASK CLI profile to use. When you don't include this option, ASK CLI uses the default profile.
- --debug
- When you include this option, debug messages appear in the output of the command.
get-experiment
Retrieves detailed information about an existing test.
Calls the Get A/B test SMAPI API.
Syntax:
ask smapi get-experiment [-s,--skill-id <skill-id>]
[--experiment-id <experiment-id>]
[--profile | -p <profile>]
[--debug]
Options:
- -s, --skill-id
- Required. The skill ID associated with your A/B test.
- --experiment-id
- Required. The unique ID of the A/B test you want to retrieve details for.
- --profile, -p
- The ASK CLI profile to use. When you don't include this option, ASK CLI uses the default profile.
- --debug
- When you include this option, debug messages appear in the output of the command.
update-experiment
Updates the settings of an existing A/B test. You can use this command if your test is in the CREATED
state. You can't use this command to update the values of a test that's already started and in the RUNNING
state.
Calls the Update A/B test SMAPI API.
Syntax:
ask smapi update-experiment [-s,--skill-id <skill-id>]
[--experiment-id <experiment-id>]
[--experiment-description <experiment-description>]
[--experiment-planned-duration <experiment-planned-duration>]
[--experiment-exposure-percentage <experiment-exposure-percentage>]
[--experiment-metric-configurations <experiment-metric-configurations>]
[--profile | -p <profile>]
[--debug]
Options:
- -s, --skill-id
- Required. The skill ID associated with your A/B test.
- --experiment-description
- Required. A description of the hypothesis you set for your test. Only used for reference purposes and doesn't impact the test outcome.
- --experiment-planned-duration
- Required. The number of weeks you're running the test. Only used for reference purposes and doesn't impact the actual test length.
Accepted values: Any value between 1 and 4 weeks represented in the ISO-8601 format. For example, you represent a value of four weeks with the ISO-8601 value ofP4W
. - --experiment-exposure-percentage
- Required. The percentage of unique customers that you're including your A/B test. This number doesn't have to match your total users. For example, if you have 100 total customers and you set your
exposurePercentage
to 40 percent, you're including 40 customers in your test. in this case, your test includes 20 customers in C and 20 customers in T1. The remaining 60 customers aren't included in the test and receive the default behavior equivalent to C, meaning they don't contribute to the test metrics.
Accepted values: Any value between0
to100
. - --experiment-metric-configurations
- Required. A list of accepted test metrics you're using in your A/B test. You must designate each metric with either a
KEY
orGUARDRAIL
value. Each metric must also contain an expected metric direction of eitherINCREASE
orDECREASE
.
You can specify the JSON as a string or in a file. If you specify a file, you can specify it using either an absolute or relative path. For example, "$(cat {filepath})" or "file:{filepath}". - --profile, -p
- The ASK CLI profile to use. When you don't include this option, ASK CLI uses the default profile.
- --debug
- When you include this option, debug messages appear in the output of the command.
delete-experiment
Deletes an existing A/B test.
Calls the Delete A/B test SMAPI API.
Syntax:
ask smapi delete-experiment [-s,--skill-id <skill-id>]
[--experiment-id <experiment-id>]
[--profile | -p <profile>]
[--debug]
Options:
- -s, --skill-id
- Required. The skill ID associated with your A/B test.
- --experiment-id
- Required. The unique ID of the A/B test you want to delete.
update-exposure
Updates the exposure percentage value of an existing A/B test. You can only use this command on a test that's in the CREATED
or RUNNING
state.
Calls the Update exposure SMAPI API.
Syntax:
ask smapi update-exposure [-s,--skill-id <skill-id>]
[--experiment-id <experiment-id>]
[--exposure-percentage <exposure-percentage>]
[--profile | -p <profile>]
[--debug]
Options:
- -s, --skill-id
- Required. The skill ID associated with your A/B test.
- --experiment-id
- Required. The unique ID of the A/B test you want to update.
- --exposure-percentage
- Required. The new exposure percentage you want to update your test to. You can only increase the exposure percentage value of a running test, not decrease it.
Accepted values: Any value between0
and100
. This new value must be higher than your current exposure percentage. - --profile, -p
- The ASK CLI profile to use. When you don't include this option, ASK CLI uses the default profile.
- --debug
- When you include this option, debug messages appear in the output of the command.
get-customer-treatment-override
Retrieves the current customer treatment override value of an existing A/B test.
Calls the Get Customer Treatment Override SMAPI API.
Syntax:
ask smapi get-customer-treatment-override [-s,--skill-id <skill-id>]
[--experiment-id <experiment-id>]
[--profile | -p <profile>]
[--debug]
Options:
- -s, --skill-id
- Required. The skill ID associated with your A/B test.
- --experiment-id
- Required. The unique ID of the A/B test you want to retrieve details for.
- --profile, -p
- The ASK CLI profile to use. When you don't include this option, ASK CLI uses the default profile.
- --debug
- When you include this option, debug messages appear in the output of the command.
set-customer-treatment-override
Sets a new customer treatment override value of an existing A/B test.
Calls the Set Customer Treatment Override SMAPI API.
Syntax:
ask smapi set-customer-treatment-override [-s,--skill-id <skill-id>]
[--experiment-id <experiment-id>]
[--treatment-id <treatment-id>]
[--profile | -p <profile>]
[--debug]
Options:
- -s, --skill-id
- Required. The skill ID associated with your A/B test.
- --experiment-id
- Required. The unique ID of the A/B test you want to update.
- --treatment-id
- Required. The test group that you want to set for your skill.
Accepted values:- C – Control. The skill behavior you aren't testing. This group of customers continue to receive your current skill experience.
- T1 – Treatment. The skill behavior you're testing. This group of customers receive your new experimental skill experience.
- --profile, -p
- The ASK CLI profile to use. When you don't include this option, ASK CLI uses the default profile.
- --debug
- When you include this option, debug messages appear in the output of the command.
get-experiment-state
Retrieves the current state of an A/B test.
Calls the Get A/B test state SMAPI API.
Syntax:
ask smapi get-experiment-state [-s,--skill-id <skill-id>]
[--experiment-id <experiment-id>]
[--profile | -p <profile>]
[--debug]
Options:
- -s, --skill-id
- Required. The skill ID associated with your A/B test.
- --experiment-id
- Required. The unique ID of the A/B test you want to retrieve.
- --profile, -p
- The ASK CLI profile to use. When you don't include this option, ASK CLI uses the default profile.
- --debug
- When you include this option, debug messages appear in the output of the command.
manage-experiment-state
Updates the current state of an A/B test.
Calls the Update A/B test state SMAPI API.
Syntax:
ask smapi manage-experiment-state [-s,--skill-id <skill-id>]
[--experiment-id <experiment-id>]
[--target-state <target-state>]
[--profile | -p <profile>]
[--debug]
Options:
- -s, --skill-id
- Required. The skill ID associated with your A/B test.
- --experiment-id
- Required. The unique ID of the A/B test you want to update.
- --target-state
- Required. The new state you want to update your test to. For more details about how to use states, see Lifecycle of an experiment.
Accepted values:ENABLED
- Deploy your test. You can only transition to this state if your test is in theCREATED
state.RUNNING
- Start your test. You can only transition to this state if your test is in theCREATED
orENABLED
state.STOPPED
- Stop your test. You can only transition to this state if your test is in theENABLED
orRUNNING
state.
- --profile, -p
- The ASK CLI profile to use. When you don't include this option, ASK CLI uses the default profile.
- --debug
- When you include this option, debug messages appear in the output of the command.
list-experiment-metric-snapshots
Lists all metric snapshots associated with an experimentId
. A metric snapshot represents all the metric data that's available for your A/B test in a specific time range. To get detailed analytics about a test, use the get-experiment-metric-snapshot
command.
Calls the Get all metric snapshots SMAPI API.
Syntax:
ask smapi list-experiment-metric-snapshots [-s,--skill-id <skill-id>]
[--experiment-id <experiment-id>]
[--next-token <next-token>]
[--max-results <max-results>]
[--profile | -p <profile>]
[--debug]
Options:
- -s, --skill-id
- Required. The skill ID associated with your A/B test.
- --experiment-id
- Required. The unique ID of the A/B test you want to retrieve.
- --next-token
- Optional. Token used to retrieve the next page of results.
- --max-results
- Optional. Sets the maximum number of results returned in the response body.
maxResults
shouldn't exceed the upper limit. The response might contain fewer results than maxResults, but it never contains more. - --profile, -p
- The ASK CLI profile to use. When you don't include this option, ASK CLI uses the default profile.
- --debug
- When you include this option, debug messages appear in the output of the command.
get-experiment-metric-snapshot
Lists detailed metric data and analytics associated with the metricSnapshotId
of an A/B test.
Calls the Get metric snapshot data SMAPI API.
Syntax:
ask smapi get-experiment-metric-snapshot [-s,--skill-id <skill-id>]
[--experiment-id <experiment-id>]
[--metric-snapshot-id <metric-snapshot-id>]
[--profile | -p <profile>]
[--debug]
Options:
- -s, --skill-id
- Required. The skill ID associated with your A/B test.
- --experiment-id
- Required. The unique ID of the A/B test details you want to retrieve.
- --metric-snapshot-id
- Required. The unique ID of the test metric snapshot details you want to retrieve.
- --profile, -p
- The ASK CLI profile to use. When you don't include this option, ASK CLI uses the default profile.
- --debug
- When you include this option, debug messages appear in the output of the command.
Related topics
- About A/B tests
- Configuration Attributes and Metrics
- Troubleshoot A/B tests
- Quick Start: Alexa Skills Kit Command Line Interface (ASK CLI)
Last updated: Oct 13, 2023