A/B Testing CLI Commands

Note: Sign in to the developer console to build or publish your skill.

Use the following ASK CLI commands to create and manage A/B tests for your skills.

create-experiment

Creates a new A/B test for a skill.

Syntax:

ask smapi create-experiment [-s, --skill-id <skill-id>]
                            [--experiment-name <experiment-name>]
                            [--experiment-description <experiment-description>]   
                            [--experiment-type <experiment-type>]  
                            [--experiment-planned-duration <experiment-planned-duration>]  
                            [--experiment-exposure-percentage <experiment-exposure-percentage>]
                            [--experiment-metric-configurations <experiment-metric-configurations>]  
                            [--profile | -p <profile>]
                            [--debug]

Options:

-s, --skill-id

Required. The skill ID associated with your A/B test.

--experiment-name

Required. A unique name for your A/B test. You can't use an identical name for two tests. However, you can use the same name if you delete one of the tests. Maximum 255 characters.

--experiment-description

Required. A description of the hypothesis you set for your test. Only used for reference purposes and doesn't impact the test outcome.

--experiment-type

Required. The type of A/B test you want to run for your skill.

Accepted values:

ENDPOINT_BASED- Sets your tests T1 value to your skill version in the LIVE stage. To separate your C and T1 behaviors, you must add conditional logic in your skill code. For more details about separating endpoint-based tests, see Set up an Endpoint-based A/B Experiment.

--experiment-planned-duration

Required. The number of weeks you're running an A/B test. Only used for reference purposes and doesn't impact the actual test length.

Accepted values: Any value between 1 and 4 weeks represented in the ISO-8601 format. For example, you represent a value of four weeks with the ISO-8601 value of P4W.

--experiment-exposure-percentage

Required. The percentage of unique customers that you're including in your A/B test. This number doesn't have to match your total users. For example, if you have 100 total customers and you set your exposurePercentage to 40 percent, you're including 40 customers in your test. In this case, your test includes 20 customers in C and 20 customers in T1. The remaining 60 customers aren't included in the test and receive the default behavior equivalent to C, meaning they don't contribute to the test metrics.

Accepted values: Any value between 0 to 100.

--experiment-metric-configurations

Required. A list of accepted metrics that you're testing in your A/B test. You must designate each metric with either a KEY or GUARDRAIL value. Each metric must also contain an expected metric direction of either INCREASE or DECREASE.

You can specify the JSON as a string or in a file. If you specify a file, you can specify it using either an absolute or relative path. For example, "$(cat {filepath})" or "file:{filepath}".

--profile, -p

The ASK CLI profile to use. When you don't include this option, ASK CLI uses the default profile.

--debug

When you include this option, debug messages appear in the output of the command.

list-experiments

Gets a list of tests associated with a skill ID. Returns both completed and in-progress tests.

Calls the Get A/B test SMAPI API.

Syntax:

ask smapi list-experiments [-s,--skill-id <skill-id>]
                           [--next-token <next-token>]
                           [--max-results <max-results>]   
                           [--profile | -p <profile>]
                           [--debug]

Options:

-s, --skill-id: Required. The skill ID associated with your A/B test.
--next-token: Optional. Token used to retrieve the next page of results.
--max-results: Optional. Sets the maximum number of results returned in the response body. maxResults shouldn't exceed the upper limit. The response might contain fewer results than maxResults, but it never contains more.
--profile, -p: The ASK CLI profile to use. When you don't include this option, ASK CLI uses the default profile.
--debug: When you include this option, debug messages appear in the output of the command.

get-experiment

Retrieves detailed information about an existing test.

Calls the Get A/B test SMAPI API.

Syntax:

ask smapi get-experiment [-s,--skill-id <skill-id>]
                         [--experiment-id <experiment-id>]
                         [--profile | -p <profile>]
                         [--debug]

Options:

-s, --skill-id: Required. The skill ID associated with your A/B test.
--experiment-id: Required. The unique ID of the A/B test you want to retrieve details for.
--profile, -p: The ASK CLI profile to use. When you don't include this option, ASK CLI uses the default profile.
--debug: When you include this option, debug messages appear in the output of the command.

update-experiment

Updates the settings of an existing A/B test. You can use this command if your test is in the CREATED state. You can't use this command to update the values of a test that's already started and in the RUNNING state.

Calls the Update A/B test SMAPI API.

Syntax:

ask smapi update-experiment [-s,--skill-id <skill-id>]
                            [--experiment-id <experiment-id>]
                            [--experiment-description <experiment-description>]   
                            [--experiment-planned-duration <experiment-planned-duration>]  
                            [--experiment-exposure-percentage <experiment-exposure-percentage>]
                            [--experiment-metric-configurations <experiment-metric-configurations>]        
                            [--profile | -p <profile>]
                            [--debug]

Options:

-s, --skill-id: Required. The skill ID associated with your A/B test.
--experiment-description: Required. A description of the hypothesis you set for your test. Only used for reference purposes and doesn't impact the test outcome.
--experiment-planned-duration: Required. The number of weeks you're running the test. Only used for reference purposes and doesn't impact the actual test length.

Accepted values: Any value between 1 and 4 weeks represented in the ISO-8601 format. For example, you represent a value of four weeks with the ISO-8601 value of P4W.
--experiment-exposure-percentage: Required. The percentage of unique customers that you're including your A/B test. This number doesn't have to match your total users. For example, if you have 100 total customers and you set your exposurePercentage to 40 percent, you're including 40 customers in your test. in this case, your test includes 20 customers in C and 20 customers in T1. The remaining 60 customers aren't included in the test and receive the default behavior equivalent to C, meaning they don't contribute to the test metrics.

Accepted values: Any value between 0 to 100.
--experiment-metric-configurations: Required. A list of accepted test metrics you're using in your A/B test. You must designate each metric with either a KEY or GUARDRAIL value. Each metric must also contain an expected metric direction of either INCREASE or DECREASE.

You can specify the JSON as a string or in a file. If you specify a file, you can specify it using either an absolute or relative path. For example, "$(cat {filepath})" or "file:{filepath}".
--profile, -p: The ASK CLI profile to use. When you don't include this option, ASK CLI uses the default profile.
--debug: When you include this option, debug messages appear in the output of the command.

delete-experiment

Deletes an existing A/B test.

Calls the Delete A/B test SMAPI API.

Syntax:

ask smapi delete-experiment [-s,--skill-id <skill-id>]
                            [--experiment-id <experiment-id>]    
                            [--profile | -p <profile>]
                            [--debug]

Options:

-s, --skill-id: Required. The skill ID associated with your A/B test.
--experiment-id: Required. The unique ID of the A/B test you want to delete.

update-exposure

Updates the exposure percentage value of an existing A/B test. You can only use this command on a test that's in the CREATED or RUNNING state.

Calls the Update exposure SMAPI API.

Syntax:

ask smapi update-exposure [-s,--skill-id <skill-id>]
                          [--experiment-id <experiment-id>]
                          [--exposure-percentage <exposure-percentage>]    
                          [--profile | -p <profile>]
                          [--debug]

Options:

-s, --skill-id: Required. The skill ID associated with your A/B test.
--experiment-id: Required. The unique ID of the A/B test you want to update.
--exposure-percentage: Required. The new exposure percentage you want to update your test to. You can only increase the exposure percentage value of a running test, not decrease it.

Accepted values: Any value between 0 and 100. This new value must be higher than your current exposure percentage.
--profile, -p: The ASK CLI profile to use. When you don't include this option, ASK CLI uses the default profile.
--debug: When you include this option, debug messages appear in the output of the command.

get-customer-treatment-override

Retrieves the current customer treatment override value of an existing A/B test.

Calls the Get Customer Treatment Override SMAPI API.

Note: You can only retrieve the customer treatment override value of a skill that you own. To satisfy this requirement, your Customer ID and Vendor ID must match the information configured in the skill. For example, you can retrieve the override value of a skill you developed or retrieve the override value of another skill that shares your Vendor ID.

Syntax:

ask smapi get-customer-treatment-override [-s,--skill-id <skill-id>]
                                          [--experiment-id <experiment-id>]
                                          [--profile | -p <profile>]
                                          [--debug]

Options:

-s, --skill-id: Required. The skill ID associated with your A/B test.
--experiment-id: Required. The unique ID of the A/B test you want to retrieve details for.
--profile, -p: The ASK CLI profile to use. When you don't include this option, ASK CLI uses the default profile.
--debug: When you include this option, debug messages appear in the output of the command.

set-customer-treatment-override

Sets a new customer treatment override value of an existing A/B test.

Calls the Set Customer Treatment Override SMAPI API.

Note: You can only set the customer treatment override value of a skill that you own. To satisfy this requirement, your Customer ID and Vendor ID must match the information configured in the skill. For example, you can set the override value of a skill you developed or set the override value of another skill that shares your Vendor ID.

Syntax:

ask smapi set-customer-treatment-override  [-s,--skill-id <skill-id>]
                                           [--experiment-id <experiment-id>]  
                                           [--treatment-id <treatment-id>]
                                           [--profile | -p <profile>]
                                           [--debug]

Options:

-s, --skill-id

Required. The skill ID associated with your A/B test.

--experiment-id

Required. The unique ID of the A/B test you want to update.

--treatment-id

Required. The test group that you want to set for your skill.
Accepted values:

C – Control. The skill behavior you aren't testing. This group of customers continue to receive your current skill experience.
T1 – Treatment. The skill behavior you're testing. This group of customers receive your new experimental skill experience.

--profile, -p

The ASK CLI profile to use. When you don't include this option, ASK CLI uses the default profile.

--debug

When you include this option, debug messages appear in the output of the command.

get-experiment-state

Retrieves the current state of an A/B test.

Calls the Get A/B test state SMAPI API.

Syntax:

ask smapi get-experiment-state  [-s,--skill-id <skill-id>]
                                [--experiment-id <experiment-id>]  
                                [--profile | -p <profile>]
                                [--debug]

Options:

-s, --skill-id: Required. The skill ID associated with your A/B test.
--experiment-id: Required. The unique ID of the A/B test you want to retrieve.
--profile, -p: The ASK CLI profile to use. When you don't include this option, ASK CLI uses the default profile.
--debug: When you include this option, debug messages appear in the output of the command.

manage-experiment-state

Updates the current state of an A/B test.

Calls the Update A/B test state SMAPI API.

Syntax:

ask smapi manage-experiment-state  [-s,--skill-id <skill-id>]
                                   [--experiment-id <experiment-id>]  
                                   [--target-state <target-state>]
                                   [--profile | -p <profile>]
                                   [--debug]

Options:

-s, --skill-id

Required. The skill ID associated with your A/B test.

--experiment-id

Required. The unique ID of the A/B test you want to update.

--target-state

Required. The new state you want to update your test to. For more details about how to use states, see Lifecycle of an experiment.
Accepted values:

ENABLED - Deploy your test. You can only transition to this state if your test is in the CREATED state.
RUNNING - Start your test. You can only transition to this state if your test is in the CREATED or ENABLED state.
STOPPED - Stop your test. You can only transition to this state if your test is in the ENABLED or RUNNING state.

--profile, -p

The ASK CLI profile to use. When you don't include this option, ASK CLI uses the default profile.

--debug

When you include this option, debug messages appear in the output of the command.

list-experiment-metric-snapshots

Lists all metric snapshots associated with an experimentId. A metric snapshot represents all the metric data that's available for your A/B test in a specific time range. To get detailed analytics about a test, use the get-experiment-metric-snapshot command.

Calls the Get all metric snapshots SMAPI API.

Syntax:

ask smapi list-experiment-metric-snapshots  [-s,--skill-id <skill-id>]
                                            [--experiment-id <experiment-id>]  
                                            [--next-token <next-token>]
                                            [--max-results <max-results>]
                                            [--profile | -p <profile>]
                                            [--debug]

Options:

-s, --skill-id: Required. The skill ID associated with your A/B test.
--experiment-id: Required. The unique ID of the A/B test you want to retrieve.
--next-token: Optional. Token used to retrieve the next page of results.
--max-results: Optional. Sets the maximum number of results returned in the response body. maxResults shouldn't exceed the upper limit. The response might contain fewer results than maxResults, but it never contains more.
--profile, -p: The ASK CLI profile to use. When you don't include this option, ASK CLI uses the default profile.
--debug: When you include this option, debug messages appear in the output of the command.

get-experiment-metric-snapshot

Lists detailed metric data and analytics associated with the metricSnapshotId of an A/B test.

Calls the Get metric snapshot data SMAPI API.

Syntax:

ask smapi get-experiment-metric-snapshot   [-s,--skill-id <skill-id>]
                                           [--experiment-id <experiment-id>]  
                                           [--metric-snapshot-id <metric-snapshot-id>]
                                           [--profile | -p <profile>]
                                           [--debug]

Options:

-s, --skill-id: Required. The skill ID associated with your A/B test.
--experiment-id: Required. The unique ID of the A/B test details you want to retrieve.
--metric-snapshot-id: Required. The unique ID of the test metric snapshot details you want to retrieve.
--profile, -p: The ASK CLI profile to use. When you don't include this option, ASK CLI uses the default profile.
--debug: When you include this option, debug messages appear in the output of the command.

Was this page helpful?

Provide feedback

Last updated: Oct 13, 2023

A/B Testing CLI Commands

create-experiment

list-experiments

get-experiment

update-experiment

delete-experiment

update-exposure

get-customer-treatment-override

set-customer-treatment-override

get-experiment-state

manage-experiment-state

list-experiment-metric-snapshots

get-experiment-metric-snapshot

Related topics

Was this page helpful?