A/B Testing SMAPI APIs

Note: Sign in to the developer console to build or publish your skill.

Use the following SMAPI APIs to create, update, and manage A/B tests for your skills.

Authorization

Each API header request must contain an access token retrieved from Login with Amazon. For more details about obtaining an access token, see Get an Access Token for SMAPI.

Create A/B test

Creates a new A/B test for a skill.

You can also call this API using the create-experiment ASK CLI command.

Request

POST /v1/skills/{skillId}/experiments/
Host: api.amazonalexa.com
Accept: application/json
Authorization: Bearer {LWA Token}

Request fields

Field	Description	Type	Required?
`skillId`	The skill ID associated with your A/B test.	Path	Yes

Request body structure

The following example illustrates the structure of the request body.

{
  "experiment": {
    "name": "my_first_experiment",
    "description": "experiment_1",
    "type": "ENDPOINT_BASED",
    "plannedDuration": "P4W",
    "exposurePercentage": 20,
    "metricConfigurations": [
      {
        "name": "Customer Friction",
        "metricTypes": [
          "GUARDRAIL"
        ],
        "expectedChange": "INCREASE"
      }
    ]
  }
}

Request body fields

Field	Description	Type	Required?
`experiment`	An object that contains A/B test settings for your skill.	Object	Yes
`experiment.name`	A unique name for your A/B test. You can't use an identical name for two tests. However, you can use the same name if you delete one of the tests. Maximum 255 characters.	String	Yes
`experiment.description`	A description of the hypothesis you set for your A/B test. Only used for reference purposes and doesn't impact the outcome.	String	Yes
`experiment.type`	The type of A/B test you want to run for your skill. Accepted values: `ENDPOINT_BASED`- Sets your test's T1 value to your skill version in the `LIVE` stage. To separate your C and T1 behaviors, you must add conditional logic in your skill code. For more details about separating endpoint-based tests, see Set up an Endpoint-based A/B Test.	String	Yes
`experiment.plannedDuration`	The number of weeks you're running your A/B test. Only used for reference purposes and doesn't impact the actual test length. Accepted values: Any value between 1 and 4 weeks represented in the ISO-8601 format. For example, you represent a value of four weeks with the ISO-8601 value of `P4W`.	String	Yes
`experiment.exposurePercentage`	The percentage of unique customers that you're including in your A/B test. This number doesn't have to match your total users. For example, if you have 100 total customers and you set your `exposurePercentage` to 40 percent, you're including 40 customers in your test. In this case, your test includes 20 customers in C and 20 customers in T1. The remaining 60 customers aren't included in the test and receive the default behavior equivalent to C, meaning they don't contribute to the test metrics. Accepted values: Any value between `0` to `100`.	Integer	Yes
`experiment.metricConfigurations`	A list of accepted A/B test metrics that you're testing in your experiment.	Array	Yes
`experiment.metricConfigurations.name`	The unique name of an A/B test metric. Maximum 125 characters.	String	Yes
`experiment.metricConfigurations.metricTypes`	A list of values used to detect changes in your A/B test. Accepted values: • `KEY` - A metric configuration that provides clear evidence of expected changes caused by your new treatment experience. • `GUARDRAIL` - A metric configuration that detects unexpected regressions caused by your new treatment experience.	Array	Yes
`experiment.expectedChange`	The expected outcome of each A/B test metric. Only used for reference purposes and doesn't impact the actual test outcome. Accepted values: • `INCREASE` - An upwards change in one of your defined metrics configurations. • `DECREASE` - A downwards change in one of your defined metrics configurations.	String	Yes

Response

HTTP/1.1 201 Experiment created. Returns the generated experiment identifier in 'Location' header.
Content-Type: application/json
Location: {url}
X-Amzn-RequestId: {request-id}

Response header fields

Field	Description	Type
`Location`	The relative URL used to retrieve your A/B test details.	String
`X-Amzn-RequestId`	The unique identifier for the request. If a problem occurs, Amazon can use this value to troubleshoot the problem.	String

HTTP response codes

Status code	Description
201	Experiment created. Returns the generated experiment identifier in 'Location' header.
400	Server can't process the request due to a client error.
401	The auth token is invalid/expired or doesn't have access to the resource.
403	The operation being requested isn't allowed.
404	The requested resource isn't found.
429	Exceeds the permitted request limit. Throttling criteria includes total requests, per API, ClientId, and CustomerId.
500	Internal Server Error.
503	Service Unavailable.

Get A/B test

Retrieves detailed information about an A/B test.

You can also call this API using the get-experiment ASK CLI command.

Request

GET /v1/skills/{skillId}/experiments/{experimentId}
Host: api.amazonalexa.com
Accept: application/json
Authorization: Bearer {LWA Token}

Request fields

Field	Description	Type	Required?
`skillId`	The skill ID associated with your A/B test.	Path	Yes
`experimentId`	The unique ID of the A/B test you want to retrieve.	Path	Yes

Response

HTTP/1.1 200 Returned skill experiment
Host: api.amazonalexa.com
X-Amzn-RequestId: {request-id}
Content-Type: application/json

Response header fields

Field	Description	Type
`X-Amzn-RequestId`	The unique identifier for the request. If a problem occurs, Amazon can use this value to troubleshoot the problem.	String

Response body fields

The following example illustrates the structure of the response body.

{
  "experiment": {
    "name": "my_first_experiment",
    "description": "experiment_1",
    "type": "ENDPOINT_BASED",
    "plannedDuration": "P3W",
    "exposurePercentage": 0,
    "metricConfigurations": [
      {
        "name": "Customer Friction",
        "metricTypes": [
          "GUARDRAIL"
        ],
        "expectedChange": "INCREASE"
      }
    ],
    "state": "CREATED",
    "history": {
      "creationTime": "2021-04-19T18:43:48.383Z",
      "startTime": "2021-04-19T18:43:48.383Z",
      "stoppedReason": "string"
    },
    "trigger": {
      "scheduledEndTime": "2021-04-19T18:43:48.383Z"
    }
  }
}

Response body fields

Field	Description	Type
`experiment`	An object that contains A/B test settings for your skill.	Object
`experiment.name`	The unique name of your A/B test.	String
`experiment.description`	The description of the hypothesis you're trying to prove with your A/B test. Only used for reference purposes and doesn't impact the outcome.	String
`experiment.type`	The type of A/B test you set for your skill. Possible values: `ENDPOINT_BASED`- Sets your test's T1 value to your skill version in the `LIVE` stage. To separate your C and T1 behaviors, you must add conditional logic in your skill code. For more details about separating endpoint-based tests, see Set up an Endpoint-based A/B Test.	String
`experiment.plannedDuration`	The number of weeks you're running the A/B test. Only used for reference purposes and doesn't impact the actual test length. Accepted values: Any value between 1 and 4 weeks represented in the ISO-8601 format. For example, you represent a value of four weeks with the ISO-8601 value of `P4W`.	String
`experiment.exposurePercentage`	The percentage of unique customers included in your A/B test.	Integer
`experiment.metricConfigurations`	A list of accepted A/B test metrics you selected for your test.	Array
`experiment.metricConfigurations.name`	The unique name of an A/B test metric.	String
`experiment.metricConfigurations.metricTypes`	A list of values used to detect changes in your A/B test. Possible values: • `KEY` - A test metric that provides clear evidence of expected changes caused by your new treatment experience. • `GUARDRAIL` - A test metric that detects unexpected regressions caused by your new treatment experience.	Array
`experiment.expectedChange`	The expected outcome of each A/B test metric. Only used for reference purposes and doesn't impact the actual test outcome. Possible values: • `INCREASE` - An upwards change in one of your test metrics. • `DECREASE` - A downwards change in one of your test metrics.	String
`experiment.history.creationTime`	The creation date of your A/B test.	String
`experiment.history.startTime`	The start time of your A/B test.	String
`experiment.history.stoppedReason`	The reason you stopped your A/B test.	String
`experiment.trigger`	An object that contains A/B test triggers for your skill.	Object
`experiment.trigger.scheduledEndTime`	The expected end time of your A/B test.	String

HTTP response codes

Status code	Description
200	Returned skill test.
401	The auth token is invalid/expired or doesn't have access to the resource.
403	The requested operation isn't allowed.
404	The requested resource isn't found.
429	Exceeds the permitted request limit. Throttling criteria includes total requests, per API, ClientId, and CustomerId.
500	Internal Server Error.
503	Service Unavailable.

Get list of A/B tests

Retrieves a list of tests associated with a skill ID. Returns both completed and in-progress tests.

You can also call this API using the list-experiment ASK CLI command.

Request

GET /v1/skills/{skillId}/experiments
Host: api.amazonalexa.com
Accept: application/json
Authorization: Bearer {LWA Token}

Request header fields

Field	Description	Type	Required?
`skillId`	The skill ID associated with your test.	Path	Yes
`NextToken`	A token used to retrieve the next page of results.	Path	No
`MaxResults`	Sets the maximum number of results returned in the response body. `maxResults` shouldn't exceed the upper limit. The response might contain fewer results than `maxResults`, but it never contains more.	Path	No

Response

HTTP/1.1 200 Returned skill experiments.
Host: api.amazonalexa.com
X-Amzn-RequestId: {request-id}
Content-Type: application/json

Response header fields

Field	Description	Type
`X-Amzn-RequestId`	The unique identifier for the request. If a problem occurs, Amazon can use this value to troubleshoot the problem.	String

Response body structure

The following example illustrates the structure of the response body.

{
  "experiments": [
    {
      "experimentId": <experiment-id> ,
      "name": "my_first-experiment",
      "state": "CREATED",
      "experimentHistory": {
        "creationTime": "2021-04-29T18:20:50.671Z",
        "startTime": "2021-04-29T18:20:50.671Z",
        "stoppedReason": "string"
      }
    }
  ]
}

Response body fields

Field	Description	Type
`experiments`	An object that contains A/B test settings for your skill.	Array
`experiments.experimentId`	The unique identifier of the A/B test associated with your skill.	String
`experiments.name`	The unique name of your A/B test.	String
`experiments.state`	The current state of your A/B test. For more details about how to use states, see Lifecycle of an A/B Test. Possible values: • `CREATED` - Your test is created but isn't enabled or started. • `ENABLING` - Your test is transitioning from `CREATED` to `ENABLED`. • `ENABLED` - Your test configurations are deployed. • `RUNNING` - Your test has started. • `STOPPING` - Your test is transitioning from `RUNNING` to `STOPPED`. • `STOPPED` - Your test has ended. • `DELETED` - Your test is deleted. • `FAILED` - Your test configurations failed to deploy when trying to enable or start the test.	String
`experiments.experimentHistory`	An object that contains historical properties of your A/B test.	Object
`experiments.experimentHistory.creationTime`	The creation date of your A/B test.	String
`experiments.experimentHistory.startTime`	The start time of your A/B test.	String
`experiments.experimentHistory.stoppedReason`	The reason you stopped your A/B test.	String

HTTP response codes

Status code	Description
200	Returned a list of skill tests.
401	The auth token is invalid/expired or doesn't have access to the resource.
403	The requested operation isn't allowed.
404	The requested resource isn't found.
429	Exceeds the permitted request limit. Throttling criteria includes total requests, per API, ClientId, and CustomerId.
500	Internal Server Error.
503	Service Unavailable.

Update A/B test

Update the settings of an A/B test. You can only call this API if your test is in the CREATED state, meaning you can't update an active test. To update the exposure value of an active test, use Update exposure.

You can also call this API using the update-experiment ASK CLI command.

Request

POST /v1/skills/{skillId}/experiments/{experimentId}/properties
Host: api.amazonalexa.com
Accept: application/json
Authorization: Bearer {LWA Token}

Request header fields

Field	Description	Type	Required?
`skillId`	The skill ID associated with your A/B test.	Path	Yes
`experimentId`	The unique ID of the A/B test you want to update.	Path	Yes

Request body structure

The following example illustrates the structure of the response body.

{
  "experiment": {
    "description": "experiment_1",
    "plannedDuration": "P3W",
    "exposurePercentage": 0,
    "metricConfigurations": [
      {
        "name": "Customer Friction",  
        "metricTypes": [
          "GUARDRAIL"
        ],
        "expectedChange": "INCREASE"
      }
    ]
  }
}

Request body fields

Field	Description	Type
`experiment`	An object that contains A/B test settings for your skill.	Object
`experiment.description`	A description of the hypothesis you set for your A/B test. Only used for reference purposes and doesn't impact the outcome.	String
`experiment.plannedDuration`	The number of weeks you're running the A/B test. Only used for reference purposes and doesn't impact the actual test length. Accepted values: Any value between 1 and 4 weeks represented in the ISO-8601 format. For example, you represent a value of four weeks with the ISO-8601 value of `P4W`.	String
`experiment.exposurePercentage`	The percentage of unique customers that you're including in the A/B test. This number doesn't have to match your total users. For example, if you have 100 total customers and you set your `exposurePercentage` to 40 percent, you're including 40 customers in your test. In this case, your test includes 20 customers in C and 20 customers in T1. The remaining 60 customers aren't included in the test and receive the default behavior equivalent to C – they don't contribute to the test metrics. Accepted values: Any value between `0` to `100`.	Integer
`experiment.metricConfigurations`	A list of accepted A/B test metrics you selected for your test.	Array
`experiment.metricConfigurations.name`	The unique name of a test metric.	String
`experiment.metricConfigurations.metricTypes`	A list of values used to detect changes in your A/B test. Possible values: • `KEY` - A test metric that provides clear evidence of expected changes caused by your new treatment experience. • `GUARDRAIL` - A test metric that detects unexpected regressions caused by your new treatment experience.	Array
`experiment.expectedChange`	The expected outcome of each A/B test metric. Only used for reference purposes and doesn't impact the actual test outcome. Accepted values: • `INCREASE` - An upwards change in one of your test metrics. • `DECREASE` - A downwards change in one of your test metrics.	String

Response

HTTP/1.1 204 Success. No content.
Host: api.amazonalexa.com
X-Amzn-RequestId: {request-id}
Content-Type: application/json

Response header fields

Field	Description	Type
`X-Amzn-RequestId`	The unique identifier for the request. If a problem occurs, Amazon can use this value to troubleshoot the problem.	String

Response body example

The response returns your updated test values.

{
  "experiment": {
    "name": "Customer Friction",  
    "description": "experiment_1",
    "type": "ENDPOINT_BASED",
    "plannedDuration": "P3W",
    "exposurePercentage": 0,
    "metricConfigurations": [
      {
        "name": "Customer Friction",  
        "metricTypes": [
          "GUARDRAIL"
        ],
        "expectedChange": "INCREASE"
      }
    ]
  }
}

HTTP response codes

Status code	Description
204	Success. No content.
400	Server can't process the request due to a client error.
401	The auth token is invalid/expired or doesn't have access to the resource.
403	The requested operation isn't allowed.
404	The requested resource isn't found.
409	The request couldn't be completed due to a conflict with the current state of the target resource.
429	Exceeds the permitted request limit. Throttling criteria includes total requests, per API, ClientId, and CustomerId.
500	Internal Server Error.
503	Service Unavailable.

Delete A/B test

Delete an A/B test.

You can only call this API if your test is in CREATED state. If you need to delete your test after you start it (meaning your test is in the RUNNING state), you must completely delete your skill with the Delete skill manifest REST API.

You can also call this API using the delete-experiment ASK CLI command.

Request

DELETE /v1/skills/{skillId}/experiments/{experimentId}
Host: api.amazonalexa.com
Accept: application/json
Authorization: Bearer {LWA Token}

Request fields

Field	Description	Type	Required?
`skillId`	The skill ID associated with your test.	Path	Yes
`experimentId`	The unique ID of the A/B test you want to delete.	Path	Yes

Response header

HTTP/1.1 204 Success. No content.
Host: api.amazonalexa.com
X-Amzn-RequestId: {request-id}
Content-Type: application/json

Field	Description	Type
`X-Amzn-RequestId`	The unique identifier for the request. If a problem occurs, Amazon can use this value to troubleshoot the problem.	String

HTTP response codes

Status code	Description
204	Success. No content.
401	The auth token is invalid/expired or doesn't have access to the resource.
403	The requested operation isn't allowed.
404	The requested resource isn't found.
409	The request couldn't be completed due to a conflict with the current state of the target resource.
429	Exceeds the permitted request limit. Throttling criteria includes total requests, per API, ClientId, and CustomerId.
500	Internal Server Error.
503	Service Unavailable.

Update exposure

Updates the exposure value of an A/B test. You can only call this API if your test is in the CREATED or RUNNING state.

RUNNING – You can only increase your exposure value. To decrease your exposure value after you have started your test, stop your test and create a new one.
CREATED – Increase or decrease your exposure value. If you want to update other settings, use Update A/B test

You can also call this API using the update-exposure ASK CLI command.

Request

POST /v1/skills/{skillId}/experiments/{experimentId}/exposurePercentage
Host: api.amazonalexa.com
Accept: application/json
Authorization: Bearer {LWA Token}

Request header fields

Field	Description	Type	Required?
`skillId`	The skill ID associated with your test.	Path	Yes
`experimentId`	The unique ID of the A/B test you want to update.	Path	Yes

Request body fields

Field	Description	Type	Required?
`exposurePercentage`	The new exposure percentage you want to update your A/B test to. You can only increase the exposure percentage value of a running test, not decrease it. Accepted values: Any value between `0` and `100` that's higher than your current exposure percentage.	Integer	Yes

Response

HTTP/1.1 204 Success. No content.
Host: api.amazonalexa.com
X-Amzn-RequestId: {request-id}
Content-Type: application/json

Response header fields

Field	Description	Type
`X-Amzn-RequestId`	The unique identifier for the request. If a problem occurs, Amazon can use this value to troubleshoot the problem.	String

HTTP response codes

Status code	Description
204	Success. No content.
400	Server can't process the request due to a client error.
401	The auth token is invalid/expired or doesn't have access to the resource.
403	The requested operation isn't allowed.
404	The requested resource isn't found.
409	Conflict with the resource state.
429	Exceeds the permitted request limit. Throttling criteria includes total requests, per API, ClientId, and CustomerId.
500	Internal Server Error.
503	Service Unavailable.

Get customer treatment override

Retrieves the current customer treatment override value of an A/B test.

You can also call this API using the get-customer-treatment-override ASK CLI command.

Note: You can only retrieve the customer treatment override value of a skill that you own. To satisfy this requirement, your Customer ID and Vendor ID must match the information configured in the skill. For example, you can retrieve the override value of a skill you developed or retrieve the override value of another skill that shares your Vendor ID.

Request

GET /v1/skills/{skillId}/experiments/{experimentId}/treatmentOverrides/~current
Host: api.amazonalexa.com
Accept: application/json
Authorization: Bearer {LWA Token}

Request fields

Field	Description	Type	Required?
`skillId`	The skill ID associated with your A/B test.	Path	Yes
`experimentId`	The unique ID of the A/B test you want to update.	Path	Yes

Response header fields

HTTP/1.1 200. Returned customer treatment override details.
Host: api.amazonalexa.com
X-Amzn-RequestId: {request-id}
Content-Type: application/json

Field	Description	Type
`X-Amzn-RequestId`	The unique identifier for the request. If a problem occurs, Amazon can use this value to troubleshoot the problem.	String

Response body fields

Field Description Type

Field	Description	Type
`treatmentId`	The A/B testing group that you set your skill to. Possible values: C – Control. The skill behavior you aren't testing. This group of customers continue to receive your current skill experience. T1 – Treatment. The skill behavior you're testing. This group of customers receive your new experimental skill experience.	String
`treatmentOverrideCount`	The current number of treatment overrides you set for your A/B test.	Integer

treatmentId

The A/B testing group that you set your skill to.

Possible values:

C – Control. The skill behavior you aren't testing. This group of customers continue to receive your current skill experience.
T1 – Treatment. The skill behavior you're testing. This group of customers receive your new experimental skill experience.

String

treatmentOverrideCount

The current number of treatment overrides you set for your A/B test.

Integer

HTTP response codes

Status code	Description
200	Returned customer treatment override details.
401	Server can't process the request due to a client error.
401	The auth token is invalid/expired or doesn't have access to the resource.
403	The requested operation isn't allowed.
404	The requested resource isn't found.
429	Exceeds the permitted request limit. Throttling criteria includes total requests, per API, ClientId, and CustomerId.
500	Internal Server Error.
503	Service Unavailable.

Set customer treatment override

Sets a new customer treatment override value of an A/B test.

You can also call this API using the set-customer-treatment-override ASK CLI command.

Note: You can only set the customer treatment override value of a skill that you own. To satisfy this requirement, your Customer ID and Vendor ID must match the information configured in the skill. For example, you can set the override value of a skill you developed or set the override value of another skill that shares your Vendor ID.

Request

PUT /v1/skills/{skillId}/experiments/{experimentId}/treatmentOverrides/~current
Host: api.amazonalexa.com
Accept: application/json
Authorization: Bearer {LWA Token}

Request fields

Field	Description	Type	Required?
`skillId`	The skill ID associated with your A/B test.	Path	Yes
`experimentId`	The unique ID of the A/B test you want to update.	Path	Yes

Request body parameters

Field Description Type Required?

Field	Description	Type	Required?
`treatmentId`	The A/B testing group that you want to set for your skill. Accepted values: C – Control. The skill behavior you aren't testing. This group of customers continue to receive your current skill experience. T1 – Treatment. The skill behavior you're testing. This group of customers receive your new experimental skill experience.	String	Yes

treatmentId

The A/B testing group that you want to set for your skill.

Accepted values:

C – Control. The skill behavior you aren't testing. This group of customers continue to receive your current skill experience.
T1 – Treatment. The skill behavior you're testing. This group of customers receive your new experimental skill experience.

String

Yes

Response header fields

HTTP/1.1 204. Success. No content.
Host: api.amazonalexa.com
X-Amzn-RequestId: {request-id}
Content-Type: application/json

Field	Description	Type
`X-Amzn-RequestId`	The unique identifier for the request. If a problem occurs, Amazon can use this value to troubleshoot the problem.	String

Response body fields

Field Description Type

Field	Description	Type
`treatmentId`	The A/B testing group that you set your skill to. Possible values: C – Control. The skill behavior you aren't testing. This group of customers continue to receive your current skill experience. T1 – Treatment. The skill behavior you're testing. This group of customers receive your new experimental skill experience.	String

treatmentId

The A/B testing group that you set your skill to.

Possible values:

C – Control. The skill behavior you aren't testing. This group of customers continue to receive your current skill experience.
T1 – Treatment. The skill behavior you're testing. This group of customers receive your new experimental skill experience.

String

HTTP response codes

Status code	Description
204	Success. No content.
400	Server can't process the request due to a client error.
401	The auth token is invalid/expired or doesn't have access to the resource.
403	The requested operation isn't allowed.
404	The requested resource isn't found.
409	Conflict with the resource state.
429	Exceeds the permitted request limit. Throttling criteria includes total requests, per API, ClientId, and CustomerId.
500	Internal Server Error.
503	Service Unavailable.

Get A/B test state

Retrieves the current state of an A/B test.

You can also call this API using the get-experiment-state ASK CLI command.

Request

GET /v1/skills/{skillId}/experiments/{experimentId}/state
Host: api.amazonalexa.com
Accept: application/json
Authorization: Bearer {LWA Token}

Request fields

Field	Description	Type	Required?
`skillId`	The skill ID associated with your A/B test.	Path	Yes
`experimentId`	The unique ID of the A/B test you want to retrieve.	Path	Yes

Response

HTTP/1.1 200. Returned skill experiment state.
Host: api.amazonalexa.com
X-Amzn-RequestId: {request-id}
Content-Type: application/json

Response header fields

Field	Description	Type
`X-Amzn-RequestId`	The unique identifier for the request. If a problem occurs, Amazon can use this value to troubleshoot the problem.	String

Response body

The following example illustrates the structure of the response body.

{
  "state": "CREATED"
}

Response body fields

Field	Description	Type
`state`	The current state of your A/B test. For more details about how to use states, see Lifecycle of an A/B test. Possible values: • `CREATED` - Your test is created but isn't enabled or started. • `ENABLING` - Your test is transitioning from `CREATED` to `ENABLED`. • `ENABLED` - Your test configurations are deployed. • `RUNNING` - Your test has started. • `STOPPING` - Your test is transitioning from `RUNNING` to `STOPPED`. • `STOPPED` - Your test has ended. • `DELETED` - Your test is deleted. • `FAILED` - Your test configurations failed to deploy when trying to enable or start the test.	String

HTTP response codes

Status code	Description
200	Returned skill test state.
401	The auth token is invalid/expired or doesn't have access to the resource.
403	The requested operation isn't allowed.
404	The requested resource isn't found.
429	Exceeds the permitted request limit. Throttling criteria includes total requests, per API, ClientId, and CustomerId.
500	Internal Server Error.
503	Service Unavailable.

Update A/B test state

Manage and update the state of an A/B test.

You can also call this API using the manage-experiment-state ASK CLI command.

Request

POST /v1/skills/{skillId}/experiments/{experimentId}/state
Host: api.amazonalexa.com
Accept: application/json
Authorization: Bearer {LWA Token}

Request header fields

Field	Description	Type	Required?
`skillId`	The skill ID associated with your test.	Path	Yes
`experimentId`	The unique ID of the A/B test you want to update.	Path	Yes

Request body fields

Field	Description	Type	Required?
`targetState`	The new state you want to update your A/B test to. For more details about how to use states, see Lifecycle of an A/B test. Accepted values: • `ENABLED` - Deploy your test. You can only transition to this state if your test is in the `CREATED` state. • `RUNNING` - Start your test. You can only transition to this state if your test is in the `CREATED` or `ENABLED` state. • `STOPPED` - Stop your test. You can only transition to this state if your test is in the `ENABLED` or `RUNNING` state.	String	Yes

Response

HTTP/1.1 202 Accepted; Returns a URL to track the experiment state in 'Location' header.
Host: api.amazonalexa.com
X-Amzn-RequestId: {request-id}
Content-Type: application/json

Response header fields

Field	Description	Type
`Location`	The relative URL to get your A/B test details.	String
`X-Amzn-RequestId`	The unique identifier for the request. If a problem occurs, Amazon can use this value to troubleshoot the problem.	String

HTTP response codes

Status code	Description
202	Accepted; Returns a URL to track the experiment state in 'Location' header.
400	Server can't process the request due to a client error.
401	The auth token is invalid/expired or doesn't have access to the resource.
403	The requested operation isn't allowed.
404	The requested resource isn't found.
409	Conflict with the resource state.
429	Exceeds the permitted request limit. Throttling criteria includes total requests, per API, ClientId, and CustomerId.
500	Internal Server Error.
503	Service Unavailable.

Get all metric snapshots

Gets a list of all metric snapshots associated with the experimentId of an A/B test. A metric snapshot represents all the metric data that's available for your test in a specific time range. This API doesn't include detailed data about each metric snapshot. To get detailed analytics about a test, use the Get metric snapshot data.

You can also call this API using the list-experiment-metric-snapshots ASK CLI command.

Request

/v1/skills/{skillId}/experiments/{experimentId}/metricSnapshots
Host: api.amazonalexa.com
Accept: application/json
Authorization: Bearer {LWA Token}

Request fields

Field	Description	Type	Required?
`skillId`	The skill ID associated with your A/B test.	Path	Yes
`experimentId`	The unique ID of the A/B test details you want to retrieve.	Path	Yes
`NextToken`	A token used to retrieve the next page of results.	Path	No
`MaxResults`	Sets the maximum number of results returned in the response body. `maxResults` shouldn't exceed the upper limit. The response might contain fewer results than `maxResults`, but it never contains more.	Path	No

Response

HTTP/1.1 200 Returned test metric snapshots.
Host: api.amazonalexa.com
X-Amzn-RequestId: {request-id}
Content-Type: application/json

Response header fields

Field	Description	Type
`X-Amzn-RequestId`	The unique identifier for the request. If a problem occurs, Amazon can use this value to troubleshoot the problem.	String

Response body

The following example illustrates the structure of the response body.

  "metricSnapshots": [
    {
      "metricSnapshotId": "string",
      "startDate": "2021-04-20T15:24:31.563Z",
      "endDate": "2021-04-20T15:24:31.563Z"
    }
  ]
}

Response body fields

Field	Description	Type
`metricSnapshots`	Object that contains metric snapshot values for your skill.	Array
`metricSnapshots.metricSnapshotId`	Unique identifier for the metric snapshot of an A/B test.	String
`metricSnapshots.startDate`	Start date of the metric snapshot.	String
`metricSnapshots.endDate`	End date of the metric snapshot.	String

HTTP response codes

Status code	Description
200	Returned test metric snapshots.
400	Server can't process the request due to a client error.
401	The auth token is invalid/expired or doesn't have access to the resource.
403	The requested operation isn't allowed.
404	The requested resource isn't found.
429	Exceeds the permitted request limit. Throttling criteria includes total requests, per API, ClientId, and CustomerId.
500	Internal Server Error.
503	Service Unavailable.

Get metric snapshot data

Lists detailed metric data and analytics associated with a single metricSnapshotId of an A/B test.

You can also call this API using the get-experiment-metric-snapshot ASK CLI command.

Request

/v1/skills/{skillId}/experiments/{experimentId}/metricSnapshots/{metricSnapshotId}
Host: api.amazonalexa.com
Accept: application/json
Authorization: Bearer {LWA Token}

Request fields

Field	Description	Type	Required?
`skillId`	Skill ID associated with your A/B test.	Path	Yes
`experimentId`	Unique ID of the A/B test details you want to retrieve.	Path	Yes
`metricSnapshotId`	Unique ID of the A/B test metric snapshot you want to retrieve.	Path	Yes

Response

HTTP/1.1 200 Returned test metric data.
Host: api.amazonalexa.com
X-Amzn-RequestId: {request-id}
Content-Type: application/json

Response header fields

Field	Description	Type
`X-Amzn-RequestId`	The unique identifier for the request. If a problem occurs, Amazon can use this value to troubleshoot the problem.	String

Response body

The following example illustrates the structure of the response body.

{
  "status": "RELIABLE",
  "statusReason": "["TAA Alarm","Message"]",
  "metrics": [
    {
      "name": "Customer Friction",  
      "treatmentId": "C",
      "values": {
        "mean": 0,
        "percentDiff": 0,
        "confidenceIntervalLower": 0,
        "confidenceIntervalUpper": 0,
        "pValue": 0,
        "userCount": 0
      }
    }
  ]
}

Response body fields

Field	Description	Type
`status`	The reliability of the your A/B testing data. For example, if an error occurred during your tests, your data might not be accurate. Possible values: RELIABLE – The metric result snapshot for your selected date range is accurate. UNRELIABLE* – The metric result snapshot for your selected date range might not be accurate.	String
`statusReason`	Explains why your metric result snapshot might be unreliable.	String
`metrics`	A list of metric snapshot values for your skill.	Array
`metrics.name`	A unique name that identifies the A/B test metric.	String
`metrics.treatmentId`	The A/B test type assigned to a group of customers associated with these metrics. Possible values: C - Control. The skill behavior you aren't testing. This group of customers continue to receive your current skill experience. T1 - Treatment. The skill behavior you're testing. This group of customers receive your new experimental skill experience.	String
`metrics.values`	An object that contains values for your skill.	Object
`metrics.values.mean`	The average of each sample. Includes either the T1 or C group.	Double
`metrics.values.percentDiff`	The relative percent difference between the mean of the T1 group and the mean of the C group.	Double
`metrics.values.confidenceIntervalLower`	The lower limit number of the confidence interval range. The confidence interval measures the probability that the mean falls within a range.	Double
`metrics.values.confidenceIntervalUpper`	The upper limit number of the confidence interval range. The confidence interval measures the probability that the mean falls within a range.	Double
`metrics.values.pValue`	The probability that the difference between the two means (from T1 and C) is due to random sampling errors.	Double
`metrics.values.userCount`	The number of users in the test version.	Long

HTTP response codes

Status code	Description
200	Returned test metric data.
400	Server can't process the request due to a client error.
401	The auth token is invalid/expired or doesn't have access to the resource.
403	The operation being requested isn't allowed.
404	The requested resource isn't found.
429	Exceeds the permitted request limit. Throttling criteria includes total requests, per API, ClientId, and CustomerId.
500	Internal Server Error.
503	Service Unavailable.

Was this page helpful?

Provide feedback

Last updated: Aug 19, 2021

A/B Testing SMAPI APIs

Authorization

Create A/B test

Request

Response

HTTP response codes

Get A/B test

Request

Response

HTTP response codes

Get list of A/B tests

Request

Response

HTTP response codes

Update A/B test

Request

Response

HTTP response codes

Delete A/B test

Request

Response header

HTTP response codes

Update exposure

Request

Response

HTTP response codes

Get customer treatment override

Request

HTTP response codes

Set customer treatment override

Request

HTTP response codes

Get A/B test state

Request

Response

HTTP response codes

Update A/B test state

Request

Response

HTTP response codes

Get all metric snapshots

Request

Response

HTTP response codes

Get metric snapshot data

Request

Response

HTTP response codes

Related topics

Was this page helpful?