A/B Testing REST API Reference


Use the A/B Testing REST API to create, update, and manage A/B tests for your custom skill.

To run A/B testing, you define existing and new skill experiences by adding conditional statements to the skill code of your live skill. These statements branch into the current skill experience, called the control version (C), or the new skill experience, called the treatment version (T1). You can use the A/B testing API to experiment with your new experience.

For more details, see About A/B Testing. You can also create and manage tests in the Alexa developer console.

API endpoint

The endpoint of the A/B Testing API is https://api.amazonalexa.com.

Authentication

Each API request must have an authorization header whose value is the access token retrieved from Login with Amazon (LWA). For details, see Get an Access Token for SMAPI.

Operations

The A/B Testing API includes the following operations.

Operation HTTP method and URI

Create A/B test

POST /v1/skills/{skillId}/experiments/

Delete A/B test

DELETE /v1/skills/{skillId}/experiments/{experimentId}

Get A/B test

GET /v1/skills/{skillId}/experiments/{experimentId}

Get A/B test state

GET /v1/skills/{skillId}/experiments/{experimentId}/state

Get customer treatment override

GET /v1/skills/{skillId}/experiments/{experimentId}/treatmentOverrides/~current

Get metric snapshot data

GET /v1/skills/{skillId}/experiments/{experimentId}/metricSnapshots/{metricSnapshotId}

List A/B test

GET /v1/skills/{skillId}/experiments&maxResults={maxResults}&nextToken={nextToken}

List metric snapshots

GET /v1/skills/{skillId}/experiments/{experimentId}/metricSnapshots&maxResults={maxResults}&nextToken={nextToken}

Set customer treatment override

PUT /v1/skills/{skillId}/experiments/{experimentId}/treatmentOverrides/~current

Update A/B test

POST /v1/skills/{skillId}/experiments/{experimentId}/properties

Update A/B test state

POST /v1/skills/{skillId}/experiments/{experimentId}/state

Update test exposure

POST /v1/skills/{skillId}/experiments/{experimentId}/exposurePercentage

Create A/B test

Create a new A/B test for the specified skill. Here you define the percentage of customers to send to the new skill experience and the resulting metrics to collect.

You can also create an A/B test by using the create-experiment ASK CLI command.

Request

To create a new test, you make a POST request to the /v1/skills/{skillId}/experiments resource.

Request path and header example

Copied to clipboard.

POST /v1/skills/{skillId}/experiments/
Host: api.amazonalexa.com
Content-Type: application/json
Authorization: Bearer {access token}

Request path and header parameters

Parameter Located in Description Type Required

skillId

Path

Identifies the skill associated with your A/B test.
Valid values: 1 – 255 characters.

String

Yes

access token

Header

LWA token.

String

Yes

Request body example

Copied to clipboard.

{
  "experiment": {
    "name": "my_first_experiment",
    "description": "experiment_1",
    "type": "ENDPOINT_BASED",
    "plannedDuration": "P4W",
    "exposurePercentage": 20,
    "metricConfigurations": [
      {
        "name": "Customer Friction",
        "metricTypes": [
          "GUARDRAIL"
        ],
        "expectedChange": "INCREASE"
      }
    ]
  }
}

Request body properties

Property Description Type Required

experiment

A/B test settings.

Object

Yes

experiment.name

Unique name for your A/B test.
You can't use an identical name for two tests. However, you can use the same name if you delete one of the tests.
Valid value: 1–255 characters.

String

Yes

experiment.description

Description of the hypothesis you set for your A/B test.
Used for reference purposes and doesn't impact the outcome.

String

Yes

experiment.type

Type of A/B test that you want to run for your skill.
Valid value:

  • ENDPOINT_BASED- Sets your test's T1 value to your skill version in the LIVE stage. To separate your control (C) and treatment (T1) behavior, you must add conditional logic in your skill code. For more details, see Set up an Endpoint-based A/B Test.

String

Yes

experiment.plannedDuration

Number of weeks you plan to run your A/B test.
Used for reference purposes and doesn't impact the test length.
Defined in ISO-8601 format, P[n]W.
Valid values:1–4 weeks.

String

Yes

experiment.exposurePercentage

Percentage of unique customers that you plan to include in your A/B test. This number doesn't have to match your total users.

For example, if you have 100 total customers and you set your exposurePercentage to 40 percent, you're including 40 customers in your test. In this case, your test includes 20 customers in C and 20 customers in T1. The remaining 60 customers aren't included in the test and receive the default behavior equivalent to C, meaning they don't contribute to the test metrics.
Valid values: 0–100.

Integer

Yes

experiment.metricConfigurations

List of A/B test metrics that you plan to use in your test.

Array

Yes

experiment.metricConfigurations[].name

Unique name of the test metric.
For valid names, see key and guardrail metrics.
Valid value: 1–125 characters.

String

Yes

experiment.metricConfigurations[].metricTypes

List of metrics used in your A/B test.
Valid values:
  •  KEY – Identifies a metric that provides clear evidence of expected changes caused by the new treatment experience.
  •  GUARDRAIL – Identifies a metric that detects unexpected regressions caused by the new treatment experience.

Array of strings

Yes

experiment.metricConfigurations[].expectedChange

Direction that you expect a metric to trend during the testing.
Used for reference purposes and doesn't impact the actual test outcome.
Valid values:
  •  INCREASE – Upward change in metric value.
  •  DECREASE – Downward change in metric value.

String

Yes

Response

A successful response returns HTTP 201 Created, along with the experiment identifier in the Location header. Use the URL in the Location header to track the test. On error, the response returns the appropriate HTTP status code and includes a response body with an error code and human readable message.

Response body example

The response has no body.

Response body properties

The response has no body.

HTTP status codes

Status Description

201 Created

A/B test created successfully for the specified skill.

400 Bad Request

Indicates that one or more properties in the request body aren't valid.
The following example shows the response body with the error code and message.

{
     "message": "The property is outside the allowed range.",
     "code": "INVALID_STRING_LENGTH"
}

401 Unauthorized

Request didn't include the authorization token, or the included token expired or isn't valid. Or, you don't have access to the resource.

403 Forbidden

Indicates that the authorization token is valid, but the requested operation isn't allowed.

404 Not Found

Requested resource not found.

429 Too Many Requests

Permitted rate limit, specified as number of requests per unit of time, exceeded. Retry the request by using exponential back-off.

500 Server Error

Error occurred on the server. Retry the request by using exponential back-off.

503 Service Unavailable

Server is down for maintenance, overloaded, or otherwise unavailable to handle the incoming request.

Delete A/B test

Delete the specified test.

Request

To remove an existing test, you make a DELETE request to the /v1/skills/{skillId}/experiments resource.

Request path and header example

Copied to clipboard.

DELETE /v1/skills/{skillId}/experiments/{experimentId}
Host: api.amazonalexa.com
Content-Type: application/json
Authorization: Bearer {access token}

Request path and header parameters

Parameter Located in Description Type Required

skillId

Path

Identifies the skill associated with your A/B test.
Valid values: 1 – 255 characters.

String

Yes

experimentId

Path

Identifies the A/B test.
Use the experiment ID returned from Create A/B test or List A/B tests.

String

Yes

access token

Header

LWA token.

String

Yes

Request body example

The request has no body.

Request body properties

The request has no body.

Response

A successful response returns HTTP 204 No Content. On error, the response returns the appropriate HTTP status code and includes a response body with an error code and human readable message.

Response body example

The response has no body.

Response body properties

The response has no body.

HTTP status codes

Status Description

204 No Content

A/B test deleted successfully.

400 Bad Request

Indicates that one or more properties in the request body aren't valid.
The following example shows the response body with the error code and message.

{
     "message": "The property is outside the allowed range.",
     "code": "INVALID_STRING_LENGTH"
}

401 Unauthorized

Request didn't include the authorization token, or the included token expired or isn't valid. Or, you don't have access to the resource.

403 Forbidden

Indicates that the authorization token is valid, but the requested operation isn't allowed.

404 Not Found

Requested resource not found.

429 Too Many Requests

Permitted rate limit, specified as number of requests per unit of time, exceeded. Retry the request by using exponential back-off.

500 Server Error

Error occurred on the server. Retry the request by using exponential back-off.

503 Service Unavailable

Server is down for maintenance, overloaded, or otherwise unavailable to handle the incoming request.

Get A/B test

Get the details an existing A/B test.

You can also retrieve an A/B test by using the get-experiment ASK CLI command.

Request

To get the test details, you make a GET request to the /v1/skills/{skillId}/experiments/{experimentId} resource.

Request path and header example

Copied to clipboard.

GET /v1/skills/{skillId}/experiments/{experimentId}
Host: api.amazonalexa.com
Content-Type: application/json
Authorization: Bearer {access token}
Accept: application/json

Request path and header parameters

Parameter Located in Description Type Required

skillId

Path

Identifies the skill associated with your A/B test.
Valid values: 1 – 255 characters.

String

Yes

experimentId

Path

Identifies the A/B test.
Use the experiment ID returned from Create A/B test or List A/B tests.

String

Yes

access token

Header

LWA token.

String

Yes

Request body example

The request has no body.

Request body properties

The request has no body.

Response

A successful response returns HTTP 200 OK, along with the details of the test. On error, the response returns the appropriate HTTP status code and includes a response body with an error code and human readable message.

Response body example

{
  "experiment": {
    "name": "my_first_experiment",
    "description": "experiment_1",
    "type": "ENDPOINT_BASED",
    "plannedDuration": "P3W",
    "exposurePercentage": 0,
    "metricConfigurations": [
      {
        "name": "Customer Friction",
        "metricTypes": [
          "GUARDRAIL"
        ],
        "expectedChange": "INCREASE"
      }
    ],
    "state": "CREATED",
    "history": {
      "creationTime": "2021-04-19T18:43:48.383Z",
      "startTime": "2021-04-19T18:43:48.383Z",
      "stoppedReason": "string"
    },
    "trigger": {
      "scheduledEndTime": "2021-04-19T18:43:48.383Z"
    }
  }
}

Response body properties

Property Description Type

experiment

A/B test settings.

Object

experiment.name

Unique name of the A/B test. Valid value: 1–255 characters.

String

experiment.description

Description of the hypothesis you set for your A/B test.
Used for reference purposes and doesn't impact the outcome.

String

experiment.type

Type of A/B test that you want to run for your skill.
Valid value:

  • ENDPOINT_BASED- Sets your test's T1 value to your skill version in the LIVE stage. To separate your C and T1 behaviors, you must add conditional logic in your skill code. For more details, see Set up an Endpoint-based A/B Test.

String

experiment.plannedDuration

Number of weeks you plan to run your A/B test.
Used for reference purposes and doesn't impact the test length.
Defined in ISO-8601 format, P[n]W.
Valid values:1–4 weeks.

String

experiment.exposurePercentage

Percentage of unique customers that you plan to include in your A/B test. This number doesn't have to match your total users.

For example, if you have 100 total customers and you set your exposurePercentage to 40 percent, you're including 40 customers in your test. In this case, your test includes 20 customers in C and 20 customers in T1. The remaining 60 customers aren't included in the test and receive the default behavior equivalent to C, meaning they don't contribute to the test metrics.
Valid values: 0–100.

Integer

experiment.metricConfigurations

List of A/B test metrics that you plan to use in your test.

Array

experiment.metricConfigurations[].name

Unique name of the test metric.
For valid names, see key and guardrail metrics.
Valid value: 1–125 characters.

String

experiment.metricConfigurations[].metricTypes

List of metrics used in your A/B test.
Valid values:
  •  KEY – Identifies a metric that provides clear evidence of expected changes caused by the new treatment experience.
  •  GUARDRAIL – Identifies a metric that detects unexpected regressions caused by the new treatment experience.

Array of strings

experiment.metricConfigurations[].expectedChange

Direction that you expect a metric to trend during the testing.
Used for reference purposes and doesn't impact the actual test outcome.
Valid values:
  •  INCREASE – Upward change in metric value.
  •  DECREASE – Downward change in metric value.

String

treatmentOverrideCount

Number of overrides that exist for the test.

Integer

experiment.state

Current state of the test.
Valid values:

  • CREATED – Created successfully but not yet enabled or started.
  • ENABLING – Initiated the transition to ENABLED.
  • ENABLED – Experiment configurations deployed but only customer treatment overrides set to T1 can view the T1 experience of a skill. In this state, no metrics collected.
  • RUNNING – Test started and a percentage of skill customers defined in the exposurePercentage enter into the test. Customers are randomly assigned the T1 or C experience. Metric collection begins.
  • STOPPING – Initiated the transition to STOPPED.
  • STOPPED – Test ended and all customers now experience the C behavior. Metrics collection stopped.
  • FAILED – Experiment configurations failed to deploy during the ENABLING state.

String

experiment.history

History of an A/B test.

Object

experiment.history.creationTime

Date and time the test was created or updated.
Defined in ISO 8601 format, YYYY-MM-DDThh:mm:ssZ.

String

experiment.history.startTime

Date and time the test started.
Defined in ISO 8601 format, YYYY-MM-DDThh:mm:ssZ.

String

experiment.history.stoppedReason

Reason the test stopped.
Valid values: DEVELOPER_REQUEST, EXPERIMENT_SUCCESS, EXPERIMENT_ISSUE.

String

experiment.trigger

Defines the triggers for the test.

Object

experiment.trigger.scheduledEndTime

Expected end date and time of the test.
Defined in ISO 8601 format, YYYY-MM-DDThh:mm:ssZ.

String

HTTP status codes

Status Description

200 OK

Response body contains the test details.

400 Bad Request

Indicates that one or more properties in the request body aren't valid.
The following example shows the response body with the error code and message.

{
     "message": "The property is outside the allowed range.",
     "code": "INVALID_STRING_LENGTH"
}

401 Unauthorized

Request didn't include the authorization token, or the included token expired or isn't valid. Or, you don't have access to the resource.

403 Forbidden

Indicates that the authorization token is valid, but the requested operation isn't allowed.

404 Not Found

Requested resource not found.

429 Too Many Requests

Permitted rate limit, specified as number of requests per unit of time, exceeded. Retry the request by using exponential back-off.

500 Server Error

Error occurred on the server. Retry the request by using exponential back-off.

503 Service Unavailable

Server is down for maintenance, overloaded, or otherwise unavailable to handle the incoming request.

Get A/B test state

Get the current state of the specified test.

You can also retrieve the test state by using the get-experiment-state ASK CLI command.

Request

To get the test state, you make a GET request to the /v1/skills/{skillId}/experiments/{experimentId}/state resource.

Request path and header example

Copied to clipboard.

GET /v1/skills/{skillId}/experiments/{experimentId}/state
Host: api.amazonalexa.com
Content-Type: application/json
Authorization: Bearer {access token}
Accept: application/json

Request path and header parameters

Parameter Located in Description Type Required

skillId

Path

Identifies the skill associated with your A/B test.
Valid values: 1 – 255 characters.

String

Yes

experimentId

Path

Identifies the A/B test.
Use the experiment ID returned from Create A/B test or List A/B tests.

String

Yes

access token

Header

LWA token.

String

Yes

Request body example

The request has no body.

Request body properties

The request has no body.

Response

A successful response returns HTTP 200 OK, along with the state of the specified test. On error, the response returns the appropriate HTTP status code and includes a response body with an error code and human readable message.

Response body example

{
  "state": "CREATED"
}

Response body properties

Property Description Type

state

Current state of the test.
Valid values:

  • CREATED – Created successfully but not yet enabled or started.
  • ENABLING – Initiated the transition to ENABLED.
  • ENABLED – Experiment configurations deployed but only customer treatment overrides set to T1 can view the T1 experience of a skill. In this state, no metrics collected.
  • RUNNING – Test started and a percentage of skill customers defined in the exposurePercentage enter into the test. Customers are randomly assigned the T1 or C experience. Metric collection begins.
  • STOPPING – Initiated the transition to STOPPED.
  • STOPPED – Test ended and all customers now experience the C behavior. Metrics collection stopped.
  • FAILED – Experiment configurations failed to deploy during the ENABLING state.

String

HTTP status codes

Status Description

200 OK

Response body contains the test state.

400 Bad Request

Indicates that one or more properties in the request body aren't valid.
The following example shows the response body with the error code and message.

{
     "message": "The property is outside the allowed range.",
     "code": "INVALID_STRING_LENGTH"
}

401 Unauthorized

Request didn't include the authorization token, or the included token expired or isn't valid. Or, you don't have access to the resource.

403 Forbidden

Indicates that the authorization token is valid, but the requested operation isn't allowed.

404 Not Found

Requested resource not found.

429 Too Many Requests

Permitted rate limit, specified as number of requests per unit of time, exceeded. Retry the request by using exponential back-off.

500 Server Error

Error occurred on the server. Retry the request by using exponential back-off.

503 Service Unavailable

Server is down for maintenance, overloaded, or otherwise unavailable to handle the incoming request.

Get customer treatment override

Get the current customer treatment override value of a specified A/B test. You use the override to test the new skill experience.

You can also retrieve the customer treatment override by using the get-customer-treatment-override ASK CLI command.

Request

To get the current customer treatment override, you make a GET request to the /v1/skills/{skillId}/experiments/{experimentId}/treatmentOverrides resource.

Request path and header example

Copied to clipboard.

GET /v1/skills/{skillId}/experiments/{experimentId}/treatmentOverrides/~current
Host: api.amazonalexa.com
Content-Type: application/json
Authorization: Bearer {access token}
Accept: application/json

Request path and header parameters

Parameter Located in Description Type Required

skillId

Path

Identifies the skill associated with your A/B test.
Valid values: 1 – 255 characters.

String

Yes

experimentId

Path

Identifies the A/B test.
Use the experiment ID returned from Create A/B test or List A/B tests.

String

Yes

access token

Header

LWA token.

String

Yes

Request body example

The request has no body.

Request body properties

The request has no body.

Response

A successful response returns HTTP 200 OK, along the current override count. On error, the response returns the appropriate HTTP status code and includes a response body with an error code and human readable message.

Response body example

{
    "treatmentId": "T1",
    "treatmentOverrideCount": 2
}

Response body properties

Property Description Type

treatmentId

Testing group: control or new treatment.
Valid values: C, T1.

String

treatmentOverrideCount

Number of treatment overrides you set for your A/B test.

Integer

HTTP status codes

Status Description

200 OK

Response body contains the treatment override.

400 Bad Request

Indicates that one or more properties in the request body aren't valid.
The following example shows the response body with the error code and message.

{
     "message": "The property is outside the allowed range.",
     "code": "INVALID_STRING_LENGTH"
}

401 Unauthorized

Request didn't include the authorization token, or the included token expired or isn't valid. Or, you don't have access to the resource.

403 Forbidden

Indicates that the authorization token is valid, but the requested operation isn't allowed.

404 Not Found

Requested resource not found.

429 Too Many Requests

Permitted rate limit, specified as number of requests per unit of time, exceeded. Retry the request by using exponential back-off.

500 Server Error

Error occurred on the server. Retry the request by using exponential back-off.

503 Service Unavailable

Server is down for maintenance, overloaded, or otherwise unavailable to handle the incoming request.

Get metric snapshot data

Get the metric data associated with the specified snapshot.

You can also retrieve the metric data by using the get-experiment-metric-snapshot ASK CLI command.

Request

To get metric data, you make a GET request to the /v1/skills/{skillId}/experiments/{experimentId}/metricSnapshots resource.

Request path and header example

Copied to clipboard.

GET /v1/skills/{skillId}/experiments/{experimentId}/metricSnapshots/{metricSnapshotId}
Host: api.amazonalexa.com
Content-Type: application/json
Authorization: Bearer {access token}
Accept: application/json

Request path and header parameters

Parameter Located in Description Type Required

skillId

Path

Identifies the skill associated with your A/B test.
Valid values: 1 – 255 characters.

String

Yes

experimentId

Path

Identifies the A/B test.
Use the experiment ID returned from Create A/B test or List A/B tests.

String

Yes

metricSnapshotId

Path

Identifies the metric snapshot.
Use the ID returned from List metric snapshots.
Valid value: 1–255 characters.

String

Yes

access token

Header

LWA token.

String

Yes

Request body example

The request has no body.

Request body properties

The request has no body.

Response

A successful response returns HTTP 200 OK, along with the metric data from the specified metric snapshot. On error, the response returns the appropriate HTTP status code and includes a response body with an error code and human readable message.

Response body example

{
  "status": "RELIABLE",
  "statusReason": "["TAA Alarm","Message"]",
  "metrics": [
    {
      "name": "Customer Friction",  
      "treatmentId": "C",
      "values": {
        "mean": 0,
        "percentDiff": 0,
        "confidenceIntervalLower": 0,
        "confidenceIntervalUpper": 0,
        "pValue": 0,
        "userCount": 0
      }
    }
  ]
}

Response body properties

Property Description Type

status

State of the metric snapshot.
Valid values: RELIABLE, UNRELIABLE.

String

statusReason

Reason why the metric snapshot status is unreliable.

String

metrics

List of metric snapshot values for your skill.

Array of objects

metrics[].name

A unique name that identifies the A/B test metric.

String

metrics[].treatmentId

Testing group assigned to the test that generated these metrics: control or new treatment.
Valid values: C, T1.

String

metrics[].values

Metric result values.

Object

metrics[].values.mean

Mean or average of each sample.
Includes either the T1 or C testing group.

Double

metrics[].values.percentDiff

Relative percent difference between the mean of the T1 group and the mean of the C group.

Double

metrics[].values.confidenceIntervalLower

Lower limit number of the confidence interval range. The confidence interval measures the probability that the mean falls within a range.

Double

metrics[].values.confidenceIntervalUpper

Upper limit number of the confidence interval range. The confidence interval measures the probability that the mean falls within a range.

Double

metrics[].values.pValue

Probability that the difference between the T1 and C means is due to random sampling errors.

Double

metrics[].values.userCount

Number of users in the T1 sample*.

Long

HTTP status codes

Status Description

200 OK

Response body contains the metric data for the specified snapsnot.

400 Bad Request

Indicates that one or more properties in the request body aren't valid.
The following example shows the response body with the error code and message.

{
     "message": "The property is outside the allowed range.",
     "code": "INVALID_STRING_LENGTH"
}

401 Unauthorized

Request didn't include the authorization token, or the included token expired or isn't valid. Or, you don't have access to the resource.

403 Forbidden

Indicates that the authorization token is valid, but the requested operation isn't allowed.

404 Not Found

Requested resource not found.

429 Too Many Requests

Permitted rate limit, specified as number of requests per unit of time, exceeded. Retry the request by using exponential back-off.

500 Server Error

Error occurred on the server. Retry the request by using exponential back-off.

503 Service Unavailable

Server is down for maintenance, overloaded, or otherwise unavailable to handle the incoming request.

List A/B tests

Get the list of A/B tests associated with the specified skill ID. This operation returns completed and in-progress tests.

You can also retrieve the list of A/B tests by using the list-experiments ASK CLI command.

Request

To list tests, you make a GET request to the /v1/skills/{skillId}/experiments resource.

Request path and header example

Copied to clipboard.

GET /v1/skills/{skillId}/experiments&maxResults={maxResults}&nextToken={nextToken}
Host: api.amazonalexa.com
Content-Type: application/json
Authorization: Bearer {access token}
Accept: application/json

Request path and header parameters

Parameter Located in Description Type Required

skillId

Path

Identifies the skill associated with your A/B test.
Valid values: 1 – 255 characters.

String

Yes

maxResults

Query

Maximum number of results to return in the response.
Valid values: 1–50. Default: 10.

Integer

No

nextToken

Query

Token from the previous response. If not included, the Alexa service returns the first page of results.
Include if iterating over a paginated response.

String

No

access token

Header

LWA token.

String

Yes

Request body example

The request has no body.

Request body properties

The request has no body.

Response

A successful response returns HTTP 200 OK, along with a list of A/B tests defined for your skill. If the number of results is too large, the response includes the paginationContext.nextToken. Use the token in your next List A/B tests request to get the next page of results. On error, the response returns the appropriate HTTP status code and includes a response body with an error code and human readable message.

Response body example

{
  "experiments": [
    {
      "experimentId": "experiment.id.1",
      "name": "my_first-experiment",
      "state": "CREATED",
      "experimentHistory": {
        "creationTime": "2021-04-29T18:20:50.671Z",
        "startTime": "2021-04-29T18:20:50.671Z",
        "stoppedReason": "string"
      }
    }
  ]
}

Response body properties

Property Description Type

experiments

List of A/B tests defined for the specified skill

Array of objects

experiments[].experimentId

Unique identifier for the A/B test.

String

experiments[].name

Unique name of the A/B test.

String

experiments[].state

Current state of the test.
Valid values:

  • CREATED - Created successfully but not yet enabled or started.
  • ENABLING - Initiated the transition to ENABLED.
  • ENABLED - Experiment configurations deployed but only customer treatment overrides set to T1 can view the T1 experience of a skill. In this state, no metrics collected.
  • RUNNING - Test started and a percentage of skill customers defined in the exposurePercentage enter into the test. Customers are randomly assigned the T1 or C experience. Metric collection begins.
  • STOPPING - Initiated the transition to STOPPED.
  • STOPPED - Test ended and all customers now experience the C behavior. Metrics collection stopped.
  • FAILED - Experiment configurations failed to deploy during the ENABLING state.

String

experiments[].experimentHistory

History of an A/B test.

Object

experiments[].experimentHistory.creationTime

Date and time the test was created or updated.
Defined in ISO 8601 format, YYYY-MM-DDThh:mm:ssZ.

String

experiments[].experimentHistory.startTime

Date and time the test started.
Defined in ISO 8601 format, YYYY-MM-DDThh:mm:ssZ.

String

experiments[].experimentHistory.stoppedReason

Reason the text stopped.
Valid values: DEVELOPER_REQUEST, EXPERIMENT_SUCCESS, EXPERIMENT_ISSUE.

String

paginationContext

Indicates there are more matching results. If present, the response contains a subset of the results. When there are no more matching results, paginationContext isn't included in the response.

Object

paginationContext.nextToken

Identifies the next set of results to return. The token expires in 24 hours.
Use this value for the nextToken parameter in a subsequent List A/B tests request.

String

HTTP status codes

Status Description

200 OK

Response body contains a list of A/B tests defined for the skill.

400 Bad Request

Indicates that one or more properties in the request body aren't valid.
The following example shows the response body with the error code and message.

{
     "message": "The property is outside the allowed range.",
     "code": "INVALID_STRING_LENGTH"
}

401 Unauthorized

Request didn't include the authorization token, or the included token expired or isn't valid. Or, you don't have access to the resource.

403 Forbidden

Indicates that the authorization token is valid, but the requested operation isn't allowed.

404 Not Found

Requested resource not found.

429 Too Many Requests

Permitted rate limit, specified as number of requests per unit of time, exceeded. Retry the request by using exponential back-off.

500 Server Error

Error occurred on the server. Retry the request by using exponential back-off.

503 Service Unavailable

Server is down for maintenance, overloaded, or otherwise unavailable to handle the incoming request.

List metric snapshots

List metric snapshots associated with the experiment ID of an A/B test. A metric snapshot represents the metric data that's available for your test in a specific time range. This operation doesn't include detailed data about each metric snapshot. To get detailed analytics about a test, use the Get metric snapshot data operation.

You can also retrieve the list of snapshots by using the list-experiment-metric-snapshots ASK CLI command.

Request

To list snapshots, you make a GET request to the /v1/skills/{skillId}/experiments/{experimentId}/metricSnapshots resource.

Request path and header example

Copied to clipboard.

GET /v1/skills/{skillId}/experiments/{experimentId}/metricSnapshots&maxResults={maxResults}&nextToken={nextToken}
Host: api.amazonalexa.com
Content-Type: application/json
Authorization: Bearer {access token}
Accept: application/json

Request path and header parameters

Parameter Located in Description Type Required

skillId

Path

Identifies the skill associated with your A/B test.
Valid values: 1 – 255 characters.

String

Yes

experimentId

Path

Identifies the A/B test.
Use the experiment ID returned from Create A/B test or List A/B tests.

String

Yes

maxResults

Query

Maximum number of results to return in the response.
Valid values: 1–50. Default: 10.

Integer

No

nextToken

Query

Token from the previous response. If not included, the Alexa service returns the first page of results.
Include if iterating over a paginated response.

String

No

access token

Header

LWA token.

String

Yes

Request body example

The request has no body.

Request body properties

The request has no body.

Response

A successful response returns HTTP 200 OK, along with a list of metric snapshots. If the number of results is too large, the response includes the paginationContext.nextToken. Use the token in your next List metric snapshots request to get the next page of results. On error, the response returns the appropriate HTTP status code and includes a response body with an error code and human readable message.

Response body example

{
    "metricSnapshots": [
    {
      "metricSnapshotId": "snapshot.id.1",
      "startDate": "2021-04-20T15:24:31.563Z",
      "endDate": "2021-04-20T15:24:31.563Z"
    },
    {
      "metricSnapshotId": "snapshot.id.2",
      "startDate": "2021-04-21T15:24:31.563Z",
      "endDate": "2021-04-21T15:24:31.563Z"
    }
  ]
}

Response body properties

Property Description Type

metricSnapshots

List of metric snapshots for the specified A/B test.

Array of objects

metricSnapshots[].metricSnapshotId

Unique identifier for the metric snapshot of an A/B test.

String

metricSnapshots[].startDate

Date and time the metric snapshot started.
Defined in ISO 8601 format, YYYY-MM-DDThh:mm:ssZ.

String

metricSnapshots[].endDate

Date and time the metric snapshot started.
Defined in ISO 8601 format, YYYY-MM-DDThh:mm:ssZ.

String

paginationContext

Indicates there are more matching results. If present, the response contains a subset of the results. When there are no more matching results, paginationContext isn't included in the response.

Object

paginationContext.nextToken

Identifies the next set of results to return. The token expires in 24 hours.
Use this value for the nextToken parameter in a subsequent List metric snapshots request.

String

HTTP status codes

Status Description

200 OK

Response body contains a list of metric snapshots for the specified A/B test.

400 Bad Request

Indicates that one or more properties in the request body aren't valid.
The following example shows the response body with the error code and message.

{
     "message": "The property is outside the allowed range.",
     "code": "INVALID_STRING_LENGTH"
}

401 Unauthorized

Request didn't include the authorization token, or the included token expired or isn't valid. Or, you don't have access to the resource.

403 Forbidden

Indicates that the authorization token is valid, but the requested operation isn't allowed.

404 Not Found

Requested resource not found.

429 Too Many Requests

Permitted rate limit, specified as number of requests per unit of time, exceeded. Retry the request by using exponential back-off.

500 Server Error

Error occurred on the server. Retry the request by using exponential back-off.

503 Service Unavailable

Server is down for maintenance, overloaded, or otherwise unavailable to handle the incoming request.

Set customer treatment override

Set the testing group to C or T1 to override the customer treatment for the specified A/B test. You can use this operation to override the treatment for testing your skill experience.

You can also set the customer treatment override by using the set-customer-treatment-override ASK CLI command.

Request

To update the customer treatment override, you make a PUT request to the ` /v1/skills/{skillId}/experiments/{experimentId}/treatmentOverrides` resource.

Request path and header example

Copied to clipboard.

PUT /v1/skills/{skillId}/experiments/{experimentId}/treatmentOverrides/~current
Host: api.amazonalexa.com
Content-Type: application/json
Authorization: Bearer {access token}

Request path and header parameters

Parameter Located in Description Type Required

skillId

Path

Identifies the skill associated with your A/B test.
Valid values: 1 – 255 characters.

String

Yes

experimentId

Path

Identifies the A/B test.
Use the experiment ID returned from Create A/B test or List A/B tests.

String

Yes

access token

Header

LWA token.

String

Yes

Request body example

Copied to clipboard.

{
    "treatmentId": "T1"
}

Request body properties

Property Description Type Required

treatmentId

Testing group: control or new treatment.
Valid values: C, T1.

String

Response

A successful response returns HTTP 202 Accepted. On error, the response returns the appropriate HTTP status code and includes a response body with an error code and human readable message.

Response body example

The response has no body.

Response body properties

The response has no body.

HTTP status codes

Status Description

202 Accepted

Override treatment updated successfully.

400 Bad Request

Indicates that one or more properties in the request body aren't valid.
The following example shows the response body with the error code and message.

{
     "message": "The property is outside the allowed range.",
     "code": "INVALID_STRING_LENGTH"
}

401 Unauthorized

Request didn't include the authorization token, or the included token expired or isn't valid. Or, you don't have access to the resource.

403 Forbidden

Indicates that the authorization token is valid, but the requested operation isn't allowed.

404 Not Found

Requested resource not found.

429 Too Many Requests

Permitted rate limit, specified as number of requests per unit of time, exceeded. Retry the request by using exponential back-off.

500 Server Error

Error occurred on the server. Retry the request by using exponential back-off.

503 Service Unavailable

Server is down for maintenance, overloaded, or otherwise unavailable to handle the incoming request.

Update A/B test

Update the settings of an existing A/B test. To make changes, the test must be in the CREATED state. You can't update an active test. To update the exposure value of an active test, use Update exposure.

You can also update an A/B test by using the update-experiment ASK CLI command.

Request

To update a test, you make a POST request to the /v1/skills/{skillId}/experiments resource.

Request path and header example

Copied to clipboard.

POST /v1/skills/{skillId}/experiments/{experimentId}/properties
Host: api.amazonalexa.com
Content-Type: application/json
Authorization: Bearer {access token}

Request path and header parameters

Parameter Located in Description Type Required

skillId

Path

Identifies the skill associated with your A/B test.
Valid values: 1 – 255 characters.

String

Yes

experimentId

Path

Identifies the A/B test.
Use the experiment ID returned from Create A/B test or List A/B tests.

String

Yes

access token

Header

LWA token.

String

Yes

Request body example

Copied to clipboard.

{
  "experiment": {
    "description": "experiment_1",
    "plannedDuration": "P3W",
    "exposurePercentage": 0,
    "metricConfigurations": [
      {
        "name": "Customer Friction",  
        "metricTypes": [
          "GUARDRAIL"
        ],
        "expectedChange": "INCREASE"
      }
    ]
  }
}

Request body properties

Property Description Type Required

experiment

A/B test settings.

Object

Yes

experiment.description

Description of the hypothesis you set for your A/B test.
Used for reference purposes and doesn't impact the outcome.

String

Yes

experiment.plannedDuration

Number of weeks you plan to run your A/B test.
Used for reference purposes and doesn't impact the test length.
Defined in ISO-8601 format, P[n]W.
Valid values:1–4 weeks.

String

Yes

experiment.exposurePercentage

Percentage of unique customers that you plan to include in your A/B test. This number doesn't have to match your total users.

For example, if you have 100 total customers and you set your exposurePercentage to 40 percent, you're including 40 customers in your test. In this case, your test includes 20 customers in C and 20 customers in T1. The remaining 60 customers aren't included in the test and receive the default behavior equivalent to C, meaning they don't contribute to the test metrics.
Valid values: 0–100.

Integer

Yes

experiment.metricConfigurations

List of A/B test metrics that you plan to use in your test.

Array

Yes

experiment.metricConfigurations[].name

Unique name of the test metric.
For valid names, see key and guardrail metrics.
Valid value: 1–125 characters.

String

Yes

experiment.metricConfigurations[].metricTypes

List of metrics used in your A/B test.
Valid values:
  •  KEY – Identifies a metric that provides clear evidence of expected changes caused by the new treatment experience.
  •  GUARDRAIL – Identifies a metric that detects unexpected regressions caused by the new treatment experience.

Array of strings

Yes

experiment.metricConfigurations[].expectedChange

Direction that you expect a metric to trend during the testing.
Used for reference purposes and doesn't impact the actual test outcome.
Valid values:
  •  INCREASE – Upward change in metric value.
  •  DECREASE – Downward change in metric value.

String

Yes

Response

A successful response returns HTTP 204 Success. On error, the response returns the appropriate HTTP status code and includes a response body with an error code and human readable message.

Response body example

The response has no body.

Response body properties

The response has no body.

HTTP status codes

Status Description

204 No content

A/B test values updated successfully.

400 Bad Request

Indicates that one or more properties in the request body aren't valid.
The following example shows the response body with the error code and message.

{
     "message": "The property is outside the allowed range.",
     "code": "INVALID_STRING_LENGTH"
}

401 Unauthorized

Request didn't include the authorization token, or the included token expired or isn't valid. Or, you don't have access to the resource.

403 Forbidden

Indicates that the authorization token is valid, but the requested operation isn't allowed.

404 Not Found

Requested resource not found.

429 Too Many Requests

Permitted rate limit, specified as number of requests per unit of time, exceeded. Retry the request by using exponential back-off.

500 Server Error

Error occurred on the server. Retry the request by using exponential back-off.

503 Service Unavailable

Server is down for maintenance, overloaded, or otherwise unavailable to handle the incoming request.

Update A/B test state

Update the state of the specified A/B test.

Request

To update the test state, you make a POST request to the /v1/skills/{skillId}/experiments/{experimentId}/state resource.

Request path and header example

Copied to clipboard.

POST /v1/skills/{skillId}/experiments/{experimentId}/state
Host: api.amazonalexa.com
Content-Type: application/json
Authorization: Bearer {access token}

Request path and header parameters

Parameter Located in Description Type Required

skillId

Path

Identifies the skill associated with your A/B test.
Valid values: 1 – 255 characters.

String

Yes

experimentId

Path

Identifies the A/B test.
Use the experiment ID returned from Create A/B test or List A/B tests.

String

Yes

access token

Header

LWA token.

String

Yes

Request body example

Copied to clipboard.

{
  "targetState": "ENABLED"
}

Request body properties

Property Description Type Required

targetState

New state of the test.
For more details about how to use states, see Lifecycle of an A/B test.
Valid values:
- ENABLED – Experiment configurations deployed but only customer treatment overrides set to T1 can view the T1 experience of a skill. In this state, no metrics collected. - RUNNING – Test started and a percentage of skill customers defined in the exposurePercentage enter into the test. Customers are randomly assigned the T1 or C experience. Metric collection begins. - STOPPED – Test ended and all customers now experience the C behavior. Metrics collection stopped.

String

Yes

stoppedReason

Reason the test stopped.
Include when you set the state to STOPPED. Valid values: DEVELOPER_REQUEST, EXPERIMENT_SUCCESS, EXPERIMENT_ISSUE.

String

No

Response

A successful response returns HTTP 202 Accepted, along with the experiment identifier in the Location header. Use the URL in the Location header to track the test state. On error, the response returns the appropriate HTTP status code and includes a response body with an error code and human readable message.

Response body example

The response has no body.

Response body properties

The response has no body.

HTTP status codes

Status Description

202 Accepted

Test state updated successfully.

400 Bad Request

Indicates that one or more properties in the request body aren't valid.
The following example shows the response body with the error code and message.

{
     "message": "The property is outside the allowed range.",
     "code": "INVALID_STRING_LENGTH"
}

401 Unauthorized

Request didn't include the authorization token, or the included token expired or isn't valid. Or, you don't have access to the resource.

403 Forbidden

Indicates that the authorization token is valid, but the requested operation isn't allowed.

404 Not Found

Requested resource not found.

429 Too Many Requests

Permitted rate limit, specified as number of requests per unit of time, exceeded. Retry the request by using exponential back-off.

500 Server Error

Error occurred on the server. Retry the request by using exponential back-off.

503 Service Unavailable

Server is down for maintenance, overloaded, or otherwise unavailable to handle the incoming request.

Update exposure

Update the exposure value of an A/B test. You can use this operation when your test is in one of the following states:

  • CREATED – Increase or decrease the exposure value. If you want to update other settings, use Update A/B test.
  • RUNNING – Increase your exposure value. To decrease your exposure value after you started the test, stop the test and create a new test.

You can also update the exposure value by using the update-exposure ASK CLI command.

Request

To update the exposure value, you make a POST request to the /v1/skills/{skillId}/experiments resource.

Request path and header example

Copied to clipboard.

POST /v1/skills/{skillId}/experiments/{experimentId}/exposurePercentage
Host: api.amazonalexa.com
Content-Type: application/json
Authorization: Bearer {access token}

Request path and header parameters

Parameter Located in Description Type Required

skillId

Path

Identifies the skill associated with your A/B test.
Valid values: 1 – 255 characters.

String

Yes

experimentId

Path

Identifies the A/B test.
Use the experiment ID returned from Create A/B test or List A/B tests.

String

Yes

access token

Header

LWA token.

String

Yes

Request body example

Copied to clipboard.

{
  "exposurePercentage": 60
}

Request body properties

Property Description Type Required

exposurePercentage

New percentage of customers to be part of the test. If the test is running, you can increase the exposure percentage value of a running test, but not decrease it.
Valid values: 0–100.

Integer

Yes

Response

A successful response returns HTTP 204 No content. On error, the response returns the appropriate HTTP status code and includes a response body with an error code and human readable message.

Response body example

The response has no body.

Response body properties

The response has no body.

HTTP status codes

Status Description

204 No content

Test exposure value updated successfully.

400 Bad Request

Indicates that one or more properties in the request body aren't valid.
The following example shows the response body with the error code and message.

{
     "message": "The property is outside the allowed range.",
     "code": "INVALID_STRING_LENGTH"
}

401 Unauthorized

Request didn't include the authorization token, or the included token expired or isn't valid. Or, you don't have access to the resource.

403 Forbidden

Indicates that the authorization token is valid, but the requested operation isn't allowed.

404 Not Found

Requested resource not found.

429 Too Many Requests

Permitted rate limit, specified as number of requests per unit of time, exceeded. Retry the request by using exponential back-off.

500 Server Error

Error occurred on the server. Retry the request by using exponential back-off.

503 Service Unavailable

Server is down for maintenance, overloaded, or otherwise unavailable to handle the incoming request.


Was this page helpful?

Last updated: Jul 17, 2024