A/B Testing SMAPI APIs

Use the following SMAPI APIs to create, update, and manage A/B tests for your skills.

Authorization

Each API header request must contain an access token retrieved from Login with Amazon. For more details about obtaining an access token, see Get an Access Token for SMAPI.

Create A/B test

Creates a new A/B test for a skill.

You can also call this API using the create-experiment ASK CLI command.

Request

POST /v1/skills/{skillId}/experiments/
Host: api.amazonalexa.com
Accept: application/json
Authorization: Bearer {LWA Token}

Request fields

Field Description Type Required?

skillId

The skill ID associated with your A/B test.

Path

Yes

Request body structure

The following example illustrates the structure of the request body.

{
  "experiment": {
    "name": "my_first_experiment",
    "description": "experiment_1",
    "type": "ENDPOINT_BASED",
    "plannedDuration": "P4W",
    "exposurePercentage": 20,
    "metricConfigurations": [
      {
        "name": "Customer Friction",
        "metricTypes": [
          "GUARDRAIL"
        ],
        "expectedChange": "INCREASE"
      }
    ]
  }
}

Request body fields

Field Description Type Required?

experiment

An object that contains A/B test settings for your skill.

Object

Yes

experiment.name

A unique name for your A/B test. You can't use an identical name for two tests. However, you can use the same name if you delete one of the tests. Maximum 255 characters.

String

Yes

experiment.description

A description of the hypothesis you set for your A/B test. Only used for reference purposes and doesn't impact the outcome.

String

Yes

experiment.type

The type of A/B test you want to run for your skill.

Accepted values:

  • ENDPOINT_BASED- Sets your test's T1 value to your skill version in the LIVE stage. To separate your C and T1 behaviors, you must add conditional logic in your skill code. For more details about separating endpoint-based tests, see Set up an Endpoint-based A/B Test.

String

Yes

experiment.plannedDuration

The number of weeks you're running your A/B test. Only used for reference purposes and doesn't impact the actual test length.

Accepted values: Any value between 1 and 20 weeks represented in the ISO-8601 format. For example, you represent a value of four weeks with the ISO-8601 value of P4W.

String

Yes

experiment.exposurePercentage

The percentage of unique customers that you're including in your A/B test. This number doesn't have to match your total users.

For example, if you have 100 total customers and you set your exposurePercentage to 40 percent, you're including 40 customers in your test. In this case, your test includes 20 customers in C and 20 customers in T1. The remaining 60 customers aren't included in the test and receive the default behavior equivalent to C, meaning they don't contribute to the test metrics.

Accepted values: Any value between 0 to 100.

Integer

Yes

experiment.metricConfigurations

A list of accepted A/B test metrics that you're testing in your experiment.

Array

Yes

experiment.metricConfigurations.name

The unique name of an A/B test metric. Maximum 125 characters.

String

Yes

experiment.metricConfigurations.metricTypes

A list of values used to detect changes in your A/B test.

Accepted values:
  •  KEY - A metric configuration that provides clear evidence of expected changes caused by your new treatment experience.
  •  GUARDRAIL - A metric configuration that detects unexpected regressions caused by your new treatment experience.

Array

Yes

experiment.expectedChange

The expected outcome of each A/B test metric. Only used for reference purposes and doesn't impact the actual test outcome.

Accepted values:
  •  INCREASE - An upwards change in one of your defined metrics configurations.
  •  DECREASE - A downwards change in one of your defined metrics configurations.

String

Yes

Response

HTTP/1.1 201 Experiment created. Returns the generated experiment identifier in 'Location' header.
Content-Type: application/json
Location: {url}
X-Amzn-RequestId: {request-id}

Response header fields

Field Description Type

Location

The relative URL used to retrieve your A/B test details.

String

X-Amzn-RequestId

The unique identifier for the request. If a problem occurs, Amazon can use this value to troubleshoot the problem.

String

HTTP response codes

Status code Description

201

Experiment created. Returns the generated experiment identifier in 'Location' header.

400

Server can't process the request due to a client error.

401

The auth token is invalid/expired or doesn't have access to the resource.

403

The operation being requested isn't allowed.

404

The requested resource isn't found.

429

Exceeds the permitted request limit. Throttling criteria includes total requests, per API, ClientId, and CustomerId.

500

Internal Server Error.

503

Service Unavailable.

Get A/B test

Retrieves detailed information about an A/B test.

You can also call this API using the get-experiment ASK CLI command.

Request

GET /v1/skills/{skillId}/experiments/{experimentId}
Host: api.amazonalexa.com
Accept: application/json
Authorization: Bearer {LWA Token}

Request fields

Field Description Type Required?

skillId

The skill ID associated with your A/B test.

Path

Yes

experimentId

The unique ID of the A/B test you want to retrieve.

Path

Yes

Response

HTTP/1.1 200 Returned skill experiment
Host: api.amazonalexa.com
X-Amzn-RequestId: {request-id}
Content-Type: application/json

Response header fields

Field Description Type

X-Amzn-RequestId

The unique identifier for the request. If a problem occurs, Amazon can use this value to troubleshoot the problem.

String

Response body fields

The following example illustrates the structure of the response body.

{
  "experiment": {
    "name": "my_first_experiment",
    "description": "experiment_1",
    "type": "ENDPOINT_BASED",
    "plannedDuration": "P3W",
    "exposurePercentage": 0,
    "metricConfigurations": [
      {
        "name": "Customer Friction",
        "metricTypes": [
          "GUARDRAIL"
        ],
        "expectedChange": "INCREASE"
      }
    ],
    "state": "CREATED",
    "history": {
      "creationTime": "2021-04-19T18:43:48.383Z",
      "startTime": "2021-04-19T18:43:48.383Z",
      "stoppedReason": "string"
    },
    "trigger": {
      "scheduledEndTime": "2021-04-19T18:43:48.383Z"
    }
  }
}

Response body fields

Field Description Type

experiment

An object that contains A/B test settings for your skill.

Object

experiment.name

The unique name of your A/B test.

String

experiment.description

The description of the hypothesis you're trying to prove with your A/B test. Only used for reference purposes and doesn't impact the outcome.

String

experiment.type

The type of A/B test you set for your skill.

Possible values:

  • ENDPOINT_BASED- Sets your test's T1 value to your skill version in the LIVE stage. To separate your C and T1 behaviors, you must add conditional logic in your skill code. For more details about separating endpoint-based tests, see Set up an Endpoint-based A/B Test.

String

experiment.plannedDuration

The number of weeks you're running the A/B test. Only used for reference purposes and doesn't impact the actual test length.

Accepted values: Any value between 1 and 20 weeks represented in the ISO-8601 format. For example, you represent a value of four weeks with the ISO-8601 value of P4W.

String

experiment.exposurePercentage

The percentage of unique customers included in your A/B test.

Integer

experiment.metricConfigurations

A list of accepted A/B test metrics you selected for your test.

Array

experiment.metricConfigurations.name

The unique name of an A/B test metric.

String

experiment.metricConfigurations.metricTypes

A list of values used to detect changes in your A/B test.

Possible values:
  •  KEY - A test metric that provides clear evidence of expected changes caused by your new treatment experience.
  •  GUARDRAIL - A test metric that detects unexpected regressions caused by your new treatment experience.

Array

experiment.expectedChange

The expected outcome of each A/B test metric. Only used for reference purposes and doesn't impact the actual test outcome.

Possible values:
  •  INCREASE - An upwards change in one of your test metrics.
  •  DECREASE - A downwards change in one of your test metrics.

String

experiment.history.creationTime

The creation date of your A/B test.

String

experiment.history.startTime

The start time of your A/B test.

String

experiment.history.stoppedReason

The reason you stopped your A/B test.

String

experiment.trigger

An object that contains A/B test triggers for your skill.

Object

experiment.trigger.scheduledEndTime

The expected end time of your A/B test.

String

HTTP response codes

Status code Description

200

Returned skill test.

401

The auth token is invalid/expired or doesn't have access to the resource.

403

The requested operation isn't allowed.

404

The requested resource isn't found.

429

Exceeds the permitted request limit. Throttling criteria includes total requests, per API, ClientId, and CustomerId.

500

Internal Server Error.

503

Service Unavailable.

Get list of A/B tests

Retrieves a list of tests associated with a skill ID. Returns both completed and in-progress tests.

You can also call this API using the list-experiment ASK CLI command.

Request

GET /v1/skills/{skillId}/experiments
Host: api.amazonalexa.com
Accept: application/json
Authorization: Bearer {LWA Token}

Request header fields

Field Description Type Required?

skillId

The skill ID associated with your test.

Path

Yes

NextToken

A token used to retrieve the next page of results.

Path

No

MaxResults

Sets the maximum number of results returned in the response body. maxResults shouldn't exceed the upper limit. The response might contain fewer results than maxResults, but it never contains more.

Path

No

Response

HTTP/1.1 200 Returned skill experiments.
Host: api.amazonalexa.com
X-Amzn-RequestId: {request-id}
Content-Type: application/json

Response header fields

Field Description Type

X-Amzn-RequestId

The unique identifier for the request. If a problem occurs, Amazon can use this value to troubleshoot the problem.

String

Response body structure

The following example illustrates the structure of the response body.

{
  "experiments": [
    {
      "experimentId": <experiment-id> ,
      "name": "my_first-experiment",
      "state": "CREATED",
      "experimentHistory": {
        "creationTime": "2021-04-29T18:20:50.671Z",
        "startTime": "2021-04-29T18:20:50.671Z",
        "stoppedReason": "string"
      }
    }
  ]
}

Response body fields

Field Description Type

experiments

An object that contains A/B test settings for your skill.

Array

experiments.experimentId

The unique identifier of the A/B test associated with your skill.

String

experiments.name

The unique name of your A/B test.

String

experiments.state

The current state of your A/B test. For more details about how to use states, see Lifecycle of an A/B Test.

Possible values:
  • CREATED - Your test is created but isn't enabled or started.
  • ENABLING - Your test is transitioning from CREATED to ENABLED.
  • ENABLED - Your test configurations are deployed.
  • RUNNING - Your test has started.
  • STOPPING - Your test is transitioning from RUNNING to STOPPED.
  • STOPPED - Your test has ended.
  • DELETED - Your test is deleted.
  • FAILED - Your test configurations failed to deploy when trying to enable or start the test.

String

experiments.experimentHistory

An object that contains historical properties of your A/B test.

Object

experiments.experimentHistory.creationTime

The creation date of your A/B test.

String

experiments.experimentHistory.startTime

The start time of your A/B test.

String

experiments.experimentHistory.stoppedReason

The reason you stopped your A/B test.

String

HTTP response codes

Status code Description

200

Returned a list of skill tests.

401

The auth token is invalid/expired or doesn't have access to the resource.

403

The requested operation isn't allowed.

404

The requested resource isn't found.

429

Exceeds the permitted request limit. Throttling criteria includes total requests, per API, ClientId, and CustomerId.

500

Internal Server Error.

503

Service Unavailable.

Update A/B test

Update the settings of an A/B test. You can only call this API if your test is in the CREATED state, meaning you can't update an active test. To update the exposure value of an active test, use Update exposure.

You can also call this API using the update-experiment ASK CLI command.

Request

POST /v1/skills/{skillId}/experiments/{experimentId}/properties
Host: api.amazonalexa.com
Accept: application/json
Authorization: Bearer {LWA Token}

Request header fields

Field Description Type Required?

skillId

The skill ID associated with your A/B test.

Path

Yes

experimentId

The unique ID of the A/B test you want to update.

Path

Yes

Request body structure

The following example illustrates the structure of the response body.

{
  "experiment": {
    "description": "experiment_1",
    "plannedDuration": "P3W",
    "exposurePercentage": 0,
    "metricConfigurations": [
      {
        "name": "Customer Friction",  
        "metricTypes": [
          "GUARDRAIL"
        ],
        "expectedChange": "INCREASE"
      }
    ]
  }
}

Request body fields

Field Description Type

experiment

An object that contains A/B test settings for your skill.

Object

experiment.description

A description of the hypothesis you set for your A/B test. Only used for reference purposes and doesn't impact the outcome.

String

experiment.plannedDuration

The number of weeks you're running the A/B test. Only used for reference purposes and doesn't impact the actual test length.

Accepted values: Any value between 1 and 20 weeks represented in the ISO-8601 format. For example, you represent a value of four weeks with the ISO-8601 value of P4W.

String

experiment.exposurePercentage

The percentage of unique customers that you're including in the A/B test. This number doesn't have to match your total users.

For example, if you have 100 total customers and you set your exposurePercentage to 40 percent, you're including 40 customers in your test. In this case, your test includes 20 customers in C and 20 customers in T1. The remaining 60 customers aren't included in the test and receive the default behavior equivalent to C – they don't contribute to the test metrics.

Accepted values: Any value between 0 to 100.

Integer

experiment.metricConfigurations

A list of accepted A/B test metrics you selected for your test.

Array

experiment.metricConfigurations.name

The unique name of a test metric.

String

experiment.metricConfigurations.metricTypes

A list of values used to detect changes in your A/B test.

Possible values:
  •  KEY - A test metric that provides clear evidence of expected changes caused by your new treatment experience.
  •  GUARDRAIL - A test metric that detects unexpected regressions caused by your new treatment experience.

Array

experiment.expectedChange

The expected outcome of each A/B test metric. Only used for reference purposes and doesn't impact the actual test outcome.

Accepted values:
  •  INCREASE - An upwards change in one of your test metrics.
  •  DECREASE - A downwards change in one of your test metrics.

String

Response

HTTP/1.1 204 Success. No content.
Host: api.amazonalexa.com
X-Amzn-RequestId: {request-id}
Content-Type: application/json

Response header fields

Field Description Type

X-Amzn-RequestId

The unique identifier for the request. If a problem occurs, Amazon can use this value to troubleshoot the problem.

String

Response body example

The response returns your updated test values.

{
  "experiment": {
    "name": "Customer Friction",  
    "description": "experiment_1",
    "type": "ENDPOINT_BASED",
    "plannedDuration": "P3W",
    "exposurePercentage": 0,
    "metricConfigurations": [
      {
        "name": "Customer Friction",  
        "metricTypes": [
          "GUARDRAIL"
        ],
        "expectedChange": "INCREASE"
      }
    ]
  }
}

HTTP response codes

Status code Description

204

Success. No content.

400

Server can't process the request due to a client error.

401

The auth token is invalid/expired or doesn't have access to the resource.

403

The requested operation isn't allowed.

404

The requested resource isn't found.

409

The request couldn't be completed due to a conflict with the current state of the target resource.

429

Exceeds the permitted request limit. Throttling criteria includes total requests, per API, ClientId, and CustomerId.

500

Internal Server Error.

503

Service Unavailable.

Delete A/B test

Delete an A/B test.

You can only call this API if your test is in CREATED state. If you need to delete your test after you start it (meaning your test is in the RUNNING state), you must completely delete your skill with the Delete a Skill API.

You can also call this API using the delete-experiment ASK CLI command.

Request

DELETE /v1/skills/{skillId}/experiments/{experimentId}
Host: api.amazonalexa.com
Accept: application/json
Authorization: Bearer {LWA Token}

Request fields

Field Description Type Required?

skillId

The skill ID associated with your test.

Path

Yes

experimentId

The unique ID of the A/B test you want to delete.

Path

Yes

Response header

HTTP/1.1 204 Success. No content.
Host: api.amazonalexa.com
X-Amzn-RequestId: {request-id}
Content-Type: application/json
Field Description Type

X-Amzn-RequestId

The unique identifier for the request. If a problem occurs, Amazon can use this value to troubleshoot the problem.

String

HTTP response codes

Status code Description

204

Success. No content.

401

The auth token is invalid/expired or doesn't have access to the resource.

403

The requested operation isn't allowed.

404

The requested resource isn't found.

409

The request couldn't be completed due to a conflict with the current state of the target resource.

429

Exceeds the permitted request limit. Throttling criteria includes total requests, per API, ClientId, and CustomerId.

500

Internal Server Error.

503

Service Unavailable.

Update exposure

Updates the exposure value of an A/B test. You can only call this API if your test is in the CREATED or RUNNING state.

  • RUNNING – You can only increase your exposure value. To decrease your exposure value after you have started your test, stop your test and create a new one.
  • CREATED – Increase or decrease your exposure value. If you want to update other settings, use Update A/B test

You can also call this API using the update-exposure ASK CLI command.

Request

POST /v1/skills/{skillId}/experiments/{experimentId}/exposurePercentage
Host: api.amazonalexa.com
Accept: application/json
Authorization: Bearer {LWA Token}

Request header fields

Field Description Type Required?

skillId

The skill ID associated with your test.

Path

Yes

experimentId

The unique ID of the A/B test you want to update.

Path

Yes

Request body fields

Field Description Type Required?

exposurePercentage

The new exposure percentage you want to update your A/B test to. You can only increase the exposure percentage value of a running test, not decrease it.

Accepted values: Any value between 0 and 100 that's higher than your current exposure percentage.

Integer

Yes

Response

HTTP/1.1 204 Success. No content.
Host: api.amazonalexa.com
X-Amzn-RequestId: {request-id}
Content-Type: application/json

Response header fields

Field Description Type

X-Amzn-RequestId

The unique identifier for the request. If a problem occurs, Amazon can use this value to troubleshoot the problem.

String

HTTP response codes

Status code Description

204

Success. No content.

400

Server can't process the request due to a client error.

401

The auth token is invalid/expired or doesn't have access to the resource.

403

The requested operation isn't allowed.

404

The requested resource isn't found.

409

Conflict with the resource state.

429

Exceeds the permitted request limit. Throttling criteria includes total requests, per API, ClientId, and CustomerId.

500

Internal Server Error.

503

Service Unavailable.

Get customer treatment override

Retrieves the current customer treatment override value of an A/B test.

You can also call this API using the get-customer-treatment-override ASK CLI command.

Request

GET /v1/skills/{skillId}/experiments/{experimentId}/treatmentOverrides/~current
Host: api.amazonalexa.com
Accept: application/json
Authorization: Bearer {LWA Token}

Request fields

Field Description Type Required?

skillId

The skill ID associated with your A/B test.

Path

Yes

experimentId

The unique ID of the A/B test you want to update.

Path

Yes

Response header fields

HTTP/1.1 200. Returned customer treatment override details.
Host: api.amazonalexa.com
X-Amzn-RequestId: {request-id}
Content-Type: application/json
Field Description Type

X-Amzn-RequestId

The unique identifier for the request. If a problem occurs, Amazon can use this value to troubleshoot the problem.

String

Response body fields

Field Description Type

treatmentId

The A/B testing group that you set your skill to.

Possible values:

  • C – Control. The skill behavior you aren't testing. This group of customers continue to receive your current skill experience.
  • T1 – Treatment. The skill behavior you're testing. This group of customers receive your new experimental skill experience.

String

treatmentOverrideCount

The current number of treatment overrides you set for your A/B test.

Integer

HTTP response codes

Status code Description

200

Returned customer treatment override details.

401

Server can't process the request due to a client error.

401

The auth token is invalid/expired or doesn't have access to the resource.

403

The requested operation isn't allowed.

404

The requested resource isn't found.

429

Exceeds the permitted request limit. Throttling criteria includes total requests, per API, ClientId, and CustomerId.

500

Internal Server Error.

503

Service Unavailable.

Set customer treatment override

Sets a new customer treatment override value of an A/B test.

You can also call this API using the set-customer-treatment-override ASK CLI command.

Request

PUT /v1/skills/{skillId}/experiments/{experimentId}/treatmentOverrides/~current
Host: api.amazonalexa.com
Accept: application/json
Authorization: Bearer {LWA Token}

Request fields

Field Description Type Required?

skillId

The skill ID associated with your A/B test.

Path

Yes

experimentId

The unique ID of the A/B test you want to update.

Path

Yes

Request body parameters

Field Description Type Required?

treatmentId

The A/B testing group that you want to set for your skill.

Accepted values:

  • C – Control. The skill behavior you aren't testing. This group of customers continue to receive your current skill experience.
  • T1 – Treatment. The skill behavior you're testing. This group of customers receive your new experimental skill experience.

String

Yes

Response header fields

HTTP/1.1 204. Success. No content.
Host: api.amazonalexa.com
X-Amzn-RequestId: {request-id}
Content-Type: application/json
Field Description Type

X-Amzn-RequestId

The unique identifier for the request. If a problem occurs, Amazon can use this value to troubleshoot the problem.

String

Response body fields

Field Description Type

treatmentId

The A/B testing group that you set your skill to.

Possible values:

  • C – Control. The skill behavior you aren't testing. This group of customers continue to receive your current skill experience.
  • T1 – Treatment. The skill behavior you're testing. This group of customers receive your new experimental skill experience.

String

HTTP response codes

Status code Description

204

Success. No content.

400

Server can't process the request due to a client error.

401

The auth token is invalid/expired or doesn't have access to the resource.

403

The requested operation isn't allowed.

404

The requested resource isn't found.

409

Conflict with the resource state.

429

Exceeds the permitted request limit. Throttling criteria includes total requests, per API, ClientId, and CustomerId.

500

Internal Server Error.

503

Service Unavailable.

Get A/B test state

Retrieves the current state of an A/B test.

You can also call this API using the get-experiment-state ASK CLI command.

Request

GET /v1/skills/{skillId}/experiments/{experimentId}/state
Host: api.amazonalexa.com
Accept: application/json
Authorization: Bearer {LWA Token}

Request fields

Field Description Type Required?

skillId

The skill ID associated with your A/B test.

Path

Yes

experimentId

The unique ID of the A/B test you want to retrieve.

Path

Yes

Response

HTTP/1.1 200. Returned skill experiment state.
Host: api.amazonalexa.com
X-Amzn-RequestId: {request-id}
Content-Type: application/json

Response header fields

Field Description Type

X-Amzn-RequestId

The unique identifier for the request. If a problem occurs, Amazon can use this value to troubleshoot the problem.

String

Response body

The following example illustrates the structure of the response body.

{
  "state": "CREATED"
}

Response body fields

Field Description Type

state

The current state of your A/B test. For more details about how to use states, see Lifecycle of an A/B test.

Possible values:
  • CREATED - Your test is created but isn't enabled or started.
  • ENABLING - Your test is transitioning from CREATED to ENABLED.
  • ENABLED - Your test configurations are deployed.
  • RUNNING - Your test has started.
  • STOPPING - Your test is transitioning from RUNNING to STOPPED.
  • STOPPED - Your test has ended.
  • DELETED - Your test is deleted.
  • FAILED - Your test configurations failed to deploy when trying to enable or start the test.

String

HTTP response codes

Status code Description

200

Returned skill test state.

401

The auth token is invalid/expired or doesn't have access to the resource.

403

The requested operation isn't allowed.

404

The requested resource isn't found.

429

Exceeds the permitted request limit. Throttling criteria includes total requests, per API, ClientId, and CustomerId.

500

Internal Server Error.

503

Service Unavailable.

Update A/B test state

Manage and update the state of an A/B test.

You can also call this API using the manage-experiment-state ASK CLI command.

Request

POST /v1/skills/{skillId}/experiments/{experimentId}/state
Host: api.amazonalexa.com
Accept: application/json
Authorization: Bearer {LWA Token}

Request header fields

Field Description Type Required?

skillId

The skill ID associated with your test.

Path

Yes

experimentId

The unique ID of the A/B test you want to update.

Path

Yes

Request body fields

Field Description Type Required?

targetState

The new state you want to update your A/B test to. For more details about how to use states, see Lifecycle of an A/B test.

Accepted values:
  • ENABLED - Deploy your test. You can only transition to this state if your test is in the CREATED state.
  • RUNNING - Start your test. You can only transition to this state if your test is in the CREATED or ENABLED state.
  • STOPPED - Stop your test. You can only transition to this state if your test is in the ENABLED or RUNNING state.

String

Yes

Response

HTTP/1.1 202 Accepted; Returns a URL to track the experiment state in 'Location' header.
Host: api.amazonalexa.com
X-Amzn-RequestId: {request-id}
Content-Type: application/json

Response header fields

Field Description Type

Location

The relative URL to get your A/B test details.

String

X-Amzn-RequestId

The unique identifier for the request. If a problem occurs, Amazon can use this value to troubleshoot the problem.

String

HTTP response codes

Status code Description

202

Accepted; Returns a URL to track the experiment state in 'Location' header.

400

Server can't process the request due to a client error.

401

The auth token is invalid/expired or doesn't have access to the resource.

403

The requested operation isn't allowed.

404

The requested resource isn't found.

409

Conflict with the resource state.

429

Exceeds the permitted request limit. Throttling criteria includes total requests, per API, ClientId, and CustomerId.

500

Internal Server Error.

503

Service Unavailable.

Get all metric snapshots

Gets a list of all metric snapshots associated with the experimentId of an A/B test. A metric snapshot represents all the metric data that's available for your test in a specific time range. This API doesn't include detailed data about each metric snapshot. To get detailed analytics about a test, use the Get metric snapshot data.

You can also call this API using the list-experiment-metric-snapshots ASK CLI command.

Request

/v1/skills/{skillId}/experiments/{experimentId}/metricSnapshots
Host: api.amazonalexa.com
Accept: application/json
Authorization: Bearer {LWA Token}

Request fields

Field Description Type Required?

skillId

The skill ID associated with your A/B test.

Path

Yes

experimentId

The unique ID of the A/B test details you want to retrieve.

Path

Yes

NextToken

A token used to retrieve the next page of results.

Path

No

MaxResults

Sets the maximum number of results returned in the response body. maxResults shouldn't exceed the upper limit. The response might contain fewer results than maxResults, but it never contains more.

Path

No

Response

HTTP/1.1 200 Returned test metric snapshots.
Host: api.amazonalexa.com
X-Amzn-RequestId: {request-id}
Content-Type: application/json

Response header fields

Field Description Type

X-Amzn-RequestId

The unique identifier for the request. If a problem occurs, Amazon can use this value to troubleshoot the problem.

String

Response body

The following example illustrates the structure of the response body.

  "metricSnapshots": [
    {
      "metricSnapshotId": "string",
      "startDate": "2021-04-20T15:24:31.563Z",
      "endDate": "2021-04-20T15:24:31.563Z"
    }
  ]
}

Response body fields

Field Description Type

metricSnapshots

Object that contains metric snapshot values for your skill.

Array

metricSnapshots.metricSnapshotId

Unique identifier for the metric snapshot of an A/B test.

String

metricSnapshots.startDate

Start date of the metric snapshot.

String

metricSnapshots.endDate

End date of the metric snapshot.

String

HTTP response codes

Status code Description

200

Returned test metric snapshots.

400

Server can't process the request due to a client error.

401

The auth token is invalid/expired or doesn't have access to the resource.

403

The requested operation isn't allowed.

404

The requested resource isn't found.

429

Exceeds the permitted request limit. Throttling criteria includes total requests, per API, ClientId, and CustomerId.

500

Internal Server Error.

503

Service Unavailable.

Get metric snapshot data

Lists detailed metric data and analytics associated with a single metricSnapshotId of an A/B test.

You can also call this API using the get-experiment-metric-snapshot ASK CLI command.

Request

/v1/skills/{skillId}/experiments/{experimentId}/metricSnapshots/{metricSnapshotId}
Host: api.amazonalexa.com
Accept: application/json
Authorization: Bearer {LWA Token}

Request fields

Field Description Type Required?

skillId

Skill ID associated with your A/B test.

Path

Yes

experimentId

Unique ID of the A/B test details you want to retrieve.

Path

Yes

metricSnapshotId

Unique ID of the A/B test metric snapshot you want to retrieve.

Path

Yes

Response

HTTP/1.1 200 Returned test metric data.
Host: api.amazonalexa.com
X-Amzn-RequestId: {request-id}
Content-Type: application/json

Response header fields

Field Description Type

X-Amzn-RequestId

The unique identifier for the request. If a problem occurs, Amazon can use this value to troubleshoot the problem.

String

Response body

The following example illustrates the structure of the response body.

{
  "status": "RELIABLE",
  "statusReason": "["TAA Alarm","Message"]",
  "metrics": [
    {
      "name": "Customer Friction",  
      "treatmentId": "C",
      "values": {
        "mean": 0,
        "percentDiff": 0,
        "confidenceIntervalLower": 0,
        "confidenceIntervalUpper": 0,
        "pValue": 0,
        "userCount": 0
      }
    }
  ]
}

Response body fields

Field Description Type

status

The reliability of the your A/B testing data. For example, if an error occurred during your tests, your data might not be accurate.

Possible values:

  • RELIABLE – The metric result snapshot for your selected date range is accurate.
  • UNRELIABLE* – The metric result snapshot for your selected date range might not be accurate.

String

statusReason

Explains why your metric result snapshot might be unreliable.

String

metrics

A list of metric snapshot values for your skill.

Array

metrics.name

A unique name that identifies the A/B test metric.

String

metrics.treatmentId

The A/B test type assigned to a group of customers associated with these metrics.

Possible values:

  • C - Control. The skill behavior you aren't testing. This group of customers continue to receive your current skill experience.
  • T1 - Treatment. The skill behavior you're testing. This group of customers receive your new experimental skill experience.

String

metrics.values

An object that contains values for your skill.

Object

metrics.values.mean

The average of each sample. Includes either the T1 or C group.

Double

metrics.values.percentDiff

The relative percent difference between the mean of the T1 group and the mean of the C group.

Double

metrics.values.confidenceIntervalLower

The lower limit number of the confidence interval range. The confidence interval measures the probability that the mean falls within a range.

Double

metrics.values.confidenceIntervalUpper

The upper limit number of the confidence interval range. The confidence interval measures the probability that the mean falls within a range.

Double

metrics.values.pValue

The probability that the difference between the two means (from T1 and C) is due to random sampling errors.

Double

metrics.values.userCount

The number of users in the test version.

Long

HTTP response codes

Status code Description

200

Returned test metric data.

400

Server can't process the request due to a client error.

401

The auth token is invalid/expired or doesn't have access to the resource.

403

The operation being requested isn't allowed.

404

The requested resource isn't found.

429

Exceeds the permitted request limit. Throttling criteria includes total requests, per API, ClientId, and CustomerId.

500

Internal Server Error.

503

Service Unavailable.