A/B Testing SMAPI APIs
Use the following SMAPI APIs to create, update, and manage A/B tests for your skills.
Authorization
Each API header request must contain an access token retrieved from Login with Amazon. For more details about obtaining an access token, see Get an Access Token for SMAPI.
Create A/B test
Creates a new A/B test for a skill.
You can also call this API using the create-experiment ASK CLI command.
Request
POST /v1/skills/{skillId}/experiments/
Host: api.amazonalexa.com
Accept: application/json
Authorization: Bearer {LWA Token}
Request fields
Field | Description | Type | Required? |
---|---|---|---|
|
The skill ID associated with your A/B test. |
Path |
Yes |
Request body structure
The following example illustrates the structure of the request body.
{
"experiment": {
"name": "my_first_experiment",
"description": "experiment_1",
"type": "ENDPOINT_BASED",
"plannedDuration": "P4W",
"exposurePercentage": 20,
"metricConfigurations": [
{
"name": "Customer Friction",
"metricTypes": [
"GUARDRAIL"
],
"expectedChange": "INCREASE"
}
]
}
}
Request body fields
Field | Description | Type | Required? |
---|---|---|---|
|
An object that contains A/B test settings for your skill. |
Object |
Yes |
|
A unique name for your A/B test. You can't use an identical name for two tests. However, you can use the same name if you delete one of the tests. Maximum 255 characters. |
String |
Yes |
|
A description of the hypothesis you set for your A/B test. Only used for reference purposes and doesn't impact the outcome. |
String |
Yes |
|
The type of A/B test you want to run for your skill.
|
String |
Yes |
|
The number of weeks you're running your A/B test. Only used for reference purposes and doesn't impact the actual test length. |
String |
Yes |
|
The percentage of unique customers that you're including in your A/B test. This number doesn't have to match your total users. |
Integer |
Yes |
|
A list of accepted A/B test metrics that you're testing in your experiment. |
Array |
Yes |
|
The unique name of an A/B test metric. Maximum 125 characters. |
String |
Yes |
|
A list of values used to detect changes in your A/B test. |
Array |
Yes |
|
The expected outcome of each A/B test metric. Only used for reference purposes and doesn't impact the actual test outcome. |
String |
Yes |
Response
HTTP/1.1 201 Experiment created. Returns the generated experiment identifier in 'Location' header.
Content-Type: application/json
Location: {url}
X-Amzn-RequestId: {request-id}
Response header fields
Field | Description | Type |
---|---|---|
|
The relative URL used to retrieve your A/B test details. |
String |
|
The unique identifier for the request. If a problem occurs, Amazon can use this value to troubleshoot the problem. |
String |
HTTP response codes
Status code | Description |
---|---|
201 |
Experiment created. Returns the generated experiment identifier in 'Location' header. |
400 |
Server can't process the request due to a client error. |
401 |
The auth token is invalid/expired or doesn't have access to the resource. |
403 |
The operation being requested isn't allowed. |
404 |
The requested resource isn't found. |
429 |
Exceeds the permitted request limit. Throttling criteria includes total requests, per API, ClientId, and CustomerId. |
500 |
Internal Server Error. |
503 |
Service Unavailable. |
Get A/B test
Retrieves detailed information about an A/B test.
You can also call this API using the get-experiment ASK CLI command.
Request
GET /v1/skills/{skillId}/experiments/{experimentId}
Host: api.amazonalexa.com
Accept: application/json
Authorization: Bearer {LWA Token}
Request fields
Field | Description | Type | Required? |
---|---|---|---|
|
The skill ID associated with your A/B test. |
Path |
Yes |
|
The unique ID of the A/B test you want to retrieve. |
Path |
Yes |
Response
HTTP/1.1 200 Returned skill experiment
Host: api.amazonalexa.com
X-Amzn-RequestId: {request-id}
Content-Type: application/json
Response header fields
Field | Description | Type |
---|---|---|
|
The unique identifier for the request. If a problem occurs, Amazon can use this value to troubleshoot the problem. |
String |
Response body fields
The following example illustrates the structure of the response body.
{
"experiment": {
"name": "my_first_experiment",
"description": "experiment_1",
"type": "ENDPOINT_BASED",
"plannedDuration": "P3W",
"exposurePercentage": 0,
"metricConfigurations": [
{
"name": "Customer Friction",
"metricTypes": [
"GUARDRAIL"
],
"expectedChange": "INCREASE"
}
],
"state": "CREATED",
"history": {
"creationTime": "2021-04-19T18:43:48.383Z",
"startTime": "2021-04-19T18:43:48.383Z",
"stoppedReason": "string"
},
"trigger": {
"scheduledEndTime": "2021-04-19T18:43:48.383Z"
}
}
}
Response body fields
Field | Description | Type |
---|---|---|
|
An object that contains A/B test settings for your skill. |
Object |
|
The unique name of your A/B test. |
String |
|
The description of the hypothesis you're trying to prove with your A/B test. Only used for reference purposes and doesn't impact the outcome. |
String |
|
The type of A/B test you set for your skill.
|
String |
|
The number of weeks you're running the A/B test. Only used for reference purposes and doesn't impact the actual test length. |
String |
|
The percentage of unique customers included in your A/B test. |
Integer |
|
A list of accepted A/B test metrics you selected for your test. |
Array |
|
The unique name of an A/B test metric. |
String |
|
A list of values used to detect changes in your A/B test. |
Array |
|
The expected outcome of each A/B test metric. Only used for reference purposes and doesn't impact the actual test outcome. |
String |
|
The creation date of your A/B test. |
String |
|
The start time of your A/B test. |
String |
|
The reason you stopped your A/B test. |
String |
|
An object that contains A/B test triggers for your skill. |
Object |
|
The expected end time of your A/B test. |
String |
HTTP response codes
Status code | Description |
---|---|
200 |
Returned skill test. |
401 |
The auth token is invalid/expired or doesn't have access to the resource. |
403 |
The requested operation isn't allowed. |
404 |
The requested resource isn't found. |
429 |
Exceeds the permitted request limit. Throttling criteria includes total requests, per API, ClientId, and CustomerId. |
500 |
Internal Server Error. |
503 |
Service Unavailable. |
Get list of A/B tests
Retrieves a list of tests associated with a skill ID. Returns both completed and in-progress tests.
You can also call this API using the list-experiment ASK CLI command.
Request
GET /v1/skills/{skillId}/experiments
Host: api.amazonalexa.com
Accept: application/json
Authorization: Bearer {LWA Token}
Request header fields
Field | Description | Type | Required? |
---|---|---|---|
|
The skill ID associated with your test. |
Path |
Yes |
|
A token used to retrieve the next page of results. |
Path |
No |
|
Sets the maximum number of results returned in the response body. |
Path |
No |
Response
HTTP/1.1 200 Returned skill experiments.
Host: api.amazonalexa.com
X-Amzn-RequestId: {request-id}
Content-Type: application/json
Response header fields
Field | Description | Type |
---|---|---|
|
The unique identifier for the request. If a problem occurs, Amazon can use this value to troubleshoot the problem. |
String |
Response body structure
The following example illustrates the structure of the response body.
{
"experiments": [
{
"experimentId": <experiment-id> ,
"name": "my_first-experiment",
"state": "CREATED",
"experimentHistory": {
"creationTime": "2021-04-29T18:20:50.671Z",
"startTime": "2021-04-29T18:20:50.671Z",
"stoppedReason": "string"
}
}
]
}
Response body fields
Field | Description | Type |
---|---|---|
|
An object that contains A/B test settings for your skill. |
Array |
|
The unique identifier of the A/B test associated with your skill. |
String |
|
The unique name of your A/B test. |
String |
|
The current state of your A/B test. For more details about how to use states, see Lifecycle of an A/B Test. |
String |
|
An object that contains historical properties of your A/B test. |
Object |
|
The creation date of your A/B test. |
String |
|
The start time of your A/B test. |
String |
|
The reason you stopped your A/B test. |
String |
HTTP response codes
Status code | Description |
---|---|
200 |
Returned a list of skill tests. |
401 |
The auth token is invalid/expired or doesn't have access to the resource. |
403 |
The requested operation isn't allowed. |
404 |
The requested resource isn't found. |
429 |
Exceeds the permitted request limit. Throttling criteria includes total requests, per API, ClientId, and CustomerId. |
500 |
Internal Server Error. |
503 |
Service Unavailable. |
Update A/B test
Update the settings of an A/B test. You can only call this API if your test is in the CREATED
state, meaning you can't update an active test. To update the exposure value of an active test, use Update exposure.
You can also call this API using the update-experiment ASK CLI command.
Request
POST /v1/skills/{skillId}/experiments/{experimentId}/properties
Host: api.amazonalexa.com
Accept: application/json
Authorization: Bearer {LWA Token}
Request header fields
Field | Description | Type | Required? |
---|---|---|---|
|
The skill ID associated with your A/B test. |
Path |
Yes |
|
The unique ID of the A/B test you want to update. |
Path |
Yes |
Request body structure
The following example illustrates the structure of the response body.
{
"experiment": {
"description": "experiment_1",
"plannedDuration": "P3W",
"exposurePercentage": 0,
"metricConfigurations": [
{
"name": "Customer Friction",
"metricTypes": [
"GUARDRAIL"
],
"expectedChange": "INCREASE"
}
]
}
}
Request body fields
Field | Description | Type |
---|---|---|
|
An object that contains A/B test settings for your skill. |
Object |
|
A description of the hypothesis you set for your A/B test. Only used for reference purposes and doesn't impact the outcome. |
String |
|
The number of weeks you're running the A/B test. Only used for reference purposes and doesn't impact the actual test length. |
String |
|
The percentage of unique customers that you're including in the A/B test. This number doesn't have to match your total users. |
Integer |
|
A list of accepted A/B test metrics you selected for your test. |
Array |
|
The unique name of a test metric. |
String |
|
A list of values used to detect changes in your A/B test. |
Array |
|
The expected outcome of each A/B test metric. Only used for reference purposes and doesn't impact the actual test outcome. |
String |
Response
HTTP/1.1 204 Success. No content.
Host: api.amazonalexa.com
X-Amzn-RequestId: {request-id}
Content-Type: application/json
Response header fields
Field | Description | Type |
---|---|---|
|
The unique identifier for the request. If a problem occurs, Amazon can use this value to troubleshoot the problem. |
String |
Response body example
The response returns your updated test values.
{
"experiment": {
"name": "Customer Friction",
"description": "experiment_1",
"type": "ENDPOINT_BASED",
"plannedDuration": "P3W",
"exposurePercentage": 0,
"metricConfigurations": [
{
"name": "Customer Friction",
"metricTypes": [
"GUARDRAIL"
],
"expectedChange": "INCREASE"
}
]
}
}
HTTP response codes
Status code | Description |
---|---|
204 |
Success. No content. |
400 |
Server can't process the request due to a client error. |
401 |
The auth token is invalid/expired or doesn't have access to the resource. |
403 |
The requested operation isn't allowed. |
404 |
The requested resource isn't found. |
409 |
The request couldn't be completed due to a conflict with the current state of the target resource. |
429 |
Exceeds the permitted request limit. Throttling criteria includes total requests, per API, ClientId, and CustomerId. |
500 |
Internal Server Error. |
503 |
Service Unavailable. |
Delete A/B test
Delete an A/B test.
You can only call this API if your test is in CREATED
state. If you need to delete your test after you start it (meaning your test is in the RUNNING
state), you must completely delete your skill with the Delete skill manifest REST API.
You can also call this API using the delete-experiment ASK CLI command.
Request
DELETE /v1/skills/{skillId}/experiments/{experimentId}
Host: api.amazonalexa.com
Accept: application/json
Authorization: Bearer {LWA Token}
Request fields
Field | Description | Type | Required? |
---|---|---|---|
|
The skill ID associated with your test. |
Path |
Yes |
|
The unique ID of the A/B test you want to delete. |
Path |
Yes |
Response header
HTTP/1.1 204 Success. No content.
Host: api.amazonalexa.com
X-Amzn-RequestId: {request-id}
Content-Type: application/json
Field | Description | Type |
---|---|---|
|
The unique identifier for the request. If a problem occurs, Amazon can use this value to troubleshoot the problem. |
String |
HTTP response codes
Status code | Description |
---|---|
204 |
Success. No content. |
401 |
The auth token is invalid/expired or doesn't have access to the resource. |
403 |
The requested operation isn't allowed. |
404 |
The requested resource isn't found. |
409 |
The request couldn't be completed due to a conflict with the current state of the target resource. |
429 |
Exceeds the permitted request limit. Throttling criteria includes total requests, per API, ClientId, and CustomerId. |
500 |
Internal Server Error. |
503 |
Service Unavailable. |
Update exposure
Updates the exposure value of an A/B test. You can only call this API if your test is in the CREATED
or RUNNING
state.
RUNNING
– You can only increase your exposure value. To decrease your exposure value after you have started your test, stop your test and create a new one.CREATED
– Increase or decrease your exposure value. If you want to update other settings, use Update A/B test
You can also call this API using the update-exposure ASK CLI command.
Request
POST /v1/skills/{skillId}/experiments/{experimentId}/exposurePercentage
Host: api.amazonalexa.com
Accept: application/json
Authorization: Bearer {LWA Token}
Request header fields
Field | Description | Type | Required? |
---|---|---|---|
|
The skill ID associated with your test. |
Path |
Yes |
|
The unique ID of the A/B test you want to update. |
Path |
Yes |
Request body fields
Field | Description | Type | Required? |
---|---|---|---|
|
The new exposure percentage you want to update your A/B test to. You can only increase the exposure percentage value of a running test, not decrease it. |
Integer |
Yes |
Response
HTTP/1.1 204 Success. No content.
Host: api.amazonalexa.com
X-Amzn-RequestId: {request-id}
Content-Type: application/json
Response header fields
Field | Description | Type |
---|---|---|
|
The unique identifier for the request. If a problem occurs, Amazon can use this value to troubleshoot the problem. |
String |
HTTP response codes
Status code | Description |
---|---|
204 |
Success. No content. |
400 |
Server can't process the request due to a client error. |
401 |
The auth token is invalid/expired or doesn't have access to the resource. |
403 |
The requested operation isn't allowed. |
404 |
The requested resource isn't found. |
409 |
Conflict with the resource state. |
429 |
Exceeds the permitted request limit. Throttling criteria includes total requests, per API, ClientId, and CustomerId. |
500 |
Internal Server Error. |
503 |
Service Unavailable. |
Get customer treatment override
Retrieves the current customer treatment override value of an A/B test.
You can also call this API using the get-customer-treatment-override ASK CLI command.
Request
GET /v1/skills/{skillId}/experiments/{experimentId}/treatmentOverrides/~current
Host: api.amazonalexa.com
Accept: application/json
Authorization: Bearer {LWA Token}
Request fields
Field | Description | Type | Required? |
---|---|---|---|
|
The skill ID associated with your A/B test. |
Path |
Yes |
|
The unique ID of the A/B test you want to update. |
Path |
Yes |
Response header fields
HTTP/1.1 200. Returned customer treatment override details.
Host: api.amazonalexa.com
X-Amzn-RequestId: {request-id}
Content-Type: application/json
Field | Description | Type |
---|---|---|
|
The unique identifier for the request. If a problem occurs, Amazon can use this value to troubleshoot the problem. |
String |
Response body fields
Field | Description | Type |
---|---|---|
|
The A/B testing group that you set your skill to.
|
String |
|
The current number of treatment overrides you set for your A/B test. |
Integer |
HTTP response codes
Status code | Description |
---|---|
200 |
Returned customer treatment override details. |
401 |
Server can't process the request due to a client error. |
401 |
The auth token is invalid/expired or doesn't have access to the resource. |
403 |
The requested operation isn't allowed. |
404 |
The requested resource isn't found. |
429 |
Exceeds the permitted request limit. Throttling criteria includes total requests, per API, ClientId, and CustomerId. |
500 |
Internal Server Error. |
503 |
Service Unavailable. |
Set customer treatment override
Sets a new customer treatment override value of an A/B test.
You can also call this API using the set-customer-treatment-override ASK CLI command.
Request
PUT /v1/skills/{skillId}/experiments/{experimentId}/treatmentOverrides/~current
Host: api.amazonalexa.com
Accept: application/json
Authorization: Bearer {LWA Token}
Request fields
Field | Description | Type | Required? |
---|---|---|---|
|
The skill ID associated with your A/B test. |
Path |
Yes |
|
The unique ID of the A/B test you want to update. |
Path |
Yes |
Request body parameters
Field | Description | Type | Required? |
---|---|---|---|
|
The A/B testing group that you want to set for your skill.
|
String |
Yes |
Response header fields
HTTP/1.1 204. Success. No content.
Host: api.amazonalexa.com
X-Amzn-RequestId: {request-id}
Content-Type: application/json
Field | Description | Type |
---|---|---|
|
The unique identifier for the request. If a problem occurs, Amazon can use this value to troubleshoot the problem. |
String |
Response body fields
Field | Description | Type |
---|---|---|
|
The A/B testing group that you set your skill to.
|
String |
HTTP response codes
Status code | Description |
---|---|
204 |
Success. No content. |
400 |
Server can't process the request due to a client error. |
401 |
The auth token is invalid/expired or doesn't have access to the resource. |
403 |
The requested operation isn't allowed. |
404 |
The requested resource isn't found. |
409 |
Conflict with the resource state. |
429 |
Exceeds the permitted request limit. Throttling criteria includes total requests, per API, ClientId, and CustomerId. |
500 |
Internal Server Error. |
503 |
Service Unavailable. |
Get A/B test state
Retrieves the current state of an A/B test.
You can also call this API using the get-experiment-state ASK CLI command.
Request
GET /v1/skills/{skillId}/experiments/{experimentId}/state
Host: api.amazonalexa.com
Accept: application/json
Authorization: Bearer {LWA Token}
Request fields
Field | Description | Type | Required? |
---|---|---|---|
|
The skill ID associated with your A/B test. |
Path |
Yes |
|
The unique ID of the A/B test you want to retrieve. |
Path |
Yes |
Response
HTTP/1.1 200. Returned skill experiment state.
Host: api.amazonalexa.com
X-Amzn-RequestId: {request-id}
Content-Type: application/json
Response header fields
Field | Description | Type |
---|---|---|
|
The unique identifier for the request. If a problem occurs, Amazon can use this value to troubleshoot the problem. |
String |
Response body
The following example illustrates the structure of the response body.
{
"state": "CREATED"
}
Response body fields
Field | Description | Type |
---|---|---|
|
The current state of your A/B test. For more details about how to use states, see Lifecycle of an A/B test. |
String |
HTTP response codes
Status code | Description |
---|---|
200 |
Returned skill test state. |
401 |
The auth token is invalid/expired or doesn't have access to the resource. |
403 |
The requested operation isn't allowed. |
404 |
The requested resource isn't found. |
429 |
Exceeds the permitted request limit. Throttling criteria includes total requests, per API, ClientId, and CustomerId. |
500 |
Internal Server Error. |
503 |
Service Unavailable. |
Update A/B test state
Manage and update the state of an A/B test.
You can also call this API using the manage-experiment-state ASK CLI command.
Request
POST /v1/skills/{skillId}/experiments/{experimentId}/state
Host: api.amazonalexa.com
Accept: application/json
Authorization: Bearer {LWA Token}
Request header fields
Field | Description | Type | Required? |
---|---|---|---|
|
The skill ID associated with your test. |
Path |
Yes |
|
The unique ID of the A/B test you want to update. |
Path |
Yes |
Request body fields
Field | Description | Type | Required? |
---|---|---|---|
|
The new state you want to update your A/B test to. For more details about how to use states, see Lifecycle of an A/B test. |
String |
Yes |
Response
HTTP/1.1 202 Accepted; Returns a URL to track the experiment state in 'Location' header.
Host: api.amazonalexa.com
X-Amzn-RequestId: {request-id}
Content-Type: application/json
Response header fields
Field | Description | Type |
---|---|---|
|
The relative URL to get your A/B test details. |
String |
|
The unique identifier for the request. If a problem occurs, Amazon can use this value to troubleshoot the problem. |
String |
HTTP response codes
Status code | Description |
---|---|
202 |
Accepted; Returns a URL to track the experiment state in 'Location' header. |
400 |
Server can't process the request due to a client error. |
401 |
The auth token is invalid/expired or doesn't have access to the resource. |
403 |
The requested operation isn't allowed. |
404 |
The requested resource isn't found. |
409 |
Conflict with the resource state. |
429 |
Exceeds the permitted request limit. Throttling criteria includes total requests, per API, ClientId, and CustomerId. |
500 |
Internal Server Error. |
503 |
Service Unavailable. |
Get all metric snapshots
Gets a list of all metric snapshots associated with the experimentId
of an A/B test. A metric snapshot represents all the metric data that's available for your test in a specific time range. This API doesn't include detailed data about each metric snapshot. To get detailed analytics about a test, use the Get metric snapshot data.
You can also call this API using the list-experiment-metric-snapshots ASK CLI command.
Request
/v1/skills/{skillId}/experiments/{experimentId}/metricSnapshots
Host: api.amazonalexa.com
Accept: application/json
Authorization: Bearer {LWA Token}
Request fields
Field | Description | Type | Required? |
---|---|---|---|
|
The skill ID associated with your A/B test. |
Path |
Yes |
|
The unique ID of the A/B test details you want to retrieve. |
Path |
Yes |
|
A token used to retrieve the next page of results. |
Path |
No |
|
Sets the maximum number of results returned in the response body. |
Path |
No |
Response
HTTP/1.1 200 Returned test metric snapshots.
Host: api.amazonalexa.com
X-Amzn-RequestId: {request-id}
Content-Type: application/json
Response header fields
Field | Description | Type |
---|---|---|
|
The unique identifier for the request. If a problem occurs, Amazon can use this value to troubleshoot the problem. |
String |
Response body
The following example illustrates the structure of the response body.
"metricSnapshots": [
{
"metricSnapshotId": "string",
"startDate": "2021-04-20T15:24:31.563Z",
"endDate": "2021-04-20T15:24:31.563Z"
}
]
}
Response body fields
Field | Description | Type |
---|---|---|
|
Object that contains metric snapshot values for your skill. |
Array |
|
Unique identifier for the metric snapshot of an A/B test. |
String |
|
Start date of the metric snapshot. |
String |
|
End date of the metric snapshot. |
String |
HTTP response codes
Status code | Description |
---|---|
200 |
Returned test metric snapshots. |
400 |
Server can't process the request due to a client error. |
401 |
The auth token is invalid/expired or doesn't have access to the resource. |
403 |
The requested operation isn't allowed. |
404 |
The requested resource isn't found. |
429 |
Exceeds the permitted request limit. Throttling criteria includes total requests, per API, ClientId, and CustomerId. |
500 |
Internal Server Error. |
503 |
Service Unavailable. |
Get metric snapshot data
Lists detailed metric data and analytics associated with a single metricSnapshotId
of an A/B test.
You can also call this API using the get-experiment-metric-snapshot ASK CLI command.
Request
/v1/skills/{skillId}/experiments/{experimentId}/metricSnapshots/{metricSnapshotId}
Host: api.amazonalexa.com
Accept: application/json
Authorization: Bearer {LWA Token}
Request fields
Field | Description | Type | Required? |
---|---|---|---|
|
Skill ID associated with your A/B test. |
Path |
Yes |
|
Unique ID of the A/B test details you want to retrieve. |
Path |
Yes |
|
Unique ID of the A/B test metric snapshot you want to retrieve. |
Path |
Yes |
Response
HTTP/1.1 200 Returned test metric data.
Host: api.amazonalexa.com
X-Amzn-RequestId: {request-id}
Content-Type: application/json
Response header fields
Field | Description | Type |
---|---|---|
|
The unique identifier for the request. If a problem occurs, Amazon can use this value to troubleshoot the problem. |
String |
Response body
The following example illustrates the structure of the response body.
{
"status": "RELIABLE",
"statusReason": "["TAA Alarm","Message"]",
"metrics": [
{
"name": "Customer Friction",
"treatmentId": "C",
"values": {
"mean": 0,
"percentDiff": 0,
"confidenceIntervalLower": 0,
"confidenceIntervalUpper": 0,
"pValue": 0,
"userCount": 0
}
}
]
}
Response body fields
Field | Description | Type |
---|---|---|
|
The reliability of the your A/B testing data. For example, if an error occurred during your tests, your data might not be accurate.
|
String |
|
Explains why your metric result snapshot might be unreliable. |
String |
|
A list of metric snapshot values for your skill. |
Array |
|
A unique name that identifies the A/B test metric. |
String |
|
The A/B test type assigned to a group of customers associated with these metrics.
|
String |
|
An object that contains values for your skill. |
Object |
|
The average of each sample. Includes either the T1 or C group. |
Double |
|
The relative percent difference between the mean of the T1 group and the mean of the C group. |
Double |
|
The lower limit number of the confidence interval range. The confidence interval measures the probability that the mean falls within a range. |
Double |
|
The upper limit number of the confidence interval range. The confidence interval measures the probability that the mean falls within a range. |
Double |
|
The probability that the difference between the two means (from T1 and C) is due to random sampling errors. |
Double |
|
The number of users in the test version. |
Long |
HTTP response codes
Status code | Description |
---|---|
200 |
Returned test metric data. |
400 |
Server can't process the request due to a client error. |
401 |
The auth token is invalid/expired or doesn't have access to the resource. |
403 |
The operation being requested isn't allowed. |
404 |
The requested resource isn't found. |
429 |
Exceeds the permitted request limit. Throttling criteria includes total requests, per API, ClientId, and CustomerId. |
500 |
Internal Server Error. |
503 |
Service Unavailable. |
Related topics
Last updated: Aug 19, 2021