Step 4: Upload Your Catalog File to Amazon

After you've verified that your catalog file is valid and set up your AWS account, you're ready to upload your catalog file to the S3 bucket that Amazon set up for your catalog. You can upload your catalog using the AWS command line interface.

• Upload Your Catalog File
• Example Workflow
• Automating the Validation and Upload Process
• Additional Resources for AWS
• Catalog Uploading FAQ
• Troubleshooting
• Next Steps

Upload Your Catalog File

You upload your catalog files to a catalogs folder inside the S3 bucket that was set up for you. Use any S3 tool to upload your catalog file to this bucket and folder. (If you haven't yet set up your AWS account, see Step 3: Set Up Your AWS Account for Catalog Ingestion first.)

To upload a catalog file to an S3 bucket:

1. Type the following command, substituting your catalog file name for <catalog_file_name.xml> and S3 bucket name for <s3_bucket_name>. Note that the --acl bucket-owner-full-control option is required so that Amazon will be able to read the file that you upload upload, and thus ingest your catalog.

   $ aws s3api put-object --body <catalog_file_name.xml> --bucket <s3_bucket_name> --key catalogs/catalog.xml --region us-east-1 --acl bucket-owner-full-control
   

   Note: If you copy-and-paste this command into a terminal window, make sure that the -- characters paste as double-dashes and not as an n-dash. Also, make sure you set the --acl parameter with bucket-owner-full-control; otherwise, the upload will fail.

   This command successfully uploads files up to 2GB in size. If successful, the AWS CLI will display a set of tags for the VersionID, ETag, and Expiration for the file.

2. Verify that you see your catalog file in your S3 bucket. Type the following command to list all files in your S3 bucket:

   $ aws s3 ls s3://<s3_bucket_name>/catalogs/ --region us-east-1
   

   (Replace <s3_bucket_name> with your bucket name.)

   You can upload multiple catalogs to the catalog bucket. However, Amazon uses only the most recently uploaded catalog, regardless of its name. If you find a problem with your catalog, or need to update the data in the catalog file, just upload a new file.

Example Workflow

The following example command uploads a catalog file named my-catalog.xml to an S3 bucket named cdf-bucket:

$ aws s3api put-object --body my-catalog.xml --bucket cdf-bucket --key catalogs/catalog.xml --region us-east-1 --acl bucket-owner-full-control

This command returns the following sample output:

{
    "VersionId": "m_QwgKPy9RJZsWperU_LEULD1waJE2He",
    "ETag": "\"e8c38d5258ad1f3b241ae2ce347e40bc\"",
    "Expiration": "expiry-date=\"Fri, 06 Jan 2017 00:00:00 GMT\", rule-id=\"Rule for the Entire Bucket\""
}

To verify that the my-catalog.xml file was successfully uploaded, use the following command:

$ aws s3 ls s3://cdf-bucket/catalogs/ --region us-east-1

This command returns a list of all catalog files currently found in the bucket:

2015-12-07 15:02:17      10236 my-catalog.xml
2015-12-01 15:10:28        166 other-catalog.xml

Automating the Validation and Upload Process

You can usually write a script that will automate the validation and upload process. Amazon strongly recommends validating your catalog file before attempting to upload. If you create a script, please include a validation step as part of your upload process.

Additional Resources for AWS

These links provide further information on AWS and Amazon S3.

• Amazon Web Services: All AWS products
• Amazon Simple Storage Service (S3): Secure object storage
• Amazon Identity and Access Management (IAM): Define users and roles for your AWS account
• Start Developing with AWS: Documentation, SDKs, and sample apps for various languages and platforms
• AWS Command Line Interface (CLI): Simple access to AWS services
• Tools for Amazon Web Services: All SDKs and sample code
• AWS SDK For Java: Integrate AWS with your Java apps
• Amazon S3 Client: Java class for accessing Amazon S3
• Upload an Object Using the AWS SDK for Java: Sample code for uploading your data to Amazon S3

Catalog Uploading FAQ

Q: What's the maximum frequency at which we can publish a catalog file to our AWS S3 bucket?

A: Amazon does not have limits on publishing frequency; we adjust our ingestion pipeline as appropriate for the volume of uploaded files.

Q: After uploading my catalog file to my S3 bucket, how soon can I expect it to be ingested?

A: Amazon retrieves new catalog file uploads from partner S3 buckets every four hours. If your catalog file can be successfully ingested, it will be ingested at this time. If your catalog file fails ingestion, you'll need to fix your file, re-upload, and wait for the next four hour pickup window. If your catalog update must be faster than this, please discuss this with your Amazon representative. Note that this four-hour pickup window is subject to change.

Q: What is the delay between a catalog file upload and the content being available on the Fire TV device?

A: Typically, content will be available to customers within 2-4 hours after the catalog is ingested; however, due to possible caching and other conditions, content may take as long as 72 hours to become available for some apps.

Q: Do we need to upload a full catalog file every time we update? Can we upload a catalog file that only includes new and updated data?

A: You need to provide the full catalog file with each upload. Amazon uses the full file to calculate what needs to be deleted. Additionally, having the full file helps ensure that Amazon's catalog will not diverge from yours over time.

Q: What if we send several catalog updates within quick succession of each other? What happens if we produce catalog updates faster than Amazon can process them?

A: If you upload your catalog files faster than Amazon can process them, Amazon will just ignore the older copies and use the most recent file version. In other words, you cannot overload Amazon's system by uploading an abundance of catalogs to the S3 system.

Q: Do I need to rename my catalog file every time that I upload a new version of it to S3? How does Amazon determine which catalog file to use if my S3 bucket contains multiple files?

A: Amazon uses timestamps to determine the newest version of your catalog file and will always ingest and use the most recent version.

Troubleshooting

The following are common errors when uploading catalog files.

"AccessDenied" response when when calling the PutObject operation.

Check two things:

• Make sure that when you upload your catalog file via the aws s3api put-object command, you include the --acl bucket-owner-full-control parameter.
• Make sure you set up your IAM user S3 bucket permissions properly. Review the section Grant IAM Users Access to Your Amazon S3 Bucket (this section appears in Step D in Step 3: Set Up Your AWS Account for Catalog Ingestion).

"InvalidAccessKeyId" response when calling the PutObject operation.

The AWS Access Key Id is no longer be valid. Check your IAM user for new access keys. Review the section Grant IAM Users Access to Your Amazon S3 Bucket (this section appears in Step D in Step 3: Set Up Your AWS Account for Catalog Ingestion).

Next Steps

Now that you've uploaded your catalog file, go to the next step: Step 5: Verify Your Uploaded Catalog File.