Sippy

Sippy is a data migration service that allows you to copy data from other cloud providers to R2 as the data is requested, without paying unnecessary cloud egress fees typically associated with moving large amounts of data.

Migration-specific egress fees are reduced by leveraging requests within the flow of your application where you would already be paying egress fees to simultaneously copy objects to R2.

​​ How it works

When enabled for an R2 bucket, Sippy implements the following migration strategy across Workers, S3 API, and public buckets:

• When an object is requested, it is served from your R2 bucket if it is found.
• If the object is not found in R2, the object will simultaneously be returned from your source storage bucket and copied to R2.
• All other operations, including put and delete, continue to work as usual.

​​ When is Sippy useful?

Using Sippy as part of your migration strategy can be a good choice when:

• You want to start migrating your data, but you want to avoid paying upfront egress fees to facilitate the migration of your data all at once.
• You want to experiment by serving frequently accessed objects from R2 to eliminate egress fees, without investing time in data migration.
• You have frequently changing data and are looking to conduct a migration while avoiding downtime. Sippy can be used to serve requests while Super Slurper can be used to migrate your remaining data.

If you are looking to migrate all of your data from an existing cloud provider to R2 at one time, we recommend using Super Slurper.

​​ Get started with Sippy

​​ Enable Sippy on your R2 bucket

During the beta, Sippy can only be enabled on your R2 bucket by using the API. Before getting started, you’ll need:

• An existing R2 bucket. If you don’t already have one, refer to Create buckets.
• AWS credentials in the form of Access Key ID and Secret Access Key with read and list permissions.
• Cloudflare API token, R2 Access Key ID and Secret Access Key with read and write permissions. For more information, refer to Authentication.

The example below shows how to enable Sippy for an R2 bucket with cURL. For information about getting started with the Cloudflare API, refer to Make API calls.

curl -X PUT https://api.cloudflare.com/client/v4/accounts/{account_id}/r2/buckets/{bucket_name}/sippy \
--header "Authorization: Bearer <API_TOKEN>" \
--data '{"provider": "AWS", "bucket": "<AWS_BUCKET_NAME>", "zone": "<AWS_REGION>","key_id": "<AWS_ACCESS_KEY_ID>", "access_key":"<AWS_SECRET_ACCESS_KEY>", "r2_key_id": "<R2_ACCESS_KEY_ID>", "r2_access_key": "<R2_SECRET_ACCESS_KEY>"}'

​​ Parameters

• provider string - The provider of your source object storage bucket. Currently, AWS is the only supported option.
• bucket string - The name of your AWS S3 bucket.
• zone (optional) string - The AWS region where your S3 bucket is located. For example: us-west-2.
• key_id string - Your AWS Access Key ID. Requires read and list access.
• access_key string - Your AWS Secret Access Key. Requires read and list access.
• r2_key_id string - Your R2 Access Key ID. Requires read and write access.
• r2_access_key string - Your R2 Secret Access Key. Requires read and write access.

​​ Disable Sippy on your R2 bucket

The example below shows how to disable Sippy for an R2 bucket with cURL.

curl -X DELETE https://api.cloudflare.com/client/v4/accounts/{account_id}/r2/buckets/{bucket_name}/sippy \
--header "Authorization: Bearer <API_TOKEN>"

​​ Supported cloud storage providers

We currently support copying data from the following cloud object storage providers to R2:

• Amazon S3

​​ R2 API interactions

When Sippy is enabled, it changes the behavior of certain actions on your R2 bucket across Workers, S3 API, and public buckets.

Actions not listed above have no change in behavior. For more information, refer to Workers API reference or S3 API compatibility.

​​ Create Amazon S3 credentials

To copy objects from Amazon S3, Sippy requires access permissions to your bucket. While you can use any AWS Identity and Access Management (IAM) user credentials with the correct permissions, Cloudflare recommends you create a user with a narrow set of permissions.

To create credentials with the correct permissions:

1. Log in to your AWS IAM account.
2. Create a policy with the following format and replace <BUCKET_NAME> with the bucket you want to grant access to:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:Get*",
                "s3:List*"
            ],
            "Resource": [
                "arn:aws:s3:::<BUCKET_NAME>",
                "arn:aws:s3:::<BUCKET_NAME>/*"
            ]
        }
    ]
}

3. Create a new user and attach the created policy to that user.

You can now use both the Access Key ID and Secret Access Key when enabling Sippy.

​​ Caveats

​​ ETags

While R2’s ETag generation is compatible with S3’s during the regular course of operations, ETags are not guaranteed to be equal when an object is migrated using Sippy. Sippy makes autonomous decisions about the operations it uses when migrating objects to optimize for performance and network usage. It may choose to migrate an object in multiple parts, which affects ETag calculation.

For example, a 320 MiB object originally uploaded to S3 using a single PutObject operation might be migrated to R2 via multipart operations. In this case, its ETag on R2 will not be the same as its ETag on S3. Similarly, an object originally uploaded to S3 using multipart operations might also have a different ETag on R2 if the part sizes Sippy chooses for its migration differ from the part sizes this object was originally uploaded with.

Relying on matching ETags before and after the migration is therefore discouraged.