Cloud Storage

Overview

Clonit can push and pull snapshots to and from S3-compatible cloud storage, enabling you to share snapshots across machines, teams, and environments. Supported providers include:

• AWS S3
• Cloudflare R2
• Any S3-compatible storage endpoint

Setting Up Storage Profiles

A storage profile defines the credentials and bucket configuration for a cloud storage provider. You can create multiple profiles for different providers or environments.

AWS S3

clonit storage-profile add \
  --name my-s3 \
  --provider s3 \
  --bucket my-snapshots \
  --region us-east-1 \
  --access-key AKIA... \
  --secret-key wJalr...

Cloudflare R2

clonit storage-profile add \
  --name my-r2 \
  --provider r2 \
  --bucket my-snapshots \
  --account-id <cloudflare-account-id> \
  --access-key <r2-access-key> \
  --secret-key <r2-secret-key>

Additional Options

Flag	Description
--awscli-profile	Use credentials from a named AWS CLI profile instead of explicit keys
--endpoint	Custom endpoint URL for S3-compatible providers
--prefix	Key prefix for organizing objects within the bucket (e.g., snapshots/prod/)

Using an AWS CLI profile

If you already have AWS credentials configured via ~/.aws/credentials, you can reference them by profile name:

clonit storage-profile add \
  --name my-s3 \
  --provider s3 \
  --bucket my-snapshots \
  --region us-east-1 \
  --awscli-profile my-profile

Custom S3-compatible endpoint

For providers like MinIO or other S3-compatible services, specify the endpoint URL:

clonit storage-profile add \
  --name my-minio \
  --provider s3 \
  --bucket my-snapshots \
  --endpoint http://minio.local:9000 \
  --access-key minioadmin \
  --secret-key minioadmin

Object key prefix

Use a prefix to organize snapshots within the bucket:

clonit storage-profile add \
  --name my-s3 \
  --provider s3 \
  --bucket my-snapshots \
  --region us-east-1 \
  --access-key AKIA... \
  --secret-key wJalr... \
  --prefix snapshots/production/

Linking a Target to a Storage Profile

When creating a target, associate it with a storage profile:

clonit targets add \
  --name mydb \
  --src-url "postgres://..." \
  --storage-profile my-s3

Or update an existing target to add a storage profile:

clonit targets update mydb --storage-profile my-s3

Push and Pull Operations

Push (upload)

Upload a snapshot to cloud storage.

# Push the latest snapshot
clonit push mydb

# Push a specific snapshot by index
clonit push mydb 0

Pull (download)

Download a snapshot from cloud storage.

# Pull the latest snapshot
clonit pull mydb

# Pull a specific snapshot by index
clonit pull mydb 0

Options

Flag	Description
--no-verify	Skip SHA-256 checksum verification after transfer

Transfer Manager

Clonit includes a robust transfer manager that handles uploads and downloads reliably:

• Retry with exponential backoff and jitter – transient network errors are automatically retried with increasing delays and randomized jitter to avoid thundering herd problems.
• SHA-256 checksum verification – after each transfer, the checksum of the local and remote files is compared to ensure data integrity.
• Transient error detection – the transfer manager distinguishes between transient errors (network timeouts, throttling) and permanent errors (access denied, bucket not found) to avoid unnecessary retries.

Managing Storage Profiles

List all storage profiles

clonit storage-profile

Remove a storage profile

clonit storage-profile remove <name>

Example Workflow

A typical cloud storage workflow for sharing sanitized snapshots across a team:

# 1. Build and sanitize a snapshot
clonit build mydb
clonit sanitize mydb

# 2. Push the sanitized snapshot to cloud storage
clonit push mydb 1

# 3. On another machine, pull the snapshot
clonit pull mydb 0

# 4. Load the pulled snapshot into the local dev database
clonit load mydb