Striped Buckets
Combine several object storage buckets into one logical bucket that stripes writes across them to scale write throughput past a single bucket's limits.
A striped bucket combines several object storage buckets into a single logical bucket by striping writes across them: each object is placed on exactly one sub-bucket, chosen deterministically by hashing the object’s key. It exists to scale write throughput past the request-rate ceiling of a single bucket — spreading writes across N sub-buckets multiplies the headroom for ingestion and compaction.
Striped buckets are configured through the same -bucketURL flag (WARPSTREAM_BUCKET_URL) as a normal single bucket — you just pass a warpstream_stripe:// URL instead of a plain s3:// (or other provider) URL.
When to use it
Section titled “When to use it”Reach for a striped bucket when a single bucket is hitting its object storage request-rate limits (for example, sustained 503 SlowDown throttling on S3) and the workload needs more write throughput than one bucket can provide. If instead you need durability across regions, use a Multi Bucket.
URL format
Section titled “URL format”warpstream_stripe://$BUCKET_1_URL<>$BUCKET_2_URL<>...<>$BUCKET_N_URLThe sub-buckets are separated by <> (2 to 32 sub-buckets), and each sub-URL is constructed exactly as described on the Object Storage Configuration page.
For example, striping across three buckets:
-bucketURL "warpstream_stripe://s3://bucket-a?region=us-east-1<>s3://bucket-b?region=us-east-1<>s3://bucket-c?region=us-east-1"How it works
Section titled “How it works”- Placement is deterministic: each object is hashed to one sub-bucket, so writers and readers independently agree on where an object lives — nothing extra is stored to track it.
- Reads go to the hashed sub-bucket and cost the same as a single bucket (one round-trip).
- Write resilience: if a sub-bucket’s circuit breaker opens (it is throttling or failing), new writes are steered to a healthy sub-bucket so ingestion keeps making progress.
Multi buckets vs. striped buckets
Section titled “Multi buckets vs. striped buckets”warpstream_multi:// |
warpstream_stripe:// |
|
|---|---|---|
| Each object is… | replicated to a quorum of buckets | written to exactly one bucket |
| Optimizes for | durability / availability | write throughput |
| Survives a bucket/region loss | yes (no data loss) | no (that bucket’s objects become unreadable) |
| Relative storage cost | ~N× | ~1× |
Requirements and constraints
Section titled “Requirements and constraints”- Every WarpStream Agent must be able to read and write all of the sub-buckets.
- Each sub-bucket needs the same permissions and bucket configuration (no object retention policy, versioning, or soft deletion) as a normal WarpStream bucket. Using the same provider and type for every sub-bucket is recommended.
- Changing the set of sub-buckets is forward-only. Existing files stay in whichever sub-bucket they were originally written to, and are always read from there. Switching a single bucket to a striped bucket (or adding sub-buckets) is safe: new files are striped across the new set while old files continue to be read from where they live.
Migrating
Section titled “Migrating”Switching a cluster to or from a striped bucket is nothing special — it works exactly like migrating between object storage buckets: point -bucketURL at the new destination (a warpstream_stripe:// URL, or a plain single-bucket URL when migrating off) and keep the previous bucket(s) in -additionalBackgroundTasksBucketURLs until the old data has drained. The same applies when retiring a single sub-bucket from a stripe: add the retired bucket (or the previous full stripe URL) to -additionalBackgroundTasksBucketURLs so its files are still read and cleaned up while it drains.