Change Log

Subscribe to the RSS Feed

https://console.warpstream.com/agent-changelog.rss

Note

By default all Agent upgrades are designed to be seamless and backwards compatible. Occasionally we have to make make breaking changes which we document in the migrations sub-page where you can see if any breaking changes were made between the version you’re running and the version you’re upgrading to.

Change Log

Release v857

November 11, 2026

• Change how Tableflow reads Parquet columns to support Delta column mapping and column renames in the future.
• For clients that don’t re-bootstrap by default (e.g. Java clients older than 4.0), Metadata responses now list every agent in the client’s Agent Group that matches the client’s targeted role (ws_pt) as a broker, regardless of the Agent’s availability zone. Partitions are still only assigned to Agents in the client’s zone. This makes those clients more resilient to situations where the agents in the client’s zone are unreachable.
• Bump gocloud.dev dependency to v0.47.0 to pick up an error code classification fix.
• Bump golang.org/x/net to v0.62.0 to address CVE-2026-78661.
• Reduce the CPU usage of idle Agents by waking up the internal batchers’ background timers much less often when there is nothing to flush.

Release v856

November 9, 2026

• Improve compaction to enforce retention.ms on low-throughput partitions.
• Fix Agents logging error loading superusers (unknown_sasl_username for the virtual cluster ID) every 30 seconds on clusters with ACLs enabled. This was harmless and not causing any trouble but it will go away in this release.
• Bump agent base image (cgr.dev/chainguard/wolfi-base) to fix CVE-2026-91403 (ca-certificates).
• Fix managed data pipelines failing to connect to Agents with Kafka TLS enabled when WARPSTREAM_DISCOVERY_KAFKA_HOSTNAME_OVERRIDE is set. Pipelines that verify the Agents’ TLS certificate now connect through the advertised hostname instead of the Agents’ internal addresses, which the certificate usually doesn’t cover. Pipelines that set tls.skip_cert_verify: true still connect directly.
• Fix managed data pipelines failing to connect when the Agent running them has its Kafka server disabled. kafka_franz_warpstream blocks without seed_brokers now use up to three Agents from service discovery as seed brokers instead of localhost, preferring Agents in the same availability zone. Pipelines that verify the Agents’ TLS certificate now also use the Agents’ advertised hostname as seed brokers instead of localhost, which the certificate usually doesn’t cover.
• Managed data pipelines now enable TLS automatically on kafka_franz_warpstream blocks that don’t configure tls or seed_brokers when the Agents serving Kafka have TLS enabled, instead of failing with a TLS mismatch, and restart them when the Agents’ TLS setting changes. Pipelines whose Agents use a certificate from a private CA still need to set tls.root_cas, and Agents that require mTLS still need tls.client_certs.
• Fix managed data pipelines with a cluster_concurrency_target running fewer instances than requested when the cluster uses pipeline groups. Agents now divide the target across the Agents in their own pipeline group instead of all Agents with the pipelines role.
• Lightning topic produce requests are no longer delayed by the control plane commit of classic topic data flushed by the same Agent. Previously a lightning topic file that was already in object storage waited for any earlier classic file’s PutFile to complete before the produce request was acknowledged, adding tens of milliseconds of latency on Agents serving both topic types. A failed classic file commit also no longer fails the lightning topic produce requests queued behind it.
• Fix Agent error logs for failed Produce requests always showing "Acks":0 and "TimeoutMillis":15000 in the logged request, regardless of what the client sent. The logged request now includes the client’s actual Acks and TimeoutMillis values.
• In ripcord mode, while the control plane is unavailable, agents stop routing clients to an agent that crashed without shutting down cleanly within about 90 seconds, instead of up to an hour.
• In ripcord mode, while the control plane is unavailable, agent discovery keeps working when another agent shuts down mid-refresh or one agent’s discovery record can’t be read, instead of failing that whole refresh.

Release v855

October 30, 2026

• Tableflow:
  • Fix issue with DLQ settings not being honored in schema_registry mode. From now on, records whose schema is not in the registry will honor the dlq_mode setting. Transient errors (like the registry being unavailable) will still stop ingestion (to be retried).
• Add a “Long empty fetch despite consumer lag” diagnostic for consumers whose fetches repeatedly wait the full fetch.max.wait.ms without data while leaving out partitions they are still lagging on.
• Fix query engine panic when grouping by multiple fields where one of them is missing or is a constant.
• Fix managed data pipelines using kafka_franz_warpstream failing with unsupported_mechanism: PLAIN on clusters that restrict -enabledSASLMechanisms to SCRAM-SHA-512. The built-in pipeline and query engine credentials can now authenticate with SCRAM-SHA-512, and pipelines automatically fall back to it when PLAIN is not enabled.
• Managed data pipelines now always authenticate kafka_franz_warpstream blocks with the pipeline’s built-in SASL credentials, instead of only when ACLs or -requireSASLAuthentication are enabled. Blocks with an explicit sasl config or with TLS client certificates configured (for mTLS authentication) are left unchanged. Pipelines can authenticate with SCRAM-SHA-512 even when -enabledSASLMechanisms excludes it (built-in internal credentials only).
• Fix Events Explorer searches across all event types reporting “No events data found” when the client_metrics topic has not been created yet.
• Close each batched file-cache read’s response stream once it is read, instead of holding its request context and timer until the request times out.

Release v854

October 28, 2026

• Bump github.com/go-jose/go-jose/v4 to v4.1.6 to fix CVE-2026-116587.
• Produce requests whose record batch declares more records than it contains (truncated or malformed records) are now rejected with the non-retriable INVALID_RECORD error instead of the retriable KAFKA_STORAGE_ERROR, so clients stop retrying corrupt batches.
• Fix a very rare bug that could mean a tombstone is deleted because delete.retention.ms has passed although a previous record still existed for that key.

Release v853

October 27, 2026

• Fix modern consumer groups using regex subscriptions failing to update assignments when matching topics are created or deleted.
• Batch ConsumerGroupHeartbeat (KIP-848) requests to the control plane once the cluster’s RSM supports it.
• Orbit Auto Migration now supports multiple source clusters. Contact the team if you plan to use it.
• Agents now spread newly written files across many more object storage key prefixes by default. In large clusters that write many files per second, this should reduce rate limiting from object storage providers (for example, S3 503 SlowDown errors). Existing data and the file naming scheme are unchanged. To go back to the previous behavior on an Agent, set WARPSTREAM_USE_RANGE_FILE_ID_ALLOCATOR=false and restart it.
• Fix unbounded memory growth in local mode, which caused long-running local agents to OOM even when idle.
• Publish FIPS 140-3 Linux Docker images and release archives tagged <version>-fips (e.g. v850-fips). They’re built against Go’s CMVP-certified cryptographic module, with FIPS mode on. In FIPS mode, TLS is restricted to TLS 1.2/1.3 with FIPS-approved cipher suites, curves and signatures: ChaCha20-Poly1305, plain X25519 key exchange, RSA keys under 2048 bits and SHA-1 signatures are rejected, so clients and certificates that rely on them can’t connect to a -fips agent. latest and latest-stable keep pointing at the regular build.
• Ship CycloneDX 1.7 (sbom.cdx.json) and SPDX 2.3 (sbom.spdx.json) SBOMs of the agent binary in the Docker images (at /sbom.cdx.json and /sbom.spdx.json) and release archives. They list every linked Go module with its version, checksum and license, plus the Go standard library, the embedded DuckDB CLI and the statically linked C libraries.
• Bump github.com/andybalholm/brotli from v1.2.3 to v1.2.7 to pick up the fix for CVE-2026-111497 in its Brotli decoder.

Release v852

October 21, 2026

• Tableflow:
  • Tune blob store retries so that compaction and deletion jobs are more resilient to intermittent failures.
• Schema Registry: Fix possible race when compiling Protobuf schemas with references.
• Schema Validation: Add eviction policy with a TTL of 1 hour for the schemas cached by the agent when validating.
• Refresh missing SASL credentials more frequently so newly created credentials become available sooner.
• Log expected file-cache actor deadline expirations with less than 500ms of caller budget at info (debug below 50ms) with clearer context, while keeping actionable failures at error.
• Log deadscanner delete failures at error level so they are forwarded to the control plane.
• Bump build to go 1.27.3 to fix CVE-2026-111943.
• Replace the rand.Rand created on every Fetch request with lock-free top-level rand functions.
• Reuse zstd compression contexts across calls instead of creating a new compressor for every Kafka batch, reducing allocations on produce, compaction, and fetch paths.
• Better spread small (batched) file-cache reads across agents.

Release v851

October 15, 2026

• Schema Registry: Fix a formatting issue affecting Protobuf schemas containing options with message literals so that they are formatted like the Confluent Schema Registry. Re-registering such a schema may create a new version and schema ID for it.
• Schema Registry: Allow using types and options from confluent/meta, confluent/variant, confluent/decimal without declaring the import in Protobuf schemas.
• Finalize GCS Rapid Storage objects synchronously by default, preventing multi-bucket reads from selecting truncated replicas. Stream batcher writes use marker-protected asynchronous finalization to preserve produce latency.
• Add -httpsProxyCACertFile (WARPSTREAM_HTTPS_PROXY_CA_CERT_FILE) to the metrics command, so the dedicated metrics pod can trust a private CA like the agent.

Release v850

October 14, 2026

• Tableflow:
  • Start sending row group offsets to populate the Iceberg manifest split_offsets field.
• Fix asymmetric signing algorithm support for SASL OAUTHBEARER.
• Bump build to go 1.27.2 and golang.org/x/net to v0.60.0 to fix CVE-2026-94440, CVE-2026-78660, CVE-2026-56866, CVE-2026-56857, CVE-2026-97031, CVE-2026-94439, CVE-2026-78667, CVE-2026-78659, CVE-2026-78663, and CVE-2026-78669.
• Fix a query engine panic and incorrect filter results when a query combines like or not like on a missing or null field with and, or, or not.
• Fix like and not like patterns written with the documented % wildcard never matching. * and ? still work, and [, ], {, and } now match literally instead of failing some queries.
• Fix a query engine panic or wrong time bucket when bin() is applied to a missing or null field.

Release v849

October 9, 2026

• Add tls-disable-hostname-verification flag to all ws cli subcommands that allows to disable hostname verification.
• Schema Registry: Fix a formatting issue affecting Protobuf schemas containing both a reference and some oneof fields so their fields are sorted like the Confluent Schema Registry. Re-registering such a schema may create a new version and schema ID for it.
• Schema Registry: Add native support for Confluent types confluent/meta, confluent/variant, confluent/decimal for protobuf schemas.
• Fix the deadscanner reporting long-lived file rewrites as successful when finishing the upload failed, which could leave the file’s modification time un-refreshed.

Release v848

October 8, 2026

• Bump franz-go package from v1.21.7 to v1.22.1.
• Add support of KIP-1242 to have proper detection and handling of misrouted connections. WarpStream agents only validate the “ClusterID” part of the protocol so everything keeps working seamlessly behind a load balancer.
• The agent metrics mode calls the prometheus endpoint exposed by our control plane instead of hopping through our console public API first. This should improve the latency and reliability of this mode.
• Upgrade Bento to v1.21.2. It fixes https://security.snyk.io/vuln/SNYK-GOLANG-GITHUBCOMSNOWFLAKEDBGOSNOWFLAKE-19638984 as it is transitively pulling github.com/snowflakedb/gosnowflake to version 2.2.0.
• Accept Produce API requests up to v12 (was v9 before)
• Accept TxnOffsetCommit requests up to v5 (was v3 before)
• Fix managed data pipelines using kafka_franz_warpstream failing with TOPIC_AUTHORIZATION_FAILED on clusters where ACLs were enabled without ever configuring ACL shadowing. The pipelines were not authenticating with their built-in credentials, so ACLs were evaluated against the anonymous principal.
• Fix the Agent creating far more S3 Express One Zone sessions (CreateSession calls) than necessary, which drove up AWS KMS costs for directory buckets encrypted with SSE-KMS. The AWS SDK’s default session provider did not deduplicate concurrent session creation and canceled in-flight session refreshes whenever the request that triggered them finished; the Agent now uses its own provider that creates at most one session per bucket at a time and refreshes it independently of individual requests.
• Tableflow: Ingestion can dictionary-encode low-cardinality string columns, choosing columns per file by sampling each column’s values. Compaction and deletion keep a column dictionary-encoded when all of their input files did. Disabled by default and enabled by WarpStream per cluster.
• Fix a query engine panic that could fail some limited queries whose fields contain a literal, a boolean expression, or an alias reusing a field name.
• Fix ACL changes sometimes not applying to a principal’s first request on an Agent it had not used for a while.
• Fix aborted transaction offload jobs reporting success, and committing the offloaded file, even when finishing the upload of that file to object storage failed.

Release v847

September 29, 2026

• Allow Kafka clients to downgrade from unsupported ApiVersions request versions instead of closing the connection.
• Enable schema validation for Protobuf records with the AWS Glue schema registry.
• Fix a false-positive inter_agent_connectivity diagnostic when agents advertise IPv6 addresses (auto-ip6).
• Fix managed data pipelines (and any client using ws_node_host=true) failing to connect to Agents deployed behind a load balancer that listens on a different port than the Agents. When WARPSTREAM_DISCOVERY_KAFKA_PORT_OVERRIDE is set, the Agent now also reports its actual Kafka listen port, and clients that bypass WARPSTREAM_DISCOVERY_KAFKA_HOSTNAME_OVERRIDE connect to that port instead of the load balancer’s.

Release v846

September 28, 2026

• Restore the virtual_cluster_name tag on Kafka handler, job runner, and other Agent metrics emitted.
• Add support of a new -metadataURLs (or WARPSTREAM_METADATA_URLS) flag to inject multiple Warpstream control plane addresses. This makes Private Link compatible with zonal load balancers introduced in v793.

Release v845

September 25, 2026

• Local, demo, and playground modes no longer query cloud instance-metadata services during startup. It makes startup much faster when using those modes on a non-cloud device such as a laptop (usually around 5-7s faster).

Release v844

September 24, 2026

• Bump agent base image (cgr.dev/chainguard/wolfi-base) to fix CVE-2026-85091 (zlib).
• Enable schema validation for Protobuf records with Confluent-compatible schema registries.
• Reduce the size of compaction commits between the Agents and the control plane, reducing egress costs.
• Removed the shutdown lock, along with the -enableShutdownLock flag and WARPSTREAM_ENABLE_SHUTDOWN_LOCK environment variable. Agents started with -enableShutdownLock will now fail to start, so remove the flag before upgrading.
• Added the agent_tls_certificate_expiry_seconds metric, which reports the number of seconds until the agent’s TLS serving certificate expires. It is tagged by cert_purpose (kafka_server, schema_registry, or internal_agent) so each serving certificate can be monitored separately, making it possible to alert on certificates that are close to expiring or not being rotated.

Release v843

September 21, 2026

• Tableflow:
  • Breaking change: Rename WARPSTREAM_TABLEFLOW_ASSUME_ROLE_ARN and WARPSTREAM_TABLEFLOW_ASSUME_ROLE_DURATION_MINUTES to WARPSTREAM_TABLEFLOW_GLUE_ASSUME_ROLE_ARN and WARPSTREAM_TABLEFLOW_GLUE_ASSUME_ROLE_DURATION_MINUTES to configure AWS Glue. If you were using the previous variable with a prefix on an existing Tableflow cluster, please reach out before upgrading as it will cause trouble.
  • Breaking change: In order to configure assume role for a Tableflow bucket, either use WARPSTREAM_BUCKET_ASSUME_ROLE_ARN_DEFAULT or convert the bucket URL to uppercase and replace . and - by _ to append it to WARPSTREAM_BUCKET_ASSUME_ROLE_ARN_. For instance for bucket warpstream-test-tableflow.cluster the expected associated env var would be WARPSTREAM_BUCKET_ASSUME_ROLE_ARN_WARPSTREAM_TEST_TABLEFLOW_CLUSTER. This allows to define one assume role per bucket if you use several ones.
• Bump franz-go package from v1.20.6 to v1.21.7.
• Fix compacted-topic compaction missing aborted transactions when one partition is merged as several output batches.

Release v842

September 17, 2026

• Stop emitting an “unknown WarpStream environment variable” diagnostic when a per-bucket BUCKET_ASSUME_ROLE_ARN_xxx environment variable is set.
• Fix a very rare bug where compacted-topic compaction could miscategorize a record as committed or aborted when its batch was merged together with one from a different input file.

Release v841

September 15, 2026

• Add retry mechanism for schema lookups in the schema validation path when they return schema_not_found to handle the race where a producer self-registers a schema and produces with it before every schema registry replica has caught up.

Release v840

September 10, 2026

• Tableflow:
  • Breaking change: Rename warpstream.tableflow_partition_offset_lag to warpstream.tableflow_ingestion_offset_lag. Tableflow cluster lag metrics now omit tags listed in -tableflowClusterMetricsTagDenyList / WARPSTREAM_TABLEFLOW_CLUSTER_METRICS_TAG_DENY_LIST. Valid values are table, topic, and partition (defaults to partition). An empty value emits all tags. table drops both the table_name and table_uuid tags.
• Bump Apache Thrift to v0.24.0, the RabbitMQ AMQP 0.9.1 client to v1.14.0, and gRPC to v1.83.2 to address CVE-2026-43871, CVE-2026-79921, and SNYK-GOLANG-GOOGLEGOLANGORGGRPCINTERNALTRANSPORT-19651825.
• Support path-prefix wildcards in SPIFFE mTLS ACL principals, e.g. User:spiffe://acme.com/team/* now authorizes any workload under that path. Previously only the whole-trust-domain wildcard User:spiffe://acme.com/* was supported.
• Rework the alter-topic-config and alter-broker-config CLI sub-commands to support incremental alter config. They now take kafka-configs style --add-config "k1=v1,k2=v2" and --delete-config "k1,k2" flags and default to IncrementalAlterConfigs, so only the specified configs are changed. Pass --full-state to replace the entire config set via the legacy AlterConfigs API. Breaking: the old single-pair --config-name/--config-value flags are removed, and the delete-broker-config sub-command is removed in favor of alter-broker-config --delete-config.

Release v839

September 8, 2026

• Tableflow:
  • Add telemetry to detect orphaned snapshot files.
  • Update the Lakehouse runtime catalog integration to automatically infer namespace locations for multi-bucket catalogs.
  • Fix a bug where the metadata.json files created automatically by the Lakehouse runtime catalog were being orphaned.
• Bump build to go 1.27.1
• Fix CVE-2026-56855
• Report object storage lifecycle policies that can corrupt your Warpstream clusters and buckets whose policies cannot be verified. Agents now require s3:GetLifecycleConfiguration for S3, s3express:GetLifecycleConfiguration for S3 Express, and storage.buckets.get for GCS to inspect lifecycle configurations (https://docs.warpstream.com/warpstream/agent-setup/different-object-stores).
• Add a diagnostic_agent_group label to warpstream.diagnostic_failure. Agent-pushed diagnostics emit one series per live and recently departed agent group. Diagnostics without an applicable or known reporting group use diagnostic_agent_group="no_group". Use max by (diagnostic_name) to monitor overall cluster health and sum by (diagnostic_name) to count failing agent groups per diagnostic.

Release v838

September 4, 2026

• Change the default max concurrent files for each fetch to 64.
• Refresh negatively cached schema validation lookups (schema_not_found) after 1 minute.

Release v837

September 4, 2026

• Set CGv2 request UUID from the Agent.
• Update the consumer group analyzer for modern consumer groups.
• Change the default max concurrent files for each fetch to 64.
• Fix query engine queries timing out on mTLS-authenticated clusters with ACLs enabled.

Release v836

August 31, 2026

• Reduce Agent memory and garbage collection overhead used to remember per-connection Metadata partition assignments. Large clusters no longer keep a full Go map of every topic-partition per Kafka connection.
• Fix misleading ACL shadow-denial logs for CreateTopics. A CLUSTER CREATE miss is no longer logged as [shadow_acl:not_enforced] when the request is allowed by a topic-level CREATE ACL (for example Kafka Connect creating its internal topics).

Release v835

August 27, 2026

• Fix Metadata v1+ treating a non-null empty topic list as “return all topics”. That now matches Apache Kafka: null (or empty on v0) returns every topic; an empty list on v1+ returns brokers only. Metadata error responses follow the same rule and no longer inject a synthetic error topic for empty v1+ lists. This could cause very high CPU and egress on clusters with many clients.
• Add agent_kafka_request_received_bytes and agent_kafka_request_transmitted_bytes counters, tagged by kafka_key, for Kafka protocol bytes received from and written to clients.
• Add a workspace_id label to hosted Prometheus cluster metrics, Agent-published metrics, and cluster_metrics events.

Release v834

August 26, 2026

• Add support for the ConsumerGroupHeartbeat API (key 68, KIP-848) for the modern consumer group protocol.
• Enable multi-source Orbit.
• Upgrade Bento to v1.21.1.
• Tableflow: Fixes a bug where we stopped reporting lag on DLQ topics when they were fully caught up.
• Bump agent base image (cgr.dev/chainguard/wolfi-base) to fix CVE-2026-38754, CVE-2026-38753, CVE-2026-38752 (busybox) and CVE-2026-54876 (libcrypto3/libssl3).
• Fix orbit_source_cluster_offsets consumer group lag reporting a false spike for a fully-migrated Orbit topic whose destination partition had aged out via retention.
• Add agent_group_overrides to Orbit pipeline configs, letting agents in different -agentGroups reach the source cluster via different bootstrap brokers.

Release v833

August 21, 2026

• Make sure that agent metrics include the virtual_cluster_name tag when possible.
• Tableflow: deletion file rewrites now stream output Parquet row groups instead of buffering the entire output file in memory until close.

Release v832

August 19, 2026

• Tableflow:
  • Add the ability to create more row groups per Parquet file for ingestion.
  • Relax ingestion file size constraint.
  • Add the following metrics:
    • The agent_tableflow_parquet_file_num_rows distribution metric tracks the number of rows per Parquet file.
    • The agent_tableflow_ingestion_live_partitions gauge tracks the number of distinct Iceberg partitions created during ingestion.
• Prevent deleted source topics from blocking Orbit fetches for healthy topics.
• Stop paging the file-cache invariant single actor stream returned N bytes, but M were requested when the Fetch is already canceled. A canceled request could close the stream while it was still being copied, which looked like a short stream.
• warpstream local: add -saslUser, -superUsers, and -enableACLs so integration tests can start a local cluster with named SASL users and Kafka ACL enforcement. Also fix -requireSASLAuthentication storing/printing the password as REDACTED.
• Fix a bug in the HTTP fetch endpoints where the pooled fetch buffers were returned to the buffer pool before the HTTP response encoder had finished reading them (the JSON response references the buffers zero-copy). Under concurrent load, another fetch could re-lease the buffers and overwrite them mid-encode, corrupting the HTTP fetch response. The buffers are now only released after the response has been fully encoded and written.
• Bump build to Go 1.26.6 and golang.org/x/mod to v0.40.0 to fix CVE-2026-39821, CVE-2026-46600, CVE-2026-33818, CVE-2026-56853, CVE-2026-56862, CVE-2026-56859, CVE-2026-56864, and CVE-2026-56865.
• Add blob_store_write_duration metric, tagged by bucket_url, reporting per-sub-bucket write latency for striped buckets.

Release v831

August 14, 2026

• Tableflow:
  • Add the WARPSTREAM_TABLEFLOW_ASSUME_ROLE_ARN and optional WARPSTREAM_TABLEFLOW_ASSUME_ROLE_DURATION_MINUTES environment variables for a shared STS AssumeRole used by both S3 bucket access and AWS Glue catalog sync. When set, the Tableflow role supersedes WARPSTREAM_BUCKET_ASSUME_ROLE_* for bucket access.
  • Fix a bug where the bucket URL prefix query parameter wasn’t being respected when AssumeRole is used.
• Stop logging an error when setting or clearing a connection’s read deadline fails because the connection was already closed. This just means the client disconnected.
• Add the loading_queue_used_count, loading_queue_discard_count, and loading_queue_load_outcome (tagged by outcome) metrics, all tagged by queue (prepared_upload, jobs, or file_id), so it’s possible to see how many items each of the agent’s loading queues handed out, aged out unused, and failed to load.

Release v830

August 13, 2026

• Tableflow:
  • A schema can now be defined inline in schema registry mode and will be used as the schema used to write records in Parquet files. This is handy for defining transforms that change of the shape of the data being inserted into the table.
• The error handling connection, closing it log now includes the client_id of the Kafka client that was using the connection.
• Remove noisy log when the context is canceled for speculative reads.
• The auto migration forwarded produce metrics warpstream.agent_kafka_produce_forwarded_records_counter, warpstream.agent_kafka_produce_forwarded_compressed_bytes_counter and warpstream.agent_kafka_produce_forwarded_compressed_bytes are now tagged by outcome, so records that the source Kafka cluster rejected are reported alongside the ones it accepted. Possible values are success, error, and unverified (used when producing with acks=0, where the source cluster sends no response and the Agent cannot confirm the records were accepted).

Release v829

August 12, 2026

• Tableflow:
  • Add the -tableflowHighCardinalityDistributionMetrics flag (WARPSTREAM_TABLEFLOW_HIGH_CARDINALITY_DISTRIBUTION_METRICS environment variable) to control whether high cardinality per table distribution metrics are emitted. Defaults to false.
• Fix pure-Go Lz4Block CompressAppend so it appends to the destination prefix instead of discarding it, which could corrupt Kafka record-batch headers written before compression.
• Fix flexible Kafka response framing so reused connection write buffers always emit an empty response-header tagged-fields count. A stale byte (for example after SASL handshake v0) could make clients fail to decode later flexible responses such as Fetch.
• Fix a race that could assign the same Kafka connection ID to an external connection and an in-process direct-dialer connection, and refuse to serve a prefetched Fetch response when the claimer’s Fetch API version does not match the originator’s.
• Fix noisy “async gcs rapid storage writer close failed” logs with context canceled during Rapid Storage uploads by decoupling the writer context from caller cancelation while preserving the deadline.
• The metrics subcommand now supports OIDC workload identity federation via the -workloadIdentityTokenSource flag (and WARPSTREAM_AGENT_WORKLOAD_IDENTITY_TOKEN_SOURCE environment variable), matching the agent subcommand. When set, -enableControlPlanePrometheusEndpoint is forced to true because the exchanged agent token is only accepted by the control plane’s prometheus endpoint.
• Enable public Agent releases to receive control-plane-generated Events on Kafka clusters, including diagnostics and Orbit migration events.
• Size the batcher’s prepared upload queue and load concurrency off the agent’s CPU quota instead of the host’s CPU count, so containerized agents no longer over-create prepared upload files.
• Stop deleting prepared upload files that lose the race for a spot in the batcher’s queue. They now wait for the next free spot, which removes the object store create and delete each one used to cost.

Release v828

August 10, 2026

• Tableflow:
  • Add the agent_tableflow_parquet_file_size_uncompressed and agent_tableflow_parquet_file_size_compressed distribution metrics for Parquet files, tagged by source (ingestion, compaction, or deletion).
  • Add the agent_tableflow_parquet_file_num_row_groups, agent_tableflow_row_group_size_uncompressed, and agent_tableflow_row_group_size_compressed distribution metrics for Parquet row groups, tagged by source (ingestion, compaction, or deletion).
  • Add support for multi-bucket catalogs in BigLake.
  • Add experimental support for upsert tables (table_type: upsert): records are deduplicated by Kafka record key within each source partition (highest offset wins) and records with null values delete the key from the table.
• Fetch requests that include an unknown or invalid topic ID (for example, a recently deleted topic referenced by TopicID only) now return a partition-level error for that topic instead of failing the entire Fetch. Valid topics in the same request continue to return data, matching Apache Kafka behavior.

Release v827

August 4, 2026

• Add diagnostic to detect when the Orbit auto migration client has connectivity issues with the source cluster.
• Fix incorrect failed to lookup AZ from CIDR mapping warnings for the Agent’s own internal clients (like the query engine).
• Improve auto migration’s logic to detect if a produce request is transactional by actually checking if the record batch has the transactional flag in its attributes.
• Upgrade Bento to v1.20.0.
• Add the user-provided credential_name to ACL denial logs.
• warpstream local: allow clients to use session / rebalance / transaction timeouts below the normal broker minimums so unit tests against local can run faster (previously rejected).
• Skip shadow ACL denial noise for in-process direct-dialer clients (e.g. the agent’s query engine / events UI).
• Accept OAuth bearer tokens on the HTTP Fetch and Produce APIs. When saslOauthIssuerURL and saslOauthAudience are configured, HTTP clients can authenticate with Authorization: Bearer <token> in addition to HTTP Basic auth; bearer tokens are validated via the same OAUTHBEARER path used by the Kafka wire protocol.
• Migrate direct imports from github.com/hamba/avro/v2 to github.com/iskorotkov/avro/v2 v2.33.1 to address CVE-2026-46384, CVE-2026-46385, and GHSA-mx64-mj3q-7prj. Transitive hamba/avro usage remains until dependent modules move off it.
• Set MaxMapAllocSize on Tableflow Avro decode of untrusted Kafka payloads to mitigate map-allocation DoS (GHSA-mx64-mj3q-7prj).
• Added bucket striping (warpstream_stripe://$URL_1<>$URL_2<>...<>$URL_N, 2–32 sub-buckets): an object-storage wrapper that scales writes past a single bucket’s request-rate ceiling by striping objects across sub-buckets. Placement is deterministic — each object lives on exactly one sub-bucket, chosen by hashing its key. Writes fail over to another sub-bucket when the target’s circuit breaker is open; reads go to the hashed sub-bucket and fall back to the others on NotFound. Unlike warpstream_multi:// (which replicates for read availability), striping stores each object once, so a sub-bucket outage makes its objects unreadable — the same blast radius as a single bucket. Retiring a sub-bucket by shortening the URL strands dead files on it unless the retired bucket (or the previous stripe URL) is added to additionalBackgroundTasksBucketURLs.

Release v826

July 29, 2026

• Tableflow:
  • Introduce a maximum uncompressed row size limit for Tableflow ingestion, measured after projection and transforms. Records that exceed the limit are handled based on the configured DLQ policy.
  • Use a more precise memory measurement for Tableflow ingestion buffering.
• Added new metric warpstream_orbit_auto_migration_unproxied_source_writes_num_records: counts the number of detected unproxied records to the source cluster that bypassed the WarpStream proxy.
  • tags: topic=<topic>
• Add agent_group to root metric tags so metrics like warpstream.agent_kafka_request_latency and warpstream_consumer_group_lag are groupable by agent group in Datadog.
• Fix SASL OAUTHBEARER rejecting non-RS256 tokens (e.g. ES512) when the OAuth 2.x authorization server is discovered via RFC 8414 and does not advertise id_token_signing_alg_values_supported.
• Fix events queries returning an internal error instead of a “no events data found” message when an event topic was created moments earlier and the stream metadata cache had not caught up yet.
• Adds support for auto-migrating idempotent producers via Orbit.

Release v825

July 24, 2026

• Add Confluent REST Proxy v2-compatible and multi-topic HTTP produce endpoints.
• Tableflow: Fix ingestion trying to flush an empty file (failed to add record to buffer: failed to sort and flush: cannot flush empty record batch).
• Bump google.golang.org/grpc to v1.82.1 to fix GHSA-hrxh-6v49-42gf.

Release v824

July 22, 2026

• Managed Data Pipelines: when the number of pipelines Agents changes, scale pipeline instance concurrency up or down instead of stopping and restarting all pipelines.
• Fix a throughput bottleneck during Orbit auto-migration PROXY.

Release v823

July 21, 2026

• Fixes a panic that impacted the demo command, the playground command, and agents connected to a Tableflow cluster (panic: failed to initialize duckdb client dependencies).
• OAUTHBEARER SASL: fall back to /.well-known/oauth-authorization-server (RFC 8414) when OpenID Connect discovery at /.well-known/openid-configuration fails, so identity providers that only implement the RFC 8414 discovery endpoint can be used.
• Tableflow: Add support for custom table names. When a table is renamed, the tagging of logs for that table will start to use the new name.

Release v822

July 20, 2026

• Tableflow:
  • Reject decimal values that exceed the column type’s declared precision with an error instead of panicking during ingestion.
  • Release sorting and column statistics tracking for Iceberg decimal types.
  • Fix false-positive “Cannot Access Tableflow Schema Registry” health diagnostic against WarpStream BYOC Schema Registry (and other registries that do not implement GET /mode).
• Remove the legacy AWS SDK v1 from agent binaries to prevent false-positive CVE-2020-8911 reports.
• GCS Rapid Storage: flush on close instead of waiting for a full synchronous Close, reducing write finalize latency.
• Add a delete-broker-config CLI sub-command (warpstream cli delete-broker-config --config-name <name>) that resets a broker config to its default via IncrementalAlterConfigs with a delete op.

Release v821

July 14, 2026

• Tableflow:
  • Store decimals with precision greater than 18 as FIXED_LEN_BYTE_ARRAY in Parquet.
  • Add a health diagnostic that checks connectivity to schema registry integrations.
• Upgrade Bento for the parse_big_decimal bloblang method (Kafka Connect / Debezium decimal decoding).
• OIDC workload identity federation now works on multi-region clusters (and across single-region cluster migrations): the agent holds a short-lived token per region and presents the one minted by whichever region it is talking to.
• Promote warpstream cli-beta to warpstream cli — it is now the default Kafka CLI. warpstream cli-beta still works as a deprecated alias (prints a deprecation warning on stderr) and will be removed in a future release. The previous warpstream cli command has been renamed to warpstream cli-old and will be retired eventually; warpstream kcmd continues to work as before. Note: this is potentially a breaking change since some commands are different and most of the flags are different.

Release v820

July 10, 2026

• Tableflow:
  • Add Schema Registry integration for Avro schemas.
• Enable LZ4 compression in benchmkark-producer CLI command and tweak defaults to be more sane.
• Fix a data race when speculative object storage uploads finish concurrently.

Release v819

July 10, 2026

• Tableflow:
  • Emit diagnostic and event when the subject is not found in the Schema Registry.
• Upgrade Bento to v1.19.0 (requires Go 1.26.5).
• Bump build to go 1.26.5.

Release v818

July 9, 2026

• Added a new warpstream local command that runs a fully in-memory, self-contained WarpStream cluster (Kafka + Schema Registry + Tableflow) with no connection to the WarpStream control plane. Intended as a replacement for warpstream playground in CI pipelines.
• Automatically run warpstream local when warpstream demo or warpstream playground is started in CI, with a warning that local mode is designed for CI environments.
• Agent: dynamically tune the direct load prefetch concurrency based on observed CPU usage instead of using a fixed value of 16. Underloaded agents can still serve up to 64MiB of concurrent prefetch per request, but as CPU usage rises the concurrency automatically steps down to avoid the agent DDOSing itself with prefetch work it can’t transmit in time, which previously caused wasted GET requests and slow recovery under high load.
• Orbit: add support for JSON key/value secrets in Cloud Provider’s Secrets Manager. A secret reference in the Orbit config can now specify an optional key to extract a single value from a secret stored as a JSON object (the standard AWS Secrets Manager key/value pattern). AWS secrets stored as binary (SecretBinary) are now supported as well.
• Ensure Close() is called on the pooled buffer after all replicas complete on the quorum-success path of multi-bucket PutBytes.
• Add cluster-level default partition auto scaler settings (warpstream.default.partitions_auto_scaler.* broker configs) that are applied to newly created topics, so a cluster can enable partition auto scaling by default for all new topics.

Release v817

July 8, 2026

• Tableflow:
  • Expose Schema Registry writer schema ID and version as warpstream.sr.writer_schema_id and warpstream.sr.writer_schema_version to ingested records before applying Bento transforms so that the transform logic can condition on those properties.
  • Make BigLake integration automatically detect catalogs with credential vending mode and send X-Iceberg-Access-Delegation header on Iceberg REST calls if so.

Release v816

July 6, 2026

• Allow Orbit to chunk requests when copying consumer group offsets.
• Schema Registry: include Confluent-compatible guid values in schema API responses and support looking up schemas by GUID with GET /schemas/guids/{guid}.

Release v815

July 2, 2026

• Add support of -disableConsumerGroupMetrics (env var WARPSTREAM_DISABLE_CONSUMER_GROUP_METRICS) and -disableConsumerGroupsMetricsTags (env var WARPSTREAM_DISABLE_CONSUMER_GROUP_METRICS_TAGS) in agent metrics mode to offer the same knobs as in regular agent mode over cluster level metrics cardinality. By default -disableConsumerGroupsMetricsTags is set to empty string (the default value for regular agents is partition).
• Tableflow:
  • Add support for OAuth as an authentication mechanism for source Kafka clusters.
• Fix Go runtime CPU utilization reporting so stale high samples decay when the Go runtime has not emitted a fresh CPU metrics snapshot.
• Auto enable bucket pre-warming when the bucket is of type Rapid Storage.
• Reduce default value of GOGC for Agents running only pipelines role from 1000 to 200.

Release v814

July 1, 2026

• Fixed a bug in the Prometheus metrics scraping path where the warpstream_max_offset and warpstream_min_offset metrics were summed across partitions instead of taking the max/min when the partition tag is disabled.
• Fix the metric double publishing problem introduced in v808 when the dedicated metrics mode was enabled (the regular agents would keep publishing the same metrics).
• Improve produce latency for GCP Rapid Buckets by pre-warming upload streams.
• Add a new -additionalBackgroundTasksBucketURLs flag and WARPSTREAM_ADDITIONAL_BACKGROUND_TASKS_BUCKET_URLS environment variable that accepts a comma-separated list of bucket URLs that should have background tasks run on them like scanning for dead files to delete or ripcord files to ingest. This is important for customers who are migrating from one bucket to another. The new flag supersedes the existing -additionalDeadscannerBucketURLs flag and WARPSTREAM_ADDITIONAL_DEADSCANNER_BUCKET_URLS environment variable.
• Add support for OIDC workload identity federation: when WARPSTREAM_AGENT_WORKLOAD_IDENTITY_TOKEN_SOURCE is set, the agent obtains a platform OIDC token and exchanges it with the control plane for a short-lived agent token, removing the need for a static agent key. Currently aws (STS) and gcp (metadata server) sources are supported, but other platforms may be added in the future. Supported for single-region clusters only.
• Add an opt-in file ID allocator (behind the use_range_file_id_allocator flag, default off) that hands out IDs from several discontiguous reserved ranges to spread object-storage key prefixes more evenly and reduce metadata-layer hot-spotting. The key layout is unchanged. The WARPSTREAM_USE_RANGE_FILE_ID_ALLOCATOR environment variable (true/false) is the final per-Agent override for the flag.

Release v813

June 29, 2026

• Tableflow:
  • Update internal compaction API usage.
  • Fix bug where the schema registry integration feature was not ungated properly.
  • Add support of Cloud Provider’s Secrets Manager in TableFlow config.
• Add a new flag -skipBucketsPermissionCheck (env var WARPSTREAM_SKIP_BUCKETS_PERMISSION_CHECK) that allows to skip the permission check for buckets when starting an agent. The default value is false.
• Improve DNS resolution of S3 addresses by automatically injecting a . at the end of any S3 endpoint we try to connect to. This is a workaround because the AWS SDK does not support it to this day https://github.com/aws/aws-sdk-go/issues/1380.
• Re-enable GCS direct connectivity client by default.
• Disable some connection-level backpressure mechanisms when TLS is enabled because once we’ve paid the price of accepting a TLS connection and doing the formal handshake, paradoxically, backpressuring that connection by closing it due to high load makes the situation worse as the client will just try to re-establish the connection and the TLS handshake is extremely expensive.
• Stop playground agents from logging query engine state refresh errors when query engine credentials are not required.

Release v812

June 22, 2026

• Move orbit consumer offset copying job entirely into the agents.

Release v811

June 22, 2026

• Tableflow:
  • Add Schema Registry integration for Protobuf schemas.
  • Fix Tableflow ingestion getting stuck (no ranges were processed, possibly due to timeout) on source topics written by transactional producers (e.g. CDC), where the offset at the high watermark is a transaction control marker that a read-committed consumer never receives.
• Add support of Cloud Provider’s Secrets Manager in Orbit config.
• Schema Registry: prevent an incompatible type change of a oneof field when that field’s oneof block is renamed or dissolved. This matches a bug fix in Confluent’s protobuf compatibility that will be released in Confluent Platform 8.3.1.

Release v810

June 19, 2026

• Tableflow:
  • Support sorting and column statistics on float and double columns.
• Schema Registry: Match Confluent’s protobuf compatibility behavior by allowing a type change on a oneof field is now allowed when that field’s oneof block is renamed or dissolved.

Release v809

June 18, 2026

• Orbit now creates empty source topics with a non-zero log end offset at that offset in WarpStream. This fixes a bug where an empty source topic could be migrated to WarpStream at offset 0 instead of preserving its source log end offset.
• Add a cli-beta validate-pipeline-config command for validating managed data pipeline config files locally. Usage: warpstream cli-beta validate-pipeline-config --config-file pipeline.yaml.
• Restrict which clients see support for KIP-714 client metrics APIs. The GetTelemetrySubscriptions and PushTelemetry endpoints will no longer be advertised to clients older than this version.

Release v808

June 18, 2026

• Tableflow:
  • Add support for sorting and column statistics tracking for Iceberg tables.
  • Fix time-based partition transform on an Avro date field.
• Update the publish metrics job implementation to scrape metrics from the control plane Prometheus endpoint. This should not change any of the existing metric, but will add a few that were only available in the control plane Prometheus endpoint.

Release v807

June 12, 2026

• Orbit Auto Migration is now available.

Release v806

June 12, 2026

• Forward produce backpressure metrics for observability.
• Add support for OAuth as an authentication mechanism for source clusters in Orbit.
• Tableflow: Validate properly the required fields nested under untyped maps via additionalProperties for JSON ingestion.
• Bump agent base image (cgr.dev/chainguard/wolfi-base) to pick up busybox fix for CVE-2023-39810.

Release v805

June 10, 2026

• KIP-714 client metrics are now generally available to all customers.

Release v804

June 9, 2026

• Add global sort duration in batcher to flush_end log.
• Fix a bug where the agent could have problems listing ripcord files if there were too many of them and your bucket URL was long, because the generated output was too big.
• Fix Orbit and Tableflow source connections silently dropping SASL when TLS is enabled without mTLS.

Release v803

June 8, 2026

Warning: This version drops SASL when connecting to an Orbit or Tableflow source cluster over TLS without mTLS. It is fixed in v804. Please upgrade to v804 directly and skip v803.

• Fix panic when emitting the client availability zone mismatch diagnostic during interzone load balancing.
• Upgrade Bento to v1.18.1.
• Fix Orbit and Tableflow source cluster TLS configuration to allow setting a private CA (mtls_server_ca_cert_env) without requiring mTLS.
• Schema Registry: the error_code for incompatible schema responses now returns 40901 (was 409) to align with Confluent’s documented API. The HTTP status code remains 409.
• Schema Registry: Avro schemas with invalid field defaults are now rejected at registration time (aligns with Confluent SR 8.2.1 behavior). Existing schemas registered before this change are unaffected.
• Add batching for JoinGroup and SyncGroup requests via RSM batch commands, reducing the number of RSM proposals when many consumers join/sync concurrently.

Release v802

June 4, 2026

• Bump build to go 1.26.4 to fix CVE-2026-42504
• Add an agent flag to override max inflight fetch compressed bytes per CPU.
• Upgrade Bento to v1.18.0
• Tableflow:
  • Fix bug for dlq_keep_settings.retention: we now allow for d and w units for days and weeks respectively.
  • Fix bug that considered null values set within transforms as non-null when fields were marked as required.
  • Tableflow agents can now create Iceberg table metadata even when they cannot access external blob store buckets directly. This can be enabled by setting the tableflowMetadataSyncMode flag to proxy.
  • Support converting between booleans and strings, and between integers and strings, in both directions when mapping records to the destination schema.
  • Breaking Change (AWS Glue): Fix a bug in the AWS Glue integration that caused us to hit the TableVersions limit when syncing the table. This fixes the error Number of TABLE_VERSION resources exceeds the limit 100000 per TABLE. Note that agents will now require additional roles: glue:GetTableVersions and glue:BatchDeleteTableVersion.
  • Fix some tags on the event emitted when sending records to the DLQ.

Release v801

May 29, 2026

• Deprecate a legacy code path for computing Data Lake sorting bounds and column statistics.
• Fix events scheduler writing a corrupt active state when an event type was disabled while events were globally enabled. Agents now drop records with stream ID 0 instead of poisoning the file batch.
• Tableflow: Implement basic validation for required / nullable fields in JSON.

Release v800

May 27, 2026

• Tableflow:
  • Fix bug on transforms: Bento transforms replacing the full root object instead of modifying it field by field are now handled correctly (e.g. root = { ...} now works).
• Fix warpstream_consumer_group_estimated_lag_very_coarse_do_not_use_to_measure_e2e_seconds reporting full retention as the lag for partitions with no committed offset. The metric now uses the age of the oldest surviving record (bounded by retention).

Release v799

May 27, 2026

• Fix events queries failing with UNSUPPORTED_SASL_MECHANISM on clusters that restrict enabledSASLMechanisms to exclude PLAIN.

Release v798

May 26, 2026

• Fix a query engine bug that could cause Agent panics for some sorted and limited queries.
• Fix false positive diagnostic for cross az traffic when the query engine was used.
• Fix false positive diagnostic for small fetch timeout when the query engine was used.
• Return clear error when schema registry request body size exceeds limit.

Release v797

May 19, 2026

• Tableflow:
  • Fix decoding of Protobuf oneof fields (introduced in v796, upgrade directly to v797 if you use a configuration with Protobuf schemas).
  • Recreate BigLake tables when the Iceberg table UUID changes.
• Fix DNS-backed Agent file cache retries so failed replicas can fall back to other resolved Agents.
• Add per-pipeline log level controls for managed data pipelines.
• Fix event queries on agents with SASL enabled.

Release v796

May 18, 2026

Warning: There is a problem for Tableflow protobuf schemas with that version fixed in v797. Please upgrade to v797 directly and skip v796 if you use Tableflow with a protobuf schema.

• Upgrade the embedded version of Bento to v1.17.0.
• Start supporting regional migrations.
• Add a rate limiter in the metrics mode to throttle metrics emission. This avoids flooding the Datadog agent when Datadog metrics are enabled.
• Adds an experimental agent flag that can be used to reduce E2E latency for workloads that do not produce new records consistently on all partitions all the time.
• Tableflow:
  • Breaking change for very old agent versions using protobuf, they would need to upgrade to version 796+.
  • Add support for field type remapping

Release v795

May 14, 2026

• Forward schema registry metrics for observability.
• Tableflow: Fix the Tableflow configuration used in the demo mode.
• Reduced telemetry sent from Agents to control plane.

Release v794

May 12, 2026

• Add a diagnostic for invalid regex subscriptions in modern consumer group heartbeat requests.
• Bump golang.org/x/net to v0.54.0 to fix CVE-2026-33814
• Detect when Kafka clients request one availability zone via ws_az but produce or fetch against an Agent in a different availability zone and fire a diagnostic.

Release v793

May 11, 2026

• Unless customers specify a -metadataURL/WARPSTREAM_METADATA_URL the agents will start using 3 different “zonal” endpoints to communicate with WarpStream - meaning one per WarpStream control plane availability zone. The previous behavior can be put back by setting the environment variable WARPSTREAM_AGENT_ENABLE_ZONAL_URLS to false. Removes a dependency on inconsistent behavior across cloud providers when load balancing across multiple availability zones and increases the product’s resiliency to zonal outages significantly.
• Bump build to go 1.26.3
• Fixed CVEs: CVE-2026-39820 / CVE-2026-33811 / CVE-2026-42499 / CVE-2026-39820 / CVE-2026-33811 / CVE-2026-39836
• Agent will now enforce read-only Schema Registry credentials and only allow read schema operations for read-only credentials.

Release v792

May 8, 2026

• Add optional PROXY protocol v2 support on the Kafka listener via kafkaProxyProtocol (or WARPSTREAM_KAFKA_PROXY_PROTOCOL). When enabled, every connection must include a v2 PROXY header before any other bytes; v1 headers are rejected. Operators can also set kafkaProxyProtocolPrincipalTLVType (a hex byte in the PP2 user-defined range 0xE0–0xEF) to source the connection’s ACL principal from a custom TLV in the PROXY header — the TLV value must already include the User: prefix and overrides any principal that would otherwise be derived from mTLS.
• Tableflow:
  • Fix a bug in ingestion that was not respecting some types from the input schema.
  • Fix ingestion of date types for Avro input schemas.
  • Fix ingestion of Avro map types with optional values when the deprecated way of specifying schemas is used.
  • Fix several correctness bugs in the Protobuf decoder so records now align with the proto3 spec (oneof last-wins, merging of repeated singular message fields, strict wire-type validation).

Release v791

May 6, 2026

• Fix a bug in the agent metrics mode that would cause it to stall when any of the metrics scraped contained a comma in its labels.
• Mark file cache closed-pipe logs as debug.

Release v790

May 5, 2026

• Tableflow: Make ingestion handle partition evolution.
• Bump our Chainguard Wolfi Docker image to 79af0917ba7ac066ebf1f99e5967ee9e77e6a350facec5be43a7a236c705cf10 to fix CVE-2026-5450 and CVE-2026-5928.
• Fix a bug in the query engine when querying across events that mix integers and floats in the same field name.
• Fix a bug in the query when combining multiple filters like: filter a==“b” | filter c==“d” where only the latter filter would be applied.

Release v789

April 30, 2026

• Fix a bug where object store errors due to networking issues (DNS or TCP timeouts, context cancellation, etc) were misclassified as NotFound errors, causing red-herring invariant violations.
• Tableflow: Fix ingestion of list types with optional elements.
• Public agent docker images and release tarballs now ship notices.txt (auto-generated from the binary’s transitive Go dependencies) and non_ibm_license.txt at the container/archive root, for IBM third-party legal compliance.
• Fix the reason for a decision made during compaction in a log from being wrong.
• Treat “unexpected EOF” as a retriable error for long-running compactions.

Release v788

April 29, 2026

• Bump github.com/aws/smithy-go to v1.25.1. This fixes a very slow memory leak associated with using S3 Express One Zone in the AWS SDK.

Release v787

April 28, 2026

• Query engine: disable the admin RPC metadata cache because it causes a bug with inconsistent results when querying events.
• Query engine: fix ::int, ::float, and ::str casts in event queries, and make integer/float comparisons work in filters.
• Agents will now automatically backpressure Kafka protocol requests when their CPU is >= 98% for a sustained period of time. This threshold can be adjusted by setting the WARPSTREAM_HIGH_CPU_BACKPRESSURE_THRESHOLD_PERCENT environment variable and disabled by setting it to 0.
• All Datadog metrics now include virtual_cluster_id, agent_id, and agent_version tags.

Release v786

April 28, 2026

• Emit cluster-level Prometheus-style metrics (consumer group lag, topic details, diagnostics, tableflow state, …) as cluster_metrics CloudEvents whenever events are enabled on the cluster. Emission is independent of the Datadog push: events are still emitted even when DisableAllMetrics or DisableConsumerGroupMetrics is set. Events are always at topic-level granularity (partition tag is always stripped) to keep the per-job event volume bounded. The event’s metric_name field carries the canonical warpstream_-prefixed name (e.g. warpstream_consumer_group_lag), identical to the Prometheus scrape name and the describe_cluster_metrics API response, so the same string identifies a metric regardless of how it is queried.
• Tableflow: improve DLQ health reporting so records routed to the DLQ are no longer surfaced as skipped, reducing misleading health warnings and making it clearer when data was retained for later replay.
• Improve query engine performance by caching metadata lookups and aligning query time boundaries.
• Move modern consumer group heartbeat regex matching and topic DESCRIBE ACL filtering from saasy into the agent, and forward the resolved regex stream IDs to saasy for statemachine application.
• Allow reserved keywords like topic to parse as part of .-deleted field identifiers in the query engine.

Release v785

April 23, 2026

• Emit cluster-level Prometheus-style metrics (consumer group lag, topic details, diagnostics, tableflow state, …) as cluster_metrics CloudEvents whenever events are enabled on the cluster. Emission is independent of the Datadog push: events are still emitted even when DisableAllMetrics or DisableConsumerGroupMetrics is set. Events are always at topic-level granularity (partition tag is always stripped) to keep the per-job event volume bounded.
• Make metrics pod use external HTTP client to have longer connect timeouts.
• Make metrics pod re-use HTTP client across requests to reduce time spent on establishing new connections.
• Change consoleURL parameter name to apiURL .
• Fix a bug that would cause the original error to be dropped from error messages and logs when hedging requests with the fast retrier.
• Fix bug in query parser for handling sub expressions.
• Make tableflow events all have the table name and table UUID fields set properly.
• Fix panic in query engine related to nested expression and or statement.
• Emit a tableflow event when the offset scraper job handler fails.
• Enable the agent query engine by default and require disableQueryEngine to opt out of query handling.
• Removed the OBJECT_STORAGE_PUBLIC_NETWORK_PATH diagnostic (added in v765).
• Added a diagnostic that fires when a produce request contains records for both lightning and classic topics, which removes the latency benefit of the lightning topic.

Release v784

April 20, 2026

• Fix a memory leak when using the metrics agent mode.
• Bump github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream to v1.7.8, github.com/aws/aws-sdk-go-v2/service/kinesis to v1.43.5, github.com/aws/aws-sdk-go-v2/service/lambda to v1.89.0 and github.com/aws/aws-sdk-go-v2/service/s3 to v1.99.0 to fix GHSA-xmrv-pmrh-hhx2.
• WarpStream Agents will no longer report the availability zone that they’re running in as the value of the Rack field in Broker metadata in the response of Metadata RPCs. Instead, they will always report their Rack as: “warpstream-fake-rack”. The reason for this is that WarpStream’s service discovery system tracks availability zones of clients using client ID features, not this Rack field. In addition, when clients enable rack-awareness for consumers in their Kafka clients without properly configuring WarpStream’s zone-aware service discovery system, the presence of the availability zone in the Rack field in the Broker’s metadata, combined with the fact that it will change from time to time due to how WarpStream’s partition assignment strategies work, may result in an excessive number of consumer group rebalances. As a result, since this field has no value in WarpStream clusters, we’re hard-coding the value of Rack so that misconfigured clients will not experience excessive consumer group rebalances.
• Tableflow:
  • Fix ingestion of Avro fixed and time types.
  • Performance improvement when sorting is enabled on tables.

Release v783

April 16, 2026

• Tableflow:
  • Reject invalid combinations of WARPSTREAM_BUCKET_URL, WARPSTREAM_INGESTION_BUCKET_URL and WARPSTREAM_COMPACTION_BUCKET_URL with an explicit error message for datalake agents with events enabled.
• Bump go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp to v1.43.0 for CVE-2026-39882 vulnerability.
• Fix connection reuse on the query engine.
• Fix a bug that didn’t allow users to manually disable events for tableflow clusters.

Release v782

April 14, 2026

• Tableflow: add warpstream.tableflow_dlq_records_counter agent metric tagged by topic and strategy (skip or keep) to track records handled by the DLQ during ingestion. Replaces the previous warpstream.ingestion_job_dlq_skip_strategy_counter which only covered the skip strategy.
• Improve query engine performance by fixing connection reuse.
• Improve produce latency by parallelizing record sorting in the agent batcher hot path.
• Fix Avro schema validation to handle references using short name within a namespace properly.
• Bump go.opentelemetry.io/otel/sdk to v1.43.0 for CVE-2026-39883 vulnerability.
• Improve how runtime CPU measurements are taken to be more precise and not persist inaccurate values for long period of time by occasion.
• Improved agent feature versioning scheme to auto-compute latest released version from feature declarations, eliminating manual maintenance and potential for version bump errors
• Query engine: fix distributed aggregation for aliased count() and count(expr) so merge/final stages consume the correct intermediate count columns.
• CLI alter-broker-config now surfaces per-broker error responses instead of silently reporting success.
• Support setting warpstream.default.topic.type via AlterConfigs and IncrementalAlterConfigs.
• Unknown warpstream-prefixed broker and topic configs now return an error instead of being silently ignored.

Release v781

April 9, 2026

• Breaking Change (GCP): WarpStream Agents now depend on storage.buckets.get access for all configured GCP (gcs://) bucket URLs. This can be overriden by setting the environment variable WARPSTREAM_GCS_PERFORM_BUCKET_STORAGE_CLASS_LOOKUP=false .
• Allow configuring the tracer block in managed data pipeline configs.
• Automatically disable idempotent writes for managed data pipelines writing to WarpStream clusters via kafka_franz_warpstream output blocks.
• Fix non-determinism in generating schema hash for unrecognized key in Avro schemas.
• Bump build to go 1.26.2
• Add support for GCP Rapid Buckets.
• Optimize metrics code, particularly when emitting Datadog metrics.
• Fix the docker image base to properly use the multi-arch wolfi-base image

Release v780

April 8, 2026

• Reduce max diagnostic groups per type from 10 to 5 to lower payload size sent to the control plane.
• Remove spammy error log when diagnostic groups exceed the per-type limit.
• Forward sampled flush_event logs for observability.
• Emit forwarded flush synchronous duration metric.
• Bump our Chainguard Wolfi Docker image to 79af0917ba7ac066ebf1f99e5967ee9e77e6a350facec5be43a7a236c705cf10 to fix CVE-2026-4437.
• Bump github.com/go-jose/go-jose/v4 to v4.1.4 to fix GHSA-78h2-9frx-2jm8.
• Prevent a single fetch request from triggering thousands of concurrent HTTP requests if it’s querying data for many topic-partitions whose individual data is spread across many files.
• Install CURL in Agent image so it can be used in health checks for services like ECS that rely on container having command line utilities available.
• Tableflow: tableflow events are generally available and agent-side events handling is enabled by default. This is a breaking change since tableflow agents now require a bucket URL to be configured (either WARPSTREAM_BUCKET_URL or both WARPSTREAM_INGESTION_BUCKET_URL and WARPSTREAM_COMPACTION_BUCKET_URL). Agents without bucket URLs will fail to start.
• Enable events when bootstrapping demo and playground clusters.
• Added SLOW_CONSUMER push diagnostic: detects consumers that are too slow between fetch requests using a dynamic threshold (baseline + 1s per MB of previous response size) and emits the diagnostic after 3 consecutive slow fetches on a connection.
• Fix GOMEMLIMIT parsing to support byte suffixes like “24GiB” and “1000MB” instead of requiring plain integers.
• Fix GOGC, GOMAXPROCS, and GOMEMLIMIT heartbeat reporting to not report a value on parse error. Previously a parse error would silently report 0.

Release v779

April 1, 2026

• Fix bug that prevented query engine from running queries successfully on Agents configured to only run the Jobs role.

Release v778

April 1, 2026

• Add principal impersonation via record headers. Configured impersonator principals can produce messages on behalf of other users by setting a designated record header, enabling ACL evaluation as the impersonated principal.
• Fix a bug that caused events queries to fail on agents with mTLS enabled.
• Enable dual-mode Prometheus native histograms alongside classic histograms for improved metric accuracy.
• Query engine: route query jobs only to agents with the query engine enabled and fail fast if an agent is configured to accept only query jobs without enabling the query engine.
• Replace mutex-protected rand.Rand instances with lock-free top-level rand functions in certain hot paths.
• Fix a bug that was preventing events from being emitted/enabled in public Agent builds.
• Enable agent-side events handling and query engine automatically for Kafka agents.
• Expose a /debug/pprof/trace endpoint for Go runtime traces
• Tableflow: fix a bug where compaction failed on certain parquet file layouts with gaps between column chunks, leaving small files unmerged.
• Tableflow: add type: "record_ingestion_failed" to logs corresponding to decoding errors during ingestion.
• Tableflow: add warpstream.tableflow_ingestion_lag_seconds and warpstream.tableflow_query_lag_seconds metrics. Ingestion lag measures how far behind the ingestion process is from the source Kafka topic. Query lag measures the time from when data is produced to Kafka until it is queryable (ingestion lag + catalog sync delay).

Release v777

March 26, 2026

• Fix schema validation in playground by passing missing schema registry URL to config.

Release v776

March 26, 2026

• Tableflow:
  • Add warpstream.tableflow_partition_offset_lag gauge for Tableflow ingestion offset lag in records. When topic or partition tags are omitted, offset lag is summed across partitions (like consumer_group_lag); time lag remains the max per scope.

Release v775

March 25, 2026

• Add topic, partition, and streamID to orbit offset error messages for easier debugging.

Release v774

March 25, 2026

• Cache metrics in fetch and produce Kafka handlers to reduce tagset allocations when updating metrics.
• Tableflow:
  • Fix panic in demo mode by adding missing input_schema to demo config.
• Add topic tag to agent_kafka_produce_with_offset_uncompressed_bytes_counter metric when high cardinality metrics are enabled.

Release v773

March 24, 2026

• Don’t swallow circuit breaker errors and make it clear in error logs that are showing cached circuit breaker errors what is happening.
  • For instance consumers could start failing when some Kafka topics are deleted but are still trying to be accessed, because the metadata response would be incorrect
• Don’t count structured errors from the control plane (400s) as failures in the control plane client circuit breakers.

Release v772

March 23, 2026

• Fixed IPv6 address parsing.
• Tableflow:
  • Make the TABLEFLOW_ prefix optional for environment variables used to specify Kafka cluster credentials.
  • Fix BigQuery integration for Tableflow tables failing to update when the Iceberg schema evolves (e.g., new columns added).
• Add support for ConsumerGroupDescribe API (key 69, KIP-848) for the modern consumer group protocol. Disabled by default.
• Make the ORBIT prefix optional for environment variables used to specify Kafka cluster credentials.
• Fix the empty_response tag value for the warpstream.agent_kafka_fetch_single_attempt_outcome metric.
• Bump google.golang.org/grpc to v1.79.3 for CVE-2026-33186 vulnerability.
• Bump our Chainguard Wolfi Docker image to ce84795834de56c47f27b6be64388628677d18e799e9b5a05fe6538aaa17bc79 to fix CVE-2026-2673.
• Upgrade Bento to v1.16.1 to get Datadog and Bigtable outputs support.
• Query engine: stop clamping query tail offsets from cached stream metadata so Event Explorer queries do not intermittently miss recent time buckets.
• Add Datadog-compatible HTTP log intake endpoints on Agents so Datadog log batches can be routed into WarpStream topics.
• Add a zstd produce fallback decoder path so Agents can accept some batches that fail github.com/DataDog/zstd with unexpected EOF but succeed with klauspost/compress/zstd.

Release v771

March 19, 2026

• Fix the metrics prefix pushed by the agent in metrics mode: in previous versions we would add back a “warpstream_” prefix to all metrics, which was redundant.

Release v770

March 18, 2026

• Fix: re-include nc in the warpstream-agent image so that healthchecks can be performed properly on ECS.

Release v769

March 18, 2026

• Tableflow:
  • Add BigLake Metastore integration via Iceberg REST Catalog API for GCP data lake tables.
  • Add Hive Metadata Store integration

Release v768

March 12, 2026

• Disable GCS direct connectivity client by default as it is causing a regression in some GCP regions.

Release v767

March 11, 2026

• Fix tls_insecure_skip_verify not being applied for TLS+SASL connections without mTLS in Tableflow and Orbit source clusters.
• Prevent Agents from trying to issue file cache requests against Agents that no longer exist when fetching old file extents by enforcing a maximum cache staleness in the activation cache.

Release v766

March 11, 2026

• Bump build to go 1.26.1
• Tableflow:
  • Enhance logging when metadata upload fails.
  • Add a module attribute to job logs to group them logically.
  • Fix the command to query the Tableflow table logged by the demo agent.
• Fail agent startup if the ripcord mode is used for an agent connecting to a Tableflow or a Schema Registry cluster.
• Add observability by forwarding agent produce error outcomes.
• Switch agent Docker base images from Alpine to Chainguard Wolfi https://images.chainguard.dev/directory/image/wolfi-base/overview
• Misc: New way of releasing static agent features, this should be a transparent change.
• Add create-acls, describe-acls, and delete-acls clibeta commands for managing ACLs from the CLI, providing a kafka-acls.sh equivalent.

Release v765

March 5, 2026

• Tableflow: Add retries to the bucket access checker and remove extra slash used in the request URL.
• Add agent_version label/tag to all metrics emitted by the agents.
• Bump github.com/docker/cli to v29.2.1 for CVE-2025-15558 vulnerability.
• Add L7 proxy detection diagnostic for inter-agent traffic. When a load balancer or reverse proxy is detected between agents, a new high-severity diagnostic is surfaced to help users identify and remove proxy interference.
• Added new OBJECT_STORAGE_PUBLIC_NETWORK_PATH diagnostic that detects when S3 or Azure Blob Storage endpoints resolve to public IPs, indicating a missing VPC endpoint (or equivalent). This helps identify unnecessary networking costs from object storage traffic going over the public internet.
• Fix a bug where some log attributes were grouped into a !BADKEY key.

Release v764

March 3, 2026

• Tableflow: make Protobuf type google.protobuf.Timestamp usable as a custom partitioning field.

Release v763

February 27, 2026

• Increase the maximum allowed value for batchMaxSizeBytes (max uncompressed batch flush size) from 64MiB to 128MiB.
• Improve Schema Registry to properly handle trailing slashes in URL paths (e.g., /config/ now works the same as /config).

Release v762 [do not use]

Note

This version has a known bug that causes some records to be incorrectly skipped by consumers. Do not use it.

February 26, 2026

• Make sure BYOC schema registry response’s Content-Type is application/json if the request’s Accept header is application/json.
• Fixed a bug where the demo agent would get stuck waiting for the Tableflow Iceberg table to be created.
• Optimizes fetch to use less CPU and IO when fetching a single record, or when max.partition.fetch.bytes is low.

Release v761

February 25, 2026

• Add PSI (Pressure Stall Information) metrics collection for CPU, memory, and I/O pressure. New metrics: pressure_cpu_some_pct, pressure_cpu_full_pct, pressure_memory_some_pct, pressure_memory_full_pct, pressure_io_some_pct, pressure_io_full_pct. These metrics report the percentage of time tasks were stalled due to resource contention over the last sampling interval (0-100%).
• Update orbit to cover for protocol incompatibility in some systems which implement kafka api

Release v760

February 24, 2026

• Tableflow: Performance improvement on ingestion saving 5 to 10% of CPU usage.
• Add metrics category to --jobSelector flag, allowing agents to run only metrics publishing jobs (e.g., --jobSelector metrics).

Release v759

February 24, 2026

• Add --jobSelector flag (env: WARPSTREAM_JOB_SELECTOR) to specify which job types an agent handles.

Release v758

February 20, 2026

• Tableflow:
  • Fix ingestion of Avro and JSON when lists or map values contain decimal fields.
  • Make Proto unsigned integers (uint32, uint64, fixed32, fixed64) convert into decimals. Before that, making a Protobuf backward-compatible change that changed a uint32 into a uint64 (or vice-versa) was forbidden because it would end in a non-backward-compatible Iceberg change (from long to string or vice-versa). Now it’s possible. This is a breaking change for Protobuf tables as the Iceberg column type changes for those uints.
  • Add support for google.protobuf.Timestamp type in Protobuf. It gets converted to an Iceberg timestamptz.
• Fix time_since_last_top_level_fetch_ms to avoid negative values when there are concurrent fetch requests.
• Improve the “Cross AZ Kafka Clients” diagnostic to show more than 1 clientId.
• Fix a bug in the instance type detection logic for Kubernetes environments that could cause it to fail to detect the instance type in some cases and return “unknown” instead.
• Add log message for mTLS authentication that includes certificate serial number and validity period in order to help monitor client certificate expirations.
• This functionality is disabled by default and can be controlled with the -logMTLSCertificateInfo flag or WARPSTREAM_LOG_MTLS_CERTIFICATE_INFO environment variable.

Release v757

February 19, 2026

• Added a new agent flag -injectPrincipalHeader (env var WARPSTREAM_INJECT_PRINCIPAL_HEADER) that, when set to a header key name, injects the kafka principal username as a header in each produced record. This allows downstream consumers to identify which user produced each record.
• Fixes a bug in the ripcord deletion queue that could cause the “last opened sequence” to grow until the agent is restarted, because the agent silently discards files that should be deleted in some circumstances.
• Query engine: Fix a bug that caused fields escaped with backticks to not be queryable properly.
• Tableflow:
  • Emit the correct logical type annotation for time fields in Parquet files.
  • Fix ingestion of Avro records with decimal fields.

Release v756

February 17, 2026

• Reject all traffic to HTTP fetch endpoints if mTLS is enabled on the Agents.
• Validate SASL credentials in HTTP fetch endpoints even if SASL authentication is not required by the Agents.
• Tableflow: fix regression requiring the IngestionBucketURL to be set for datalake clusters (new unreleased feature).

Release v755

February 12, 2026

• Add new /v1/kafka/fetch HTTP/JSON endpoint for manually issuing Fetch requests against the Agents.
• Add convenience endpoint GET /v1/kafka/fetch_single_record?topic=TOPIC&partition=PARTITION&offset=OFFSET for fetching a single record by offset.
• Add convenience endpoint GET /v1/kafka/topics/{topic}/partitions/{partition}/records/{offset} as a REST-style alternative for fetching a single record by offset.
• Due to false positives, skip the Unregistered WarpStream Environment Variables diagnostic if the agents is deployed as a Kubernetes service with a name that begins with “warpstream_” or “warpstream-”.

Release v754

February 12, 2026

• Fixed a bug introduced in v746 where the Agents could get deadlocked after experiencing tends of thousands of errors.

Release v753

February 11, 2026

• [experimental] Add -disableOutOfRangeOrbit flag (env: WARPSTREAM_DISABLE_OUT_OF_RANGE_ORBIT): when enabled, orbit-managed topics return KAFKA_STORAGE_ERROR instead of OFFSET_OUT_OF_RANGE in the fetch path, preventing clients from resetting offsets when data has not yet been replicated from the source cluster.

Release v752

February 11, 2026

• Redact agent key from startup logs.
• Tableflow: Fix rare bug in parquet footer length tracking that could cause compaction jobs to fail with “unsupported thrift type” errors when reading input files.

Release v751

February 9, 2026

• Fix default value for -batchMaxCompressedSizeBytes to be 16MB instead of a value higher than the max.

Release v750

February 9, 2026

Warning: There is a problem with that version fixed in v751. Please upgrade to v751 directly and skip v750.

• Flushing behaviour update
  • Before v750, the agent would flush a new file if it was over your configured -batchMaxSizeBytes value, and its default value was 4MiB. This means that the agent would not create files that contain more than 4i of uncompressed data by default.
  • After this release, by default, the Agent will create a new file if it estimates that it’s going to be more than 1MiB after compression by default.
  • In more details:
    • This version introduces a -batchMaxCompressedSizeBytes flag (and the corresponding WARPSTREAM_BATCH_MAX_COMPRESSED_SIZE_BYTES environment variable).
    • If you define just -batchMaxSizeBytes, then -batchMaxCompressedSizeBytes will not be used, the agent will use exactly your uncompressed bytes limit to choose when to create a new file.
    • If you define just -batchMaxCompressedSizeBytes, then -batchMaxSizeBytes will be automatically set to a very high value (64MiB)
    • If you define no flags, then -batchMaxCompressedSizeBytes will be set to 1MiB and -batchMaxSizeBytes will be set to 64MiB.
    • If you define both, a new file will be created when it becomes higher than either limit.
• SyncGroup handler now returns validation errors (invalid group ID, missing member ID, etc.) embedded in the response instead of as a top-level error. This improves compatibility with Kafka clients and ensures these expected protocol errors are not logged/metriced as server errors.
• Tableflow:
  • Fix BigQuery integration bug, where orphaned tables were preventing table recreation.
  • Update ingestion to handle empty topics gracefully
  • Fix timestamp field indexing for JSON format: timezone is not required anymore for fields with type timestamp.
• Fix bug in batcher which would increase the number of files flushed when disable_only_flush_after_timeout is set to true.
• Include virtual cluster ID in auto-generated client IDs for Managed Data Pipelines. This ensures it works with VirtualClusterIDClientStrictValidation enabled.
• Improve performance of metrics code by swapping a mutex for an RWMutex.
• Bump franz-go package from v1.18.1 to v1.20.6.
• Add low severity diagnostic to detect misspelled environment variables beginning with WARPSTREAM_.
• Fix noisy ACL denial logs for “all topics” metadata requests. When clients refresh metadata for all topics, unauthorized topics are now silently filtered without logging denials (per Kafka protocol behavior).

Release v749

February 2, 2026

• Tableflow: Support skip_raw_record_values that doesn’t store the raw record values in the produced data files.
• Bump build to go 1.25.6 to fix vulnerability CVE-2025-61726.

Release v748

January 30, 2026

• Return an error when MTLS is enabled (without -spiffeMTLSAuthentication) but the agent is unable to determine the username from the client certificate subject.
• Add a new push diagnostic that fires when the timeout on produce requests is less than 5 seconds. This helps identify clients with misconfigured timeout settings that may cause issues with WarpStream’s object storage-based architecture.
• Allow clients to reauthenticate via SASL even when the control plane is down for Ripcord mode.
• Tableflow:
  • Add support for custom partitioning.
  • Add support for compressing data files, default being snappy.
  • New metrics:
    • warpstream.tableflow_partitions_count: Total number of table partitions in the cluster.
    • warpstream.tableflow_partitions_limit: Maximum number of table partitions allowed in the cluster.

Release v747

January 29, 2026

• Buffer audit logs agent side in case of failure to publish them.
• Patch CVE-2025-15467 & CVE-2025-69419 vulnerabilities in libssl3/libcrypto3.
• Tableflow:
  • Fix decoding of JSON records with date fields serialized in the “YYYY-MM-DD” format.
  • Update how we track source data timestamp metadata (kafka timestamps) for more efficient retention enforcement.

Release v746

January 27, 2026

• Flushes will wait for the entire batch timeout even if the batch max size bytes limit is exceeded. Files will be split and flushed in parallel in accordance to the batch max size bytes limit.

Release v745

January 23, 2026

• Add support for SASL OAUTHBEARER authentication mechanism in the WarpStream Agent for Kafka clients that support it.
• New flags and environment variables:
• -saslOAuthIssuerURL / WARPSTREAM_SASL_OAUTH_ISSUER_URL: The OAuth issuer URL used to validate tokens presented by clients.
• -saslOAuthAudience / WARPSTREAM_SASL_OAUTH_AUDIENCE: The expected audience claim in the OAuth tokens presented by clients.
• When these settings are configured, the Agent will accept OAuth tokens from clients and validate them against the specified issuer and audience.
• ACL rules can be created with OAuth principals in the format User:<subject> to allow or deny access based on the subject claim in the token.
• Tableflow: Add support for Protobuf schemas.

Release v744

January 21, 2026

• Fix issue with lightning topics configuration validation in previous versions that would prevent the agents from starting on some machines (introduced in v737)
• Tableflow: Add memory safeguard to prevent potential OOM when ingesting data spanning many partitions.

Release v743

Caution

This version of the agent panics at startup on some machines with a high CPU count. Upgrade to v744 or higher if you have this problem. The exact error message is

panic: error starting agent: error validating config: error validating ripcord config: error validating delete queue config: QueueSize must be less than or equal to 1M, it will use too much memory

January 20, 2026

• Tableflow: Optimize ingestion by seeking to offsets that correspond to records to speed up the ingestion of topic with large offset gaps in the beginning.
• Returns the “invalid topic” error rather than the “kafka storage error” when attempting to fetch from a topic name that does not exist. This is usually gated from the kafka client though.
• The WarpStream agent can now create WarpStream lightning topics.
• The WarpStream agent can now be started in ripcord mode, with -enableRipcord or by setting the WARPSTREAM_ENABLE_RIPCORD environment variable to true. See the documentation for more details.

Release v742

Caution

This version of the agent panics at startup on some machines with a high CPU count. Upgrade to v744 or higher if you have this problem. The exact error message is

panic: error starting agent: error validating config: error validating ripcord config: error validating delete queue config: QueueSize must be less than or equal to 1M, it will use too much memory

January 19, 2026

• Add additional attributes i.e. client_id and username to shadow ACL logs.
• Enable SASL Authentication automatically for Managed Data Pipelines when ACL shadowing is enabled.
• Put back tag virtual_cluster_id and virtual_cluster_name to agent_kafka_request_outcome metric that got removed by mistake in v740.
• Fix bug in ACL Shadowing to stop emitting false-positive deny diagnostics.

Release v741

Caution

This version of the agent panics at startup on some machines with a high CPU count. Upgrade to v744 or higher if you have this problem. The exact error message is

panic: error starting agent: error validating config: error validating ripcord config: error validating delete queue config: QueueSize must be less than or equal to 1M, it will use too much memory

January 15, 2026

• Update playground url to point to the api.warpstream.com instead of console.warpstream.com.
  • Please upgrade to this release for playground functionality to continue working.
• Rename -consoleURL to -apiURL flags in playground mode.

Release v740

Caution

This version of the agent panics at startup on some machines with a high CPU count. Upgrade to v744 or higher if you have this problem. The exact error message is

panic: error starting agent: error validating config: error validating ripcord config: error validating delete queue config: QueueSize must be less than or equal to 1M, it will use too much memory

January 14, 2026

• Downgrade gosnowflake dependency to remove glibc error related to cgo execution.
• Fail to start tableflow agents if incorrect roles are provided / configured.
• Add warpstream_agent_acl_denied metric to track ACL denials.
• Treat more kafka server errors as canceled instead of error if the errors are indicative of the client disconnecting.

Release v739

Caution

This version of the agent panics at startup on some machines with a high CPU count. Upgrade to v744 or higher if you have this problem. The exact error message is

panic: error starting agent: error validating config: error validating ripcord config: error validating delete queue config: QueueSize must be less than or equal to 1M, it will use too much memory

Note: This release contains a glibc/cgo error due to the Bento upgrade. Please use v740 instead.

January 12, 2026

• Return COORDINATOR_LOAD_IN_PROGRESS instead of KAFKA_STORAGE_ERROR as the generic error code for consumer group coordinator RPCs (JoinGroup, SyncGroup, Heartbeat, LeaveGroup, DescribeGroups, DeleteGroups, ListGroups, OffsetDelete, OffsetFetch) and TxnOffsetCommit. This improves compatibility with the Java consumer client which will automatically retry COORDINATOR_LOAD_IN_PROGRESS errors for these RPCs, but not KAFKA_STORAGE_ERROR.
• Tableflow: Fix BigQuery integration to handle table recreation. When a datalake table is recreated with a new UUID, the BigQuery external table is now automatically dropped and recreated to point to the new metadata location. Previously, updates would fail because BigQuery validates that both old and new metadata files exist.
• Merge batcher related metrics into single metrics with a name tag:
  • warpstream.xxx_batcher_batches_count -> warpstream.batcher_batches_count
  • warpstream.xxx_batcher_batches_distribution -> warpstream.batcher_batches_distribution
  • warpstream.xxx_batcher_called -> warpstream.batcher_called
• Fix bug in Fetch logic that could cause massive amounts of overfetching if clients (like librdkafka) sent fetch requests where the value of partition max bytes was >> the value of fetch max bytes for the entire request.
• Reduce default value of kafkaMaxFetchRequestBytesUncompressedOverride and kafkaMaxFetchPartitionBytesUncompressedOverride from 1GiB to 256MiB. Now that the fetch code has pre-fetching logic, requesting huge amounts of data in a single fetch request is not nearly as useful so we should not allow it by default.
• Upgrade Bento to v1.14.1 to fix a regression where functions with optional arguments were not being honored.

Release v738

Caution

This version of the agent panics at startup on some machines with a high CPU count. Upgrade to v744 or higher if you have this problem. The exact error message is

panic: error starting agent: error validating config: error validating ripcord config: error validating delete queue config: QueueSize must be less than or equal to 1M, it will use too much memory

January 5, 2026

• Add describe-broker-configs, describe-groups, describe-log-dirs, and describe-topic-configs commands to the warpstream agent clibeta command for inspecting broker, consumer group, log directory, and topic configuration respectively.
• Add client_az tag to metrics warpstream_agent_kafka_fetch_compressed_bytes_counter and warpstream_agent_kafka_produce_compressed_bytes_counter. The client_az tag prioritizes using the availability zone set in the client ID, otherwise it uses the resolved client ID from the Agent’s subnet mapping. If neither is provided, then a value of none is set.
• Bump github.com/eclipse/paho.mqtt.golang to fix CVE-2025-10543

Release v737

Caution

This version of the agent panics at startup on some machines with a high CPU count. Upgrade to v744 or higher if you have this problem. The exact error message is

panic: error starting agent: error validating config: error validating ripcord config: error validating delete queue config: QueueSize must be less than or equal to 1M, it will use too much memory

December 22, 2025

• Ignore GCS context canceled error in circuit breakers to prevent context cancelation from accidentally opening circuit breakers inappropriately.
• Tableflow:
  • Add BigQuery integration support for external Iceberg tables. Tables can now be automatically created and updated in BigQuery when new metadata is committed.
  • Add “stop” DLQ mode for Tableflow, which blocks ingestion of invalid records instead of skipping them. “skip” mode will remain the default for now, but a later release will make “stop” the default.

Release v736

December 17, 2025

• Treat context.Canceled errors in ACL code as WARN instead of ERROR since it just indicates that the client disconnected, not that anything is wrong.
• Add pipeline_name tag to Managed Data Pipeline metrics
• Emit logs for shadow ACL denials.

Release v735

December 11, 2025

• Add support for spiffe URIs in mTLS authentication between kafka clients and ACLs
• Add support for spiffe URIs in ACLs
  • ACLs can now be created with spiffe URIs as the principal
    • User:spiffe://example.org/service to allow a specific trust domain and workload ID.
    • User:spiffe://example.org/* to allow a specific trust domain and any workload ID.
  • spiffe super users can be created by creating cluster credentials with spiffe URI as the name.
• Improve the logic for determining when to delay partition assignment changes for individual clients.
• Fix ACL handling for ANY resource type.

Release v734

December 9, 2025

• Add support for internal SPIFFE trust domain and workload ID for mTLS authentication between agents.
  • Trust domain and workload ID can be specified via new flags (-internalSpiffeTrustDomain and -internalSpiffeWorkloadID) and environment variables (WARPSTREAM_INTERNAL_SPIFFE_TRUST_DOMAIN and WARPSTREAM_INTERNAL_SPIFFE_WORKLOAD_ID).
  • If set, certificates without the specified trust domain and workload identity will be rejected.

Release v733

December 8, 2025

• Tableflow:
• Write always null count statistics, even if they are zero, in the parquet data statistics, to make it work with Query Engines that have strict requirements of reading those statistics.
• Bump build to go 1.25.5 to fix vulnerability CVE-2025-61729.

Release v732

December 3, 2025

• Improve logic for controlling delays between polling for new records in the Fetch code. This change dramatically reduces P99 E2E latency for some workloads.
• Upgrade Bento version to v1.13.1 in order to include fix on parquet_encode processor: it now allows for column names starting with an underscore.

Release v731

November 28, 2025

• Adds a -enabledSASLMechanisms flag in the agent (and a corresponding WARPSTREAM_ENABLED_SASL_MECHANISMS environment variable).
  • If you provide it with a comma-separated list of SASL mechanisms, only those will be enabled. Valid values are PLAIN and SCRAM-SHA-512.
  • For example, if you set WARPSTREAM_ENABLED_SASL_MECHANISMS=SCRAM-SHA-512, you will not be able to use the PLAIN mechanism to connect, only SCRAM-SHA-512.
• Fixes a bug preventing the agent from starting successfully on hosts with IP v6 only (assuming they start with -advertiseHostnameStrategy=auto-ip6 or setting the env var WARPSTREAM_ADVERTISE_HOSTNAME_STRATEGY=auto-ip6)
• Tableflow:
  • Fix AVRO record decoding for optional nested structs. This resolves validation errors like “missing required field” that could occur during ingestion when using AVRO schemas with optional nested records.

Release v730

November 26, 2025

• Prevent partition assignments for each client from being able to change on every Metadata refresh. This dramatically improves load-balancing behavior when many clients have synchronized Metadata refresh intervals.
• Tableflow:
• Add support for arbitrary partition transforms during ingestion.
• Make transforms work with Avro encoded data.
• Validate ‘required’ fields are present for JSON records during ingestion.

Release v729

November 24, 2025

• Tableflow: fix a bug in data cleanup where some data files that were compacted away or out of retention weren’t being removed from the blob storage correctly.
• Fix the agent_roles metric tag to be deterministic.

Release v728

November 21, 2025

• Perform ACL shadowing when ACLs are configured but disabled, and surface the result as a diagnostic. This helps users detect and fix invalid ACL rules even if ACLs are not currently enforced.
• Use much more up to date Agent load information in the partition assignment strategies that use consistent hashing. This should dramatically improve load balancing for those strategies as previously the information that was used could be up to 1m stale and now it should never be more than a few seconds stale.
• Add support for custom Bento transforms in Tableflow ingestions jobs.
• Add support for dropping records as part of custom Bento transforms in Tableflow ingestion jobs.
• Update demo command to demonstrate custom Bento tranforms for Tableflow.
• Reduce log level from ERROR to WARN for failure to upload a profile.
• Treat “connection timed out” errors when reading requests from Kafka client connections the same as “idle connection closed” from a logging perspective (reduces error log spam).
• Reduce log level from ERROR to WARN when background prefetches fail.

Release v727

November 19, 2025

• Re-enable watching for changes to the bucket URLs so that the Agents can refresh them if they change through overrides on the cluster settings page.
• Return COORDINATOR_LOAD_IN_PROGRESS instead of KAFKA_STORAGE_ERROR as the generic error code for the OffsetCommit RPC. This improves compatibility with the Java consumer client which will automatically retry COORDINATOR_LOAD_IN_PROGRESS errors for that RPC, but not KAFKA_STORAGE_ERROR.

Release v726

November 18, 2025

• Enable a stricter check when warpstream_agent_group/ws_agent_group/ws_ag is set in the client. If an Agent in group A receives a request from a client that indicated its intended target is Agents in group b, then the Agent in group A will reject the request with an error before closing the connection. This prevents issues that can occur where clients end up connected Agents in the wrong group due to IP reuse in high-churn environments like Kubernetes
• Only consider internal errors in the control_plane_errors agent diagnostic.
• Bump github.com/dvsekhvalnov/jose2go to 1.8.0 for CVE-2025-63811 vulnerability.
• Add gated support to ignore flush size in the batcher until flush timeout is hit.

Release v725

November 14, 2025

• Enable efficient consumer group rebalances, by not sending member’s metadata multiple times for a single JoinGroup request.
• Add -disableAzLookupWarnings flag and WARPSTREAM_DISABLE_AZ_LOOKUP_WARNINGS environment variable to disable warnings when availability zone lookup fails via CIDR blocks. Additionally, these warnings are now throttled to a maximum of 1 log per minute to prevent log spam when clients intentionally connect from outside configured CIDR ranges while using the WARPSTREAM_ZONED_CIDR_BLOCKS configuration.
• Attempt to use GCS direct connectivity (gRPC) automatically with GCS buckets by default, and fallback to standard HTTP when its unavailable. This significantly reduces the P99/max latencies of object storage operations in GCP.
  • Also added a new flag to fail the Agent on startup if they can’t establish direct connectivity on GCP eligible datacenters: gcsDirectConnectivityRequired / WARPSTREAM_GCS_DIRECT_CONNECTIVITY_REQUIRED
• Fix the time_since_last_top_level_fetch_ms attribute in sample_fetch_statistics debug log which is inaccurate when the fetch is performed during prefetching. Also adds is_prefetch to the sample_fetch_statistics.
• Improve performance of metadata handler by switching to a faster api to list streams.
• Unregister agent from service discovery during graceful shutdown.
• Add diagnostic to detect when an agent does not shutdown cleanly.
• Disable prefetching when the duration between individual consumer fetch requests is longer than the period of time that prefetched results will be held in memory before being GC’d if they’re not consumed. This helps prevent the prefetching logic from putting additional load on the Agents when they’re already overloaded.
• Auto-tune the topic metadata cache refresh interval based on the number of topics in the cache so that workloads with a high number of Agents and topics do not generate excessive amounts of HTTP traffic.
• Tableflow: Migrate compaction job to stop using a deprecated version of an internal API.

Release v724

November 6, 2025

• Add support of a new warpstream_cluster_id client ID (see documentation).
• Add a new enableConfluentComponents setting to enable the use of confluent cloud connectors with warpstream clusters.
• Add new default concurrency limit of 16 concurrent Metadata requests/vCPU. This limit can be changed using the -maxConcurrentMetadataRequestsPerCPU flag or WARPSTREAM_MAX_CONCURRENT_METADATA_REQUESTS_PER_CPU environment variable. Setting the value to zero disables the concurrency limit.
• Improve partial error handling in the batcher.
• Bump build to go 1.25.3 to fix vulnerability CVE-2025-58187.
• Configure maximum staleness in the AssumeRole cache so that role tokens are refreshed proactively in the critical path after long periods of idleness instead of just passively in the background after being read. This prevents spurious api error BadRequest: Bad Request errors from occurring during some infrequent background processes like file cleanup.
• Bump github.com/opencontainers/runc to 1.3.3 for CVE-2025-31133 and CVE-2025-52565 vulnerabilities.

Release v723

November 4, 2025

• Bump build to go 1.25.2 to fix vulnerability CVE-2025-47912.
• Reduce CPU and allocations when processing Metadata requests for specific topics in clusters with a large number of topics/partitions.
• Fix the metric agent_kafka_produce_records_counter to have the right tags when EnableHighCardinalityMetrics is enabled.

Release v722

October 31, 2025

• Added a cache to AWS assume role credentials to prevent rate limiting in large workloads.
• Tableflow: Improve ingestion telemetry reporting.
• Schema registry: Add support for Protobuf.
• Schema registry: Add support for schema normalization.

Release v721

October 29, 2025

• Tableflow: Reduces agent memory usage during table compaction by using buffers pooling.
• Fix a bug that would sometimes cause the Agents to reject Produce requests during shutdown with errors like “error getting allocated file ID: context deadline exceeded: 10.002999653s” that would result in latency spikes and error log spam.
• Add a diagnostic to detect when the Agent file descriptor limit is less than the configured connection limit.
• Add the ability to use the Bento opensnowcat processor in managed data pipelines.

Release v720

October 27, 2025

• Bump Bento to v1.12.1 which does proper error handling for GCP BigQuery output.
• Auto tune TCP buffer size based on observed number of connections. This makes it so well behaved workloads with low connection counts benefit from large buffers and minimal syscalls, but workloads with extremely high connection counts work too without requiring excessive amounts of memory.
• Increase maximum allowed message size from 128 MiB to 256 MiB.
• Change the log level of “successfully fetched credentials for assume role provider” from info to analytics
• Split produce batch into per shard batches in the batcher.

Release v719

October 23, 2025

• Tableflow: Fix ingestion of json records when the json key contains non-alphanumeric characters.
• Reduce the amount of polling/networking done between the Agents and control plane for Agents running in demo and playground modes
• Increase default maximum number of connections per vCPU from 8_192 to 32_768.
• WARPSTREAM_MAX_PRODUCE_RECORD_SIZE_BYTES can now support record size from 1 MiB to 128 MiB with default at 32 MiB.
• Fix a bug in the loading cache that may return transient invalid results, resulting in errors such as “stream metadata cache returned not exist which should never happen”.
• Turn current_buffer_stats log into a debug log.

Release v718

October 21, 2025

• Fix JSON Schema Validation to handle the $defs keyword for draft version 2020-12.

Release v717

October 16, 2025

• Tune number of allowed retries for blob store operations during compactions slightly.
• Add a cache that makes partition assignment strategies that use consistent hashing much more performant when the volume of Metadata requests is extremely high.
• Increase default number of inflight files for Producing from 16/vCPU to 64/vCPU. This makes it much less likely that the Agents will start backpressuring due to hitting the inflight files limit before hitting the inflight bytes limit which is much more important and accurate.
• Increase the default ratelimit for max processed bytes per second/vCPU from 50MiB/s to 100MiB/s for Agents running only the pipelines role to promote more CPU usage and better auto-scaling.

Release v716

October 14, 2025

• Reduce the amount of observability metadata we send back to the control plane on fetch and produce.
• Changed default Tableflow HTTP port from 10001 to 8081 in demo/playground mode.
• Upgrade the embedded version of bento to v1.11.0
• Tableflow:
  • Made Tableflow HTTP port in demo/playground mode overrideable using the -tableflowInternalHTTPPort flag.
  • Add a TABLEFLOW_ prefix to the environment variables for passing cluster credentials.

Release v715

October 13, 2025

• Accept JSON Schemas with draft versions 04, 06, and 07 for schema validation.

Release v714

October 13, 2025

• Improve the performance of GCS client when direct connectivity is enabled by reducing the number of in-memory clients/connections that are in use at once.
• Add new diagnostic for when a consumer gets an offset out of range error.
• Bump our Alpine base image to version 3.22.2 to fix vulnerability CVE-2025-9230

Release v713

October 10, 2025

• Bump our Alpine base image to version 3.22.1 to fix vulnerability CVE-2025-9230

Release v712

October 8, 2025

• Fix spanner data plane bug.

Release v711 (this release introduced a spanner data plane bug which has been fixed in 712)

October 8, 2025

• Improve Spanner data plane performance.
• Make warpstream playground command use less CPU when the Agent is idle (no traffic).
• Improve the accuracy of the CPU utilization as measured by the Agents.

Release v710

October 6, 2025

• Tableflow: Add AWS Glue Data Catalog integration to automatically register Iceberg tables and update metadata location.

Release v709 (backwards incompatible metrics change)

October 3, 2025

• Add support for Spanner and SQLite as a blob storage backends through spanner:// and sqlite:// URLs.
• Tableflow: Fix indexing of uuid types.
• Disable emitting high cardinality per-topic distribution/histogram metrics by default. See the new flag/environment variable added below to re-enable these metrics. Impacted metrics:
  • warpstream_agent_kafka_produce_with_offset_uncompressed_bytes_bucket
  • warpstream_agent_kafka_produce_uncompressed_bytes_bucket
  • warpstream_agent_kafka_produce_compressed_bytes_bucket
  • warpstream_agent_kafka_produce_uncompressed_bytes_bucket
  • warpstream_agent_kafka_produce_compressed_bytes_bucket
  • warpstream_agent_kafka_fetch_uncompressed_bytes_bucket
  • warpstream_agent_kafka_fetch_compressed_bytes_bucket
• Add new kafkaHighCardinalityDistributionMetrics and WARPSTREAM_KAFKA_HIGH_CARDINALITY_DISTRIBUTION_METRICS environment variable that when set to true enables emitting high cardinality per-topic distribution metrics that are 10-20x higher cardinality than the regular counter metrics. Defaults to false.
• Fix a bug in the pre-fetcher that would sometimes generate invalid/corrupt responses for fetch requests that requested data for a single-topic partition when the client retried a failed fetch request while the Agent was under extremely heavy load. The prefetcher is remotely disabled in previous versions, so this bug doesn’t affect any version.
• Return ErrNotLeaderOrFollower error instead of InvalidRequest error when processing a Produce or Fetch request on an Agent that is only running the proxy-consume or proxy-produce roles respectively. This should help some clients refresh their metadata and get routed to the right Agent more quickly when they’re connected to the wrong Agent role due to I.P address reuse.
• Add new environment variables WARPSTREAM_GCS_GRPC_CONNECTION_POOL_SIZE and WARPSTREAM_GCS_ALLOW_DIRECT_CONNECTIVITY to control new experimental GCS direct connectivity feature.

Release v708

September 29, 2025

• Add subject version attribute to schema registry log request.

Release v707

September 25, 2025

• Make bucket URLs overridable through the Cluster Settings page on the console, to swap out faulty data planes without having to restart the agents.

Release v706

September 24, 2025

• Fix bug which prevented fetch size auto tuning when prefetching.
• Tolerate leading/trailing quotes and whitespace in client IDs when parsing client ID features.
• Add WARPSTREAM_LOOKUP_AVAILABILITY_ZONE_ID environment variable that looks up the availability zone ID instead of availability zone name when running Agents in AWS. This is useful for preventing inter-zone networking fees in WarpStream clusters where the Agents and clients are running in different AWS accounts.
• Improved the diagnostic that detects when Agents are split into different proxy roles, but Kafka clients haven’t been configured to target a specific proxy role.
• When registering an incompatible schema in the schema registry, the error response now contains the list of violations explaining why it was rejected.
• Tableflow: Add support for compacted and transactional topics.

Release v705

September 19, 2025

• Add -schemaRegistryEnableLogRequest flag and WARPSTREAM_SCHEMA_REGISTRY_ENABLE_LOG_REQUEST environment variable that enables the schema registry to log every request it receives.

Release v704

September 18, 2025

• Multi-region: Add support for the “Reduced Quorum Required Bucket” setting, which, on dual-bucket data planes running in a degraded (reduced quorum) mode, specifies which bucket must succeed when writing records to object storage. This improves the resiliency of dual-bucket data planes running in a degraded mode.
• Tableflow: Fix a bug where certain records could be ingested twice when there is a large amount of uningested records.
• Diagnostics metrics: update warpstream.diagnostic_failure labels
  • Replace single severity tag with boolean tags: severity_low, severity_medium, severity_high, severity_critical.
  • For successful diagnostics the severity_* tags are all false; for failing diagnostics exactly one matching severity_* tag is true.

Release v703

September 17, 2025

• Add -jobsRunnerSlotsAmplification flag and WARPSTREAM_JOBS_RUNNER_SLOTS_AMPLIFICATION environment variable that can be used to increase the number of concurrent jobs that the Agents will run per vCPU. Defaults to 1.0.
• Treat “i/o timeout” as a resumable error during compactions. This prevents an entire compaction from failing due to a single killed connection.
• Add reduced quorum support for dual-bucket data planes, so they can operate in a degraded mode during a regional incident.

Release v702

September 16, 2025

• Bump Bento version to v1.10.1.
• Emit a warning log if writing responses takes too long due to head of line blocking or slow TCP throughput.
• Improved the logic in the job runner during Agent shutdowns so that Agents always notify the control plane that they weren’t able to complete running a job before exiting so that the control plane can reschedule the job on another Agent without waiting for the full timeout.
• Tableflow: Enhance Agent telemetry for investigating ingestion issues.
• Add new /debug/speedtest/download endpoint to Agent HTTP server for measuring the maximum rate at which an Agent is able to transfer bytes over the network to clients.

Release v701

September 12, 2025

• Make error message in diagnose-connection command more clear that the -tls flag may be missing.
• Detect when TLS-related flags are set in CLI, but -tls or -enable-tls is not set and return an error.
• Diagnostics metrics: standardize labels and binary values for easier querying
  • diagnostic_name is now lower_snake_case; diagnostic_type is lowercase
  • Deprecated: warpstream.diagnostic_status (kept temporarily for compatibility). New metric: warpstream.diagnostic_failure (gauge is 1 when failing and 0 when successful). The old metric will be removed in a few versions.

Release v700

September 11, 2025

• Fix a bug where usages of the prefix query parameter in the bucket URL without a trailing / would be transformed incorrectly.
• Fix a bug where the WARPSTREAM_DISABLE_S3_CHECKSUMS environment variable isn’t respected for multi-part uploads.

Release v699

September 10, 2025

• Fix internal bug which accidentally limits partition fetch concurrency.
• Make consistent_random_jump the new default partition assignment strategy. You can set the environment variable WARPSTREAM_DEFAULT_PARTITION_ASSIGNMENT_STRATEGY to single_agent to revert to the previous default behavior.
• Tableflow: Improve timestamp conversion when indexing JSON messages and fix indexing of date types for Avro messages.

Release v698

September 8, 2025

• Fail to start if a private IP address to advertise cannot be detected in auto-ip4 mode instead of defaulting to localhost.
• Include the remote network address (IP and port) in diagnostics that identify cross-availability zone Kafka clients.
• Fix a bug where setting WARPSTREAM_DISABLE_S3_CHECKSUMS to true doesn’t really disable checksumming.

Release v697

September 4, 2025

• Add WARPSTREAM_DISABLE_S3_CHECKSUMS environment variable that when set to true disables checksumming in the S3 client to improve compatibility with some S3-compatible object stores that are not actually S3 and haven’t been updated yet to support S3’s latest checksumming scheme.
• Fix bug where the agent was unable to start in multiregion mode when using static metadata URLs.

Release v696

September 3, 2025

• Fix schema registry GET /subjects/(string: subject)/versions/(versionId: version)/schema and GET /schemas/ids/{int: id}/schema endpoints to return properly formatted schema.
• Enable single partition prefetching by default. This behavior can be disabled by setting the environment variable WARPSTREAM_ENABLE_PREFETCHING_FOR_SINGLE_PARTITION_WORKLOADS=false.
• Adds support for Orbit fetch v2 for increased Orbit performance.

Release v695

September 3, 2025

• Reverted: “Increased the default value of maximum inflight fetch compressed bytes allowed for all role configurations by 50% (due to the new more efficient fetch code path).”
• Improved prefetching logic for single partition workloads to result in 99% hit ratios in most reasonable use cases.
• Updated default of -gracefulShutdownDuration & WARPSTREAM_GRACEFUL_SHUTDOWN_DURATION to 300s (5 minutes)
• Added the WARPSTREAM_HTTP_PROXY environment variable to allow configuring a HTTP proxy for the Agent that will act only on *.warpstream.com requests rather than using HTTP_PROXY which would act on all requests.

Release v694

August 27, 2025

• Split internal agent to agent listener to it’s own port (default 8443) when TLS is enabled.
• Fix ingestion of maps with optional values for Tableflow.

Release v693

August 26, 2025

• Improve accuracy of some distribution metrics, like agent_segment_batcher_flush_file_size_uncompressed_bytes and agent_segment_batcher_flush_file_size_compressed_bytes.
• Fix some metrics to include key tags: virtual_cluster_id, agent_group, and agent_roles.
• Add support for TLS and mTLS for the internal agent to agent communication.
• Upgrade to Bento v1.10.0. With this, aws_dynamodb output also supports deleting items from a dynamodb table.

Release v692

August 25, 2025

• Add a new environment variable WARPSTREAM_ENABLE_PREFETCHING_FOR_SINGLE_PARTITION_WORKLOADS that enables background pre-fetch for consumers that only fetch data for a single partition at a time (like Spark). This dramatically reduces latency (and as a result, improves throughput) for these consumers.
• Fix the client id used by the tableflow agent and orbit for consuming to work with Kafka agents with roles configured.
• Add severity tag to diagnostic metrics (warpstream_diagnostic_status), enabling filtering by severity (low/medium/high/critical).

Release v691

August 21, 2025

• Fix a bug where the schema registry agent doesn’t return the right content type (application/vnd.schemaregistry.v1+json).

Release v690

August 20, 2025

• Fix a bug where the agent occasionally returns validation error (“file has already been created once already”) when a client produces to an agent that hasn’t received produce requests in a while.

Release v689

August 20, 2025

• Dramatically reduce read amplification for some workloads consuming historical data from small topics.
• Bump gocloud.dev dependency to pick up a S3 path-style URLs fix.

Release v688 (This version is incompatible with S3 path-style URLs, which are enabled when s3ForcePathStyle=true is specified in the bucket URL. Please upgrade to v689.)

August 19, 2025

• Bump gocloud.dev dependency to pick up a data race fix.

Release v687

August 18, 2025

• Add read uncommitted support to Orbit agent handler.
• Update logic for reloading the Agent key from a file to fix issue of not reloading when symlinks are used.

Release v686

August 18, 2025

• Fixes a bug in service discovery where agents deployed with different roles could cause no brokers to be returned. Previously, if an availability zone contained agents but none had the required produce/consume/both role, the system would return an empty broker list instead of falling back to other zones. This release resolves the issue by properly implementing zone fallback logic.
• Fixes a bug in the fetch diagnostic that was causing it to fire even when there was no backpressure going on.

Release v685

August 12, 2025

• Enhance Agent telemetry for investigating performance issues.

Release v684

August 12, 2025

• Add new agent flag checkBucketAccessRetryCount (and the WARPSTREAM_CHECK_BUCKET_ACCESS_RETRY_COUNT env variable) to control the maximum times the agent retries object storage permissions checks during startup.
• If a produce request contains topics both owned by Orbit and not, allow the topics not owned by Orbit to produce successfully.

Release v683

August 11, 2025

• Enable new fetch code path that is lower latency and uses significantly less memory. This new fetch code path should also reduce memory usage in consumer client applications. To revert to the old fetch code path set the environment variable WARPSTREAM_FETCH_SINGLE_KAFKA_BATCH=true.
• Fixed a bug where Agents running with the proxy role had 50% less maximum inflight fetch compressed bytes allowed than intended.
• Increased the default value of maximum inflight fetch compressed bytes allowed for all role configurations by 50% (due to the new more efficient fetch code path).
• Increased default value of WARPSTREAM_KAFKA_MAX_FETCH_REQUEST_BYTES_UNCOMPRESSED_OVERRIDE environment variable and kafkaMaxFetchRequestBytesUncompressedOverride flag to 1GiB (due to the new more efficient fetch code path).
• Increased default value of WARPSTREAM_KAFKA_MAX_FETCH_PARTITION_BYTES_UNCOMPRESSED_OVERRIDE environment variable and kafkaMaxFetchPartitionBytesUncompressedOverride flag to 1GiB (due to the new more efficient fetch code path).

Release v682

August 7, 2025

• Enable topics metadata cache by default. This increases the latency of CreateTopic and DeleteTopic requests to ~1s, but dramatically reduces the latency of Metadata operations which improves latency and compatibility with many Kafka clients and tools.

Release v681

August 5, 2025

• Enhance the behavior of the Kafka lag related metrics emitted by the Agent when the topic cardinality is disabled. We would emit the same metric several times with different values, whereas we now emit a single metric with the sum of the lag across all partitions of the topic - or the max for the time lag metric.
• Double the maximum number of outstanding files/bytes that are allowed per vCPU for producing when Agents are running only the proxy-produce and proxy-consume roles in addition to doing it when the Agents are running only the proxy-produce role. This will make it easier to increase CPU usage enough that auto-scaling kicks in with this combination of roles.
• Double the the maximum number of inflight compressed bytes for fetching when Agents are running only the proxy-produce and proxy-consume roles in addition to doing it when the Agents are running only the proxy-consume role. This will make it easier to increase CPU usage enough that auto-scaling kicks in with this combination of roles.
• Use FastRetry when loading directly from object storage without going through the file cache to reduce outlier latency.
• Increase parallelism and reduce chunk size when fetching directly from object storage without going through the file cache. Reduces Fetch latency for large fetch requests.
• Improved object pooling to reduce memory usage for large Fetch requests.
• Add defaults for Bento’s output kafka_franz component: max_buffered_records (1000000, i.e. 1 million records) and max_message_bytes (16MiB). This will prevent rate-limiting the throughput for heavy workloads.

Release v680

July 31, 2025

• Fix cli-beta commands not properly parsing client tls certificates

Release v679

July 30, 2025

• Fix unit of flag -overridePipelinesSharedRateLimitPerVCPU flag and WARPSTREAM_OVERRIDE_PIPELINES_SHARED_RATE_LIMIT_PER_VCPU environment variable: it was incorrectly in bytes/agent/s. Now it is in bytes/vCPU/s.
• breaking metrics change: All metrics on Datadog will now be prefixed with warpstream. instead of warpstream_. You can fall back to the previous behavior by setting the WARPSTREAM_DATADOG_NORMALIZER_PREFIX_WITH_DOT environment variable to false. This change comes along the official release of our Datadog integration, making all the Warpstream Agent metrics free if you install the integration (and use the new naming convention).
• Bump Bento version to v1.9.1 to fix an issue with multipart uploads to S3 outputs.

Release v678

July 23, 2025

• Add jks support for Orbit. See Orbit docs.

Release v677

July 23, 2025

• Import Bento crypto package to allow using the parse_jwt_es256 method in managed data pipelines.
• Add an agent_kafka_fetch_records_counter metric that counts the number of records present in all of the data pages the agent fetched to respond to fetches.
• Tag native Bento metrics emitted by the agent with pipeline_id.

Release v676

July 23, 2025

• Fix the warpstream_agent_kafka_produce_records_counter metric for orbit topics when WARPSTREAM_KAFKA_HIGH_CARDINALITY_METRICS is true.

Release v675

July 14, 2025

• Allow individual fetch requests to fetch up to 1GiB of uncompressed data per topic-partition, per fetch request, increased from 512 MiB.

Release v674

July 14, 2025

• Remove more access control log spam by applying the same rule that was introduced in v673 to certain edge cases.

Release v673

July 11, 2025

• Fix: Cleaned up access control logs. ACL failures for metadata requests are now only logged when a user explicitly queries for a specific topic they are not authorized to see, preventing log spam from general “list all” queries.

Release v672

July 9, 2025

• Use a dedicated context for AWS calls to assume roles to ensure we always allow up to 15s to attempt to assume the role from AWS.

Release v671

July 7, 2025

• Switch to ZSTD storage engine compression by default.
• Add new diagnostic for when Agent backpressure produce/fetch/connections.
• Add new diagnostic for when clients produce to a topic that is still being actively managed / replicated by Orbit.
• Tag all logs with the “agent_id” field.
• Add support for new experimental partition assignment strategies: consistent_spread_bounded_load, consistent_random_jump, consistent_pods.
• Bump Bento version to v1.9.0. As part of the update, gcp_bigquery_write_api can receive messages formatted as protobuf (if specified via message_format).

Release v670

June 30, 2025

• Improve Schema Linking to better handle pipeline recreations.

Release v669

June 26, 2025

• Improve the shutdown lock behavior to unregister from service discovery before acquiring the shutdown lock.

Release v668

June 25, 2025

• Add support for -enableShutdownLock flag and WARPSTREAM_ENABLE_SHUTDOWN_LOCK environment variable that when set to true will force Agents to acquire a lock (per agent group) when shutting down. This is helpful when using clients like Sarama that are extremely sensitive to changes in partition assignments to prevent any disruption as a result of too many partition assignments shifting at once as Agents are rolled / killed.

Release v667

June 24, 2025

• Add support for lookup schema endpoint (POST /subjects/{subject}) for WarpStream’s BYOC Schema Registry. This endpoint checks if a schema has already been registered under the specified subject.
• Automatically detect when Agent CPU usage is high and opportunistically capture CPU/heap profiles to help with debugging.
• Add new warpstream_agent_kafka_produce_records_counter metric that can be used to measure the number of records written in the Agents on a per topic basis when the high cardinality metrics flag is enabled.
• Add new -enableMetadataTopicsCache flag and WARPSTREAM_ENABLE_METADATA_TOPICS_CACHE environment variable that when set to true makes Metadata requests much faster (10s of microseconds on average) at the expense of external consistency (topics created recently may not appear in Metadata requests for up to 1s).

Release v666

June 18, 2025

• Fix bug related to WARPSTREAM_DEFAULT_PARTITION_ASSIGNMENT_STRATEGY being used in combination with consistent_spread value.

Release v665

June 17, 2025

• Batch DescribeGroups requests for applications that aggresively monitor consumer groups.
• Add -defaultPartitionAssignmentStrategy flag and WARPSTREAM_DEFAULT_PARTITION_ASSIGNMENT_STRATEGY environment variable that allows overriding the default partition assignment strategy used for clients with no client-level override. Default value is single_agent (same behavior as before).

Release v664

June 16, 2025

• Restore the previous behavior of ws_pas=equal_spread and introduced a new ws_pas=equal_spread_v2 that handles workloads with a large number of topics with a low number of partitions much better.
• Improve the error message when an agent can’t find it’s own private IP. The error will now log the found IP address and the accepted private IP ranges.
• Add a new partition assignment strategy ws_pas=consistent_spread that uses a consistent hash ring to assign partitions to individual Agents resulting in significantly smoother data processing for large clusters during rolling deployments.

Release v663

June 12, 2025

• Add diagnostic to detect when Kafka Clients issue Fetch Requests with a fetch size bigger than the limit set in the Agent.
• Add flag batchMaxCompressedSizeBytes and environment variable WARPSTREAM_BATCH_MAX_COMPRESSED_SIZE_BYTES to define the max compressed size (similar to the already-existing flag batchMaxSizeBytes and environment variable WARPSTREAM_BATCH_MAX_SIZE_BYTES that allow to define a target based on the uncompressed size. Note: this is not a hard limit, it is based on estimations so the actual compressed size may be a bit higher than the setting.
• Enforce strict mode (i.e. error handling.strategy: reject) when running bento pipelines. This means that Bento will reject all batches containing messages with errors, propagating a nack to the input layer (instead of attempting to send message batches that contain messages with errors to the configured sink).

Release v662

June 9, 2025

• Add support for confluent processors like schema_registry_decode to managed data pipelines.

Release v661

June 9, 2025

• Enable acl deny logging by default.
• Change the behavior of equal_spread partition assignment strategy to handle a wider variety of workloads. Previously it worked well if all topics in the cluster had number of partitions >> than the number of Agents in the cluster, but worked extremely poorly if there were many topics in the cluster with a very small number of partitions.
• Bump Bento version to v1.8.0. As part of the update, the kafka_franz input would force reconnect when it sees an unknown topic/partition error (i.e. when a topic is recreated).

Release v660

June 6, 2025

• Updated cli-beta benchmark

Release v659

June 4, 2025

• Improve how Agents handle Fetch requests when there are partial errors (i.e. some of the partition requests are invalid).
• Improve how Agents preload topic metadata to prevent cold-start problems for Producers that produce to dozens or 100s of topics in a single Produce request.
• Fix bug in BatchMaxSizeBytes validation (max allowed: 16MB).

Release v658

May 30, 2025

• Add diagnostic to detect when Agents have hit connection limit.
• Add metrics for throughput and latency of RPCs between Agents and control plane.
• Bring back the warpstream_error_count metric that corresponds to the number of error logs.
• Be smarter about when to send CPU/Heap profiles back to the control plane to try and capture more interesting profiles.
• Add support for hot reloading TLS server certificates, now certificates can be changed without having to restart WarpStream agents.

Release v657

May 22, 2025

• Diagnostic to detect when Kafka Clients issue Fetch requests with very low timeout.
• Add flag httpsProxyCACertFile and environment variable WARPSTREAM_HTTPS_PROXY_CA_CERT_FILE to support enterprise MITM HTTPS Proxies.
  • If using a MITM HTTPS Proxy users may see this error Post "https://metadata.default.${region}$.${cloud}$.warpstream.com/api/v1/agent/agentpool": tls: failed to verify certificate: x509: certificate signed by unknown authority because the agent doesn’t trust the HTTPS Proxy’s CA. This flag allows the user to set the CA to the internal WarpStream HTTPS client as a trusted certificate authority.
• Add flag tlsProfile and environment variable WARPSTREAM_TLS_PROFILE to change TLS MinVersion, Curves and Ciphers.
  • See the Protect Data in Motion with TLS Encryption documentation for exact details.

Release v656

May 20, 2025

• Fix the diagnostic for unavailable storage bucket: it triggered on some errors that were not bucket errors in AWS.
• Modified the record size limit resolution logic to prioritize in the following order: 1) topic-level max.message.bytes, 2) cluster-level message.max.bytes, 3) agent-level default. This replaces the previous behavior which used the minimum value between topic and cluster limits.
• Allow Orbit to fetch more than 100MiB per kafka fetch request.

Release v655

May 20, 2025

• Increase default value for WARPSTREAM_KAFKA_MAX_FETCH_REQUEST_BYTES_UNCOMPRESSED_OVERRIDE from 128MiB to 256MiB.
• Increase default value for WARPSTREAM_KAFKA_MAX_FETCH_PARTITION_BYTES_UNCOMPRESSED_OVERRIDE from 128MiB to 256MiB.
• Adds a diagnostic that triggers when there are errors talking to the cloud storage bucket (either because the storage bucket is rate limiting reads/writes or because of a temporary problem connecting to it).
• Significantly improve the performance of consumer workloads that are fetching large amounts of compacted data.

Release v654

May 15, 2025

• Cleanly NACK jobs that failed to run during agents shutdown.
• Improve the logic for long-polling fetch requests after detecting that there is no new data for the set of topic-partitions that are being fetched.

Release v653

May 9, 2025

• Added cluster-level (message.max.bytes) and topic-level (max.message.bytes) limits. In case of Warpstream these limits apply to individual records, instead of record batches. This is because OSS Kafka doesn’t process individual records, it processes batches, but WarpStream operates on records directly. So there is no relationship between the batch size sent by producers and the size of batches received by consumers, and therefore, limiting record sizes will prevent consumers from choking on any individual records that are too large.
  • The effective limit is the minimum value across topic-level, cluster-level, and agent-level configurations.

Release v652

May 8, 2025

• Added optional prometheus metrics server for benchmark-producer and benchmark-consumer commands.

Release v651

May 7, 2025

• Added a producer benchmark tool benchmark-producer to the cli-beta commands.
• Added a consumer benchmark tool benchmark-consumer to the cli-beta commands.
• Add a 15s socket deadline for every socket read/write to ensure we detect when goroutines are stuck reading from a bad connection.
• Tune HTTP connection pools to be smaller and dont allow idle connections to remain in the pool for as long.
• Report how long each job has run to the control plane.

Release v650

May 2, 2025

• Add new -tlsBlobURL flag and WARPSTREAM_TLS_BLOB_URL environment variable to load TLS files from a blob store bucket.
• Increase default connection limit per vCPU from 4096 to 8192.
• Add metric warpstream_agent_active_pipeline_instances (tags: pipeline_id) that tracks the number of active pipeline instances run by a given agent.

Release v649

Apr 29, 2025

• Add a cache for api versions. This will reduce excessive rpcs to the control plane.
• When an agent has the “pipelines” role AND any other role, Bento pipelines are rate-limited (default: 5MB/vCPU). When the agent has only the “pipelines” role, the rate limit is higher (50 MB/vCPU).
• Add new -overridePipelinesSharedRateLimitPerVCPU flag and WARPSTREAM_OVERRIDE_PIPELINES_SHARED_RATE_LIMIT_PER_VCPU environment variable to override the default rate limit shared across all pipelines.

Release v648

April 18, 2025

• Fixes an invariant violation error that could be emitted if you tried to fetch transactional records before the beginning of a partition.
• Add support for ratelimiting the number of bytes that can be read from input streams during compactions to reduce networking microbursts.
• Tune the maximum number of concurrent input streams that can be opened during compactions based on the overall size of the compaction to reduce networking microbursts.
• Add a new flag for a preview feature, -multiregion. This tells the agent that the control plane runs across two regions, which will have the agent talk to the leader region while keeping track of leadership.
• Enable the Agent file cache to begin backpressuring sooner than before. This prevents the Agents from OOMing some scenarios when they’re completely overloaded and will now backpressure traffic more appropriately.
• Allow reducing batch timeout to 25ms now that S3OZ is so much cheaper.
• Add a new warpstream_control_plane_utilization metric (tagged by cluster). A value of 1.0 (100%) means the control plane is fully saturated and can no longer keep up with incoming requests.
• Fixes a race condition in schema validation when there are concurrent produces.
• Orbit now has an estimated time lag. The time lag is defined as time.Since(X) where X is the timestamp of the source record that is most recently copied by Orbit. The time lag will be emitted with the warpstream_consumer_group_estimated_lag_very_coarse_do_not_use_to_measure_e2e_seconds metric for the Orbit consumer group.

Release v647

April 10, 2025

• Fix a bug where S3 multi-part uploads from the produce path and the schema migrator would fail with “InvalidPart: One or more of the specified parts could not be found. The part may not have been uploaded, or the specified entity tag may not match the part’s entity tag”. The github.com/aws/aws-sdk-go-v2/config version was upgraded to 1.29.9 from 1.27.43 in Agent v643 but the upgrade is backwards-incompatible. Specifically, multi-part uploads that don’t specify a checksum algorithm will fail (see reports here and here).

Release v646 (Bad version, please upgrade to v647)

April 9, 2025

• Orbit ignores topic id in fetch responses if kafka version < 3.1.

Release v645 (Bad version, please upgrade to v647)

April 8, 2025

• Orbit makes metadata requests using topic name when max supported request version is < 12.

Release v644 (Bad version, please upgrade to v647)

April 8, 2025

• Add flags batch-size and sleep-between-batch for the delete-all-topics-unsafe command to allow configuring the batch size and sleep duration between batch deletions.
• Allow Agents to advertise IPs in range 240.0.0.0/8–255.0.0.0/8 automatically for customers using those in their private subnets.

Release v643 (Bad version, please upgrade to v647)

April 8, 2025

• Make the delete-all-topics-unsafe command iterative.

Release v642

April 7, 2025

• Add tls_insecure_skip_verify to orbit and schema registry migrator config

Release v641

April 7, 2025

• Treat “connection timed out” error as resumable for object storage GET requests to prevent long running compactions from failing.
• Fixes a bug where the agent could get stuck in some circumstances when reading data after a gap of more than 4B records created either by orbit or compacted topics compaction.

Release v640

April 2, 2025

• Tune down the interval for batching internal API calls for describing the topics in a cluster.
• Reduce memory usage for S3 clients when performing large numbers of concurrent DELETE operations.
• Double number of LOW priority jobs that can run concurrently.
• Create file-reader and file-scrubber cli-beta commands to read and scrub warpstream agent files.
• Add new -additionalDeadscannerBucketURLs flag and WARPSTREAM_ADDITIONAL_DEADSCANNER_BUCKET_URLS environment variable to allow specifying additional bucket URLs that should be scanned for dead objected to be deleted from the object store. This enables more seamless migrations from one object storage bucket to another without having to manually cleanup the dead files in the old object storage bucket.
• Add new mtls_server_ca_cert_env field to Orbit configuration to allow pointing to a file containing PEM encoded public keys of the certificate authorities that sign your server certificates.

Release v639

• Add support for WarpStream Schema Linking.
• Fix a rare panic when fetch auto-tuning is disabled explicitly and the kafka clients issue very small requests.
• Bump github.com/golang-jwt/jwt/v5 dependency for CVE-2025-30204 vulnerability.
• Return LeaderNotAvailable instead of BrokerNotAvailable for some transient / retriable Metadata errors to improve client compatibility.
• Change the verbosity of deadscanner “deadscanner_progress” logs when they hit a “not found” error.
• Allow Agents to run more L0/L1 compactions in parallel which helps prevent temporary spikes of L0/L1 lag when large L2 compactions are running.

Release v638

March 21, 2025

• Add metric agent_kafka_source_cluster_connections_counter that counts the number of connections made by agents to the source cluster (via Orbit)

Release v637

March 21, 2025

• Add consumer-group-lag command to cli-beta.
• Increase the default value of GOMEMLIMIT to 3GiB/vCPU from 2GiB/vCPU. For properly scaled clusters this should have no impact on memory usage, but it should prevent excessive garbage collection from occurring when there are a lot of inflight fetch requests for overloaded clusters.
• Ignore context.DeadlineExceeded errors for circuit breaking purposes when streaming GET requests from object store to avoid opening the circuit breaker when the Agents are overloaded, but the underlying object store is fine.
• Reduce log spam for context.Canceled errors.
• Use 4MiB pages by default instead of 16MiB pages.
• Agent handles sasl handshakes in the agent itself. This reduces excessive rpcs to the control plane.
• Orbit will prioritize fetching partitions by the order defined by the control plane.

Release v636

March 18, 2025

• Release analyzers in the Agent to feed Warpstream Diagnostics: Diagnostics continuously analyzes your clusters to identify potential problems, cost inefficiencies, and ways to make things better. It looks at the health and cost of your cluster and gives detailed explanations on how to fix and improve them.
• Add flag enableACLLogs and env var WARPSTREAM_ENABLE_ACL_LOGS to enable ACL logging.
  • As of this release Produce, Fetch, Metadata, JoinGroup, SyncGroup, DeleteRecords, InitProducerID Kafka API calls are logged if they are denied. More APIs will be added over time as we update the Control Plane.
• Reduce allocations in the fetch code path to improve performance.
• Fixes a very rare case where the agent would improperly categorize a single record as committed when it was aborted, with isolation_level=read_committed.

Release v635

March 12, 2025

• Enable a new mechanism in the Agents for cleaning up deleted files from object storage. This new mechanism makes it much easier for high volume clusters to keep up with object deletion, and also reduces costs by dramatically reducing the amount of LIST and HEAD requests that the Agent make.
• Reduces amount of data Orbit sends to the control plane to create topics.

Release v634

March 12, 2025

• Add support for using IMDSv2 to resolve instance types for Agents running in AWS.

Release v633

March 11, 2025

• Fixed a bug where setting an agent flag to an empty string by explicitly setting an environment variable to an empty string (i.e. ENV_VAR="") doesn’t work as expected as it uses the default value instead. This affects the following flags:
• If you set WARPSTREAM_DISABLE_CONSUMER_GROUP_METRICS_TAGS to an empty string, it used to default to disabling the partition tag. Now, if you set the env variable to an empty string, it doesn’t disable any tags for the consumer group offset metrics.
• If you set WARPSTREAM_AGENT_ROLES to an empty string, it used to default to proxy, jobs. Now, if you set WARPSTREAM_AGENT_ROLES to an empty string it means no roles are selected. Note that unless a role is provided via flags like enableManagedPipelines, empty roles is not allowed.
• Add begin_fetch_at_latest_offset to Orbit topic mappings which forces Orbit to fetch from the latest topic partition offset in the source cluster, for the first fetch of each topic partition.

Release v632

March 10th, 2025

• If the partition tag is disabled via the disableConsumerGroupsMetricsTags flag (or the WARPSTREAM_DISABLE_CONSUMER_GROUP_METRICS_TAGS environment variable), the warpstream_max_offset metric will be the max of the topic’s partitions’ max offsets.
• Increase WARPSTREAM_KAFKA_CLOSE_IDLE_CONN_AFTER to 1 hour to decrease the frequency of broken pipe errors on some kafka clients.
• Reduce log spam for some context canceled errors.

Release v631

March 7th, 2025

• Implement support for the v0 JoinGroup API in consumer group management, allowing older clients to join groups.

Release v630

March 6th, 2025

• If the partition tag is disabled via the disableConsumerGroupsMetricsTags flag (or the WARPSTREAM_DISABLE_CONSUMER_GROUP_METRICS_TAGS environment variable):
  • The consumer_group_lag metric will be the sum of the consumer group lag across the topic’s partitions.
  • The warpstream_consumer_group_estimated_lag_very_coarse_do_not_use_to_measure_e2e_seconds metric will be the max of the estimated lag across the topic’s partitions.
• Increase default connection idle timeout from 10m to 15m.
• Improve Agent logging (remove spurious logs, emit sampled logs for idle connections being closed for improved observability)

Release v629

Feb 28th, 2025

• The agent now accepts to create batches that span more than 4B offsets. This is useful for heavily compacted topics.
• Include warpstream proxy target in client ID for kafka_franz_warpstream blocks in managed data pipelines so that managed data pipelines works when Agents are split into proxy-consume and proxy-produce roles with no Agents running both roles simultaneously.
• Add MTLS support orbit.

Release v628

Feb 19th, 2025

• Improve visibility of ACL authorization failures.
• Return ErrKafkaStorageError instead of ErrThrottlingQuotaExceeded for backpressure errors from Produce requests because librdkafka doesn’t retry ErrThrottlingQuotaExceeded requests.
• Fix bug in PREFIXED ACLs, where sometimes were not correctly applied.

Release v627

Feb 18th, 2025

• Change the default fetch size for kafka_franz in managed data pipelines from 100MiB to 50MiB and the per-fetch fetch size from 50MiB to 25MiB.
• If sasl/mTLS auth is performed using a cluster credentials name, with or without the ccn_ prefix, allow both the credentials name and the credentials username to be used as the kafka principal for acls. Previously, if a client performed auth using a credentials name, the credentials username was used as the kafka principal, which prevented valid acls matches.

Release v626

February 18th, 2025

• Upgrade Agents to go 1.23.6.
• Bump our Alpine base image to 3.21.3 to fix some vulnerabilities
• Fix a rare panic happening on slow nodes when the publishing metric job would take too long to be handled

Release v625

February 12th, 2025

• Add batching of internal API calls for describing the topics in a cluster.

Release v623

February 10th, 2025

• Add diagnose-record to cli-beta, this command prints out diagnostic information about a specific record
• Reduce default values for fetch auto-tuning to be less aggressive so it causes less memory pressure on consumer clients if they embed multiple Kafka consumers within the same application.
• Prevent a tombstone from being deleted if it is the last record in a topic partition. Some workload use the offsets of the records consumed to determine if a consumer is caught up with the high watermark offset, and not deleting the tombstone allows the tombstone record to be consumed.

Release v622

February 6th, 2025

• Change the default value of autoTuneFetchLimits from false to true. This makes it so the WarpStream Agents automatically auto-tune the fetch limits of Kafka consumer client applications. This dramatically improves the consumer performance of consumer applications that have not explicitly tuned their application for performance with WarpStream.
• Add a new ws_dfat client ID feature that allows consumer clients to disable fetch auto-tuning if needed.
• Add flag --consume-from-beginning to the CLI console-consumer to allow consuming from the beginning of the topic partition.
• Enforce that agent group names only contain lowercase letters, numbers, and dashes.
• Add cli-beta subcommand that changes how cli flags work to have per cli subcommand flags instead of adding them all to every cli command.
• Add console-consumer to cli-beta which adds new features for printing out the offset of the message and the message key. Also add max-messages flag to limit the number of messages to consume.
• Fix a bug where the agent would return an empty response that the sarama kafka client would not know how to parse in some cases in transactional or compacted topics.
• New metrics:
  • warpstream_num_records: This is a gauge that indicates the number of records being stored per topic
    • tags: topic

Release v621

February 5th, 2025

• Add support for the min.compaction.lag.ms topic-level configuration. This configuration allows users to specify a minimum time that must pass before a segment is eligible for compaction. This configuration is useful for ensuring that data is not compacted too soon after being written in a topic that has compact or compact,delete as its cleanup policy.
• Fix a rare bug where incorrect data could be returned when using transactions in a compacted topic.

Release v620

February 3rd, 2025

• Allow Orbit to work with older kafka clusters(pre 3.1 which was when topic ids were added to kafka fetch request responses). If Orbit is used without topic ids, then it is possible that a topic was deleted and recreated in the source cluster with the exact same name which might lead to inconsistencies in Orbit. This is easily solved by deleting the topic in the target cluster, and it will be automatically recreated by Orbit.

Release v619

February 2nd, 2025

• Only retry Metadata errors that have retryable error codes.

Release v618

Jan 31st, 2025

• Allow blob storage retries to retry GET and multi-part upload requests in compaction code to prevent long-running compactions from failing due to transient errors.
• Improve error handling of the Metadata RPC to convert 500s and transient retryable errors to Kafka error LeaderNotAvailable instead of KafkaStorageError. This improves compatibility with librdkafka and prevents issues where librdkafka would refuse to accept additional records into its producer queue after receiving a KafkaStorageError in a Metadata response.

Release v617

Jan 28th, 2025

• Improve the consistent hashing ring used in the file cache by increasing the number of virtual shards and switching hashing algorithms. Agents will automatically wait until all of the other Agents in the cluster are upgraded to the latest version and then switch hash ring implementations at approximately the same time to minimize disruption during the rollout.
• Removed the allowHighCardinalityConsumerGroupTags flag and changed the default values of disableConsumerGroupsMetricTags to partition . This is a potentially breaking change for observability, but was done to prevent cluster with large numbers of topic-partitions from overloading customer’s metrics systems and bills. The partition tag for consumer group metrics can be re-enabled by setting the disableConsumerGroupsMetricTags value to an empty string.
• Add cli certificate-to-dn helper command to convert a client certificate to a DN.

Release v616

Jan 24th, 2025

• Fix JKS certificate loading to be according to the JKS spec, specifically load certificates from DER instead of PEM formatting.

Release v615

Jan 24th, 2025

• Fixed a bug where the BYOC schema registry incorrectly returns a schema incompatibility error when removing a field whose default value is null.

Release v614

Jan 23nd, 2025

• Fix naming of tlsJavaKeystorePasword and tlsJavaTruststorePasword to be tlsJavaKeystorePassword and tlsJavaTruststorePassword .

Release v613

Jan 22nd, 2025

• Orbit now copies internal kafka topics if these topics match the regex specified in the Orbit config.

Release v612

Jan 16th, 2025

• Added flags to support Java Keystores and Truststores to make migration from Kafka easier
  • tlsJavaKeystoreFile, tlsJavaKeystorePasword, tlsJavaKeystoreKeyPasword, tlsJavaTruststoreFile, tlsJavaTruststorePasword.
  • Java Keystore passwords are not used to encrypt the keystore, only to confirm it’s integrity. In most Java implementations the keystore password and private key password must be the same which negates any encryption advantages. We recommend using the same security controls when using Java Keystores that you would use with unencrypted certificate keys.
• Added support for using IMDSv2 to configure DD_AGENT_HOST for Agents running in AWS. If v2 is not available we automatically fall back to v1.
• Improved schema validation error message to return more detailed reason for why invalid record is rejected.
• Added the -schemaValidationURL flag for schema registry to replace the deprecated -schemaRegistryURL flag (still supported).

Release v611

Jan 10th, 2025

• Add tlsServerPrivateKeyPasswordFile flag to enable using encrypted TLS private keys.
  • The encrypted key given via tlsServerPrivateKeyFile must be in PKCS#8 format.
  • The password file must contain a single line which is the private key’s password.
• Honor DD_AGENT_HOST environment variable if its already set.
• Allow TLS to be used with Orbit even if SASL is not configured.
• Add support for Apache Kafka transactions. Read our announcement at https://www.warpstream.com/blog/kafka-transactions-explained-twice.

Release v610

Jan 8th, 2025

• Basic Authentication Support for BYOC Schema Registry:
  • Add schemaRegistryBasicAuth flag to enable basic authentication for schema registry.
• Support the GET /subjects/(string: subject)/versions endpoint in schema registry to allow getting a list of versions registered under the specified subject.
• Bump golang.org/x/net dependency for CVE-2024-45338 vulnerability.

Release v609

Dec 19th, 2024

• Add -requireSASLAuthentication and -requireMTLSAuthentication to playground mode. They where missed when these flags where added to the agent in v549.

Release v608

Dec 18th, 2024

• Add support for using DynamoDB as the agent’s backing store alongside S3 and S3 Express One Zone.

Release v607

Dec 18th, 2024

• Fix an issue where tombstones could be incorrectly handled, and showed as messages with empty values.
• Fix a rare issue where the agent would fail to connect to a S3 bucket on startup in some configurations.

Release v606

Dec 11th, 2024

• Make Orbit much more efficient and able to achieve higher throughput more easily.
• Double the ratelimit for the maximum inflight fetch compressed bytes per core (now that fetch is significantly more efficient).
• Make bento logging less noisy by limiting each managed pipeline to one bento log per second or less.

Release v605

Dec 10th, 2024

• Avoid noisy error logs when instance type discovery fails.
• Fix macOS Agent binaries to not segfault about a missing library

Release v604

Dec 9nd, 2024

• Add the ability to load the agent key from a file using the -agentKeyPath flag or WARPSTREAM_AGENT_KEY_PATH environment variable. The agent will reload the file when the file is updated so the key can be changed without restarting the agent.
• Upgrade to latest Bento which adds ability to ratelimit pipelines based on bytes instead of messages, and improving GCP bigquery output logging.
• Switch to C++ library for lz4 compression instead of pure Go.

Release v603

Dec 2nd, 2024

• Preserves the original DN from the TLS cert to be used as the ACL principal. Previously, while the DN was semantically correct, the order of its elements wasn’t necessarily preserved. This leads to the principal set during auth to not match the principal set when the ACL is created.
• Fix a log in the managed data pipelines feature that was logging a cluster-specific credential.
• Add support for SASL SCRAM 256/512 as an authentication mechanism for source clusters in Orbit.

Release v602

Nov 25th, 2024

• Retry object storage permissions check up to 3 times on startup to avoid false positives related to dial timeouts
• Added BYOC Schema Registry Support:
  • The agent now supports hosting a BYOC Schema Registry.
  • Use the -schemaRegistryPort flag (or the WARPSTREAM_SCHEMA_REGISTRY_PORT env variable) to specify the port (default 9094) to run the schema registry server on.
  • Use the -schemaRegistryTLS flag (or the WARPSTREAM_SCHEMA_REGISTRY_TLS_ENABLED env variable) to enable tls over the schema registry server.
  • New metrics:
    • warpstream_agent_schema_registry_inflight_connections: number of currently inflight / active connections to schema registry server.
      • tags: schema_registry_operation, outcome
    • warpstream_agent_schema_registry_request_latency: latency (seconds) for processing each Schema registry request.
      • tags: schema_registry_operation, outcome
    • warpstream_agent_schema_registry_outcome: outcome (success, error, etc) for each Schema Registry request.
      • tags: schema_registry_operation, outcome
    • warpstream_agent_schema_registry_request_bytes_counter: number of bytes of incoming requests.
      • tags: schema_registry_operation
    • warpstream_agent_schema_registry_response_bytes_counter: number of bytes of outgoing response.
      • tags: schema_registry_operation, outcome
    • warpstream_schema_versions_count: Total number of schema versions in the schema registry cluster.
    • warpstream_schema_versions_limit: Maximum number of schema versions allowed in the cluster.

Release v601

Nov 21st, 2024

• Limit maximum number of inflight bytes for fetch as compressed instead of uncompressed, and do it before issuing the fetch requests. This prevents excessive read amplifications when the Agents are highly loaded.
• Switch from offheap to onheap cache for file cache for better eviction policy.
• Fix a bug that was preventing the Agents from loading large IOs directly from object storage leading to excessive cache churn.
• Remove some prefetcher restrictions to improve the performance of large fetch requests with many topic-partitions.

Release v600

Nov 20th, 2024

• Fixed DescribeCluster API to correctly override hostname
• Fixed a bug where the demo/playground agent was incorrectly using the environment variable.

Release v599

Nov 20th, 2024

• Only restart Bento pipelines when their deployed configurations are modified. Previously, modifying one configuration restarted all deployed pipelines.
• Increase default timeout for loading data into the file cache from 5s to 15s. This has no impact on tail latencies for healthy clusters due to the fast retry mechanism, but dramatically improves the ability of overloaded clusters to make progress.
• Cap number of speculative/fast retries to 2% at most. This prevents excessive read amplification in some scenarios where the Agents are overloaded.
• Upgrade Agents to go 1.23.3 to eliminate contention in HTTP client.

Release v598

Nov 6th, 2024

• Removed noisy error log triggered when producing with ACKS=0. Previously, clients and agents generated a fake invariant error: ‘Previous sent message is not current sequence -1’ due to an unordered sequence number.
• Allow Agents to create ingestion files with up to 64MiB of uncompressed data, increased from 16 MiB.
• Orbit encrypts communication with source clusters using TLS.

Release v597

Nov 6th, 2024

• Default Agent Group: When the -agentGroup parameter is not explicitly set, the agent will now default to a group named ‘default’. Previously, if a Kafka client connects to an agent without a group specified, it could end up connecting to agents in other groups. With this change, clients connected to an agent with no group specified will now only see agents in the ‘default’ group.
  • Note: This change is backwards compatible; during rollout, agents with no group set and those in the ‘default’ group will be treated as a single group.
• Add support for SASL handshake v0.

Release v596

Nov 1st, 2024

• Add delay + jitter + randomization to the order/timing in which Bento pipelines are started to avoid pipelines synchronizing with each other and doing all of their work at the same time.
• Add support for Orbit to the Agents. See: https://docs.warpstream.com/warpstream/byoc/orbit

Release v595

October 30th, 2024

• Return success + offsets/timestamps instead of DuplicateSequenceError + offsets/timestamps when a duplicate batch is detected via the idempotent producer functionality to mirror Kafka’s behavior.
• Upgrade to latest version of Bento with more improvements for parquet encoder (bug fixes + support for int8/int16).

Release v594

October 28th, 2024

• Upgrade to latest version of Bento with improved handling of decimals and floats in parquet encoder.

Release v593

October 27th, 2024

• Fixes metrics called “warpstream_agent_segment_batcher_flush_xxx” who were reporting their successes with “flush_cause:buffer_full” as errors.
• Emit Bento metrics in Agents when managed data pipelines is enabled. All Bento metrics will be prefixed with: warpstream_bento.
• Upgrade to latest version of Bento with ability to encode maps/lists in the parquet encoder.

Release v592

October 23nd, 2024

• Upgrade to latest Bento version, which improves error logging for the GCP BigQuery output.

Release v591

October 22nd, 2024

• Add support for isolating managed data pipelines to different groups of pipelines Agents using the managedPipelinesGroupName flag and WARPSTREAM_MANAGED_PIPELINES_GROUP_NAME environment variable. Read the docs
• Make manage data pipelines product work automatically, even when the WarpStream Agents are advertising the hostname of a load balancer to the Kafka protocol by making the Kafka metadata handler always return the Agents actual IP addresses to the managed data pipelines product.
• Upgrade to latest Bento version which adds support for interpolating table name in GCP BigQuery output, as well as support for GCP’s new Storage Write API for BigQuery. Also adds support for writing to Redshift clusters in AWS.

Release v590

October 17th, 2024

• Make the Agents use the pure GO DNS resolver to work around a concurrency issue with setenv and getenv.
• Agents will now automatically tune their GC and backpressure settings automatically based on which roles are configured. proxy-consume Agents will have larger file caches, proxy-produce Agents will buffer more data before backpressuring, pipelines Agents will GC less aggressively, etc

Release v589

October 16th, 2024

• Add fetch limits auto-tuning to the Agents, which automatically adjusts fetch request size limits when the Agents deem necessary. To enable this feature, use the flag -autoTuneFetchLimits or the env variable WARPSTREAM_AUTO_TUNE_FETCH_LIMITS. By default this is disabled.

Release v588

October 11th, 2024

• Agents configured with specific roles will now reject requests that they’re not supposed to handle. For example, proxy-consume Agents will reject Fetch requests and proxy-produce Agents will reject Produce requests.
• WarpStream Agents will now automatically avoid communicating with each other if multiple clusters are deployed in the same VPC, even if I.P addresses are quickly recycled between the two clusters (due to high container churn).
• Increased default batching interval for Metadata requests to 250ms.
• Upgrade managed data pipelines to latest version of Bento, and automatically tune franz_kafka blocks to fetch data from source Kafka/WarpStream clusters faster.

Release v587

September 27th, 2024

• Increase the default value of GOMEMLIMIT from 1GiB/vCPU to 1.5GiB/vCPU. This should result in less GC overhead for highly loaded Agents.
• Automatically disable profile forwarding if Datadog profiling is turned on.

Release v586

September 25th, 2024

• Change the default value of disableProfileForwarding/WARPSTREAM_DISABLE_PROFILE_FORWARDING to false, which means that the Agents will start forwarding profiles to WarpStream’s control plane.

Release v585

September 24th, 2024

• Add support for scheduling blocks in managed data pipelines

Release v584

September 23rd, 2024

• Add support for AWS Glue Schema Registry schema validation
  • Users can now use schemas stored in their AWS Glue Schema Registry to validate records.
  • Add new topic-level configuration warpstream.schema.registry.type to specify what type of schema registry the agent should fetch the remote schemas from. Supported values include "STANDARD" and "AWS_GLUE". Defaults to "STANDARD"
• Add -zonedCIDRBlocks flag and WARPSTREAM_ZONED_CIDR_BLOCKS env variable as an alternative way to provide information on the Kafka client’s availability zone. The value is a mapping of availability zones to Kafka client IPs and should be a <> delimited list of AZ to CIDR range pairs, where each pair starts with an AZ, a @, and a comma separated list of CIDR blocks for that given AZ. For example, us-east-1a@10.0.0.0/19,10.0.32.0/19<>us-east-1b@10.0.64.0/19<>us-east-1c@10.0.96.0/19.

Release v583

September 18th, 2024

• Bump the otel dependency to fix spurious error logs introduced in v582

Release v582

September 18th, 2024

• Increase default clean shutdown interval from 60s to 80s.
• Increase the max number of connections per CPU (from 2048 to 4096)

Release v581

September 12th, 2024

• Bump build to go 1.22.7 to fix some stdlib vulnerabilities

Release v580

September 11th, 2024

• Add agent profile forwarding to WarpStream’s control plane. Note that this feature cannot be enabled together with Datadog profiling and is not supported for self-hosted control planes. The following flags are added for this feature.
  • disableProfileForwarding/WARPSTREAM_DISABLE_PROFILE_FORWARDING: this flag can be used to disable profile forwarding. By default this is set to true and no profiles will be forwarded.
  • maxProfileSize/WARPSTREAM_MAX_PROFILE_SIZE: max number of bytes allowed for buffering profiles in memory.
• Bump our Alpine base image to 3.20.3 to fix some vulnerabilities

Release v579

September 9th, 2024

• Add -maxProduceRecordSizeBytes flag and WARPSTREAM_MAX_PRODUCE_RECORD_SIZE_BYTES env variable to override the maximum uncompressed size of a record that can be produced.

Release v578

August 30th, 2024

• Add “ratelimited” outcome for S3-backed blob storage metrics warpstream_blob_store_*
• Increased maximum allowed ingestion file size from 8MiB to 16MiB
• Improve buckets for distribution metrics for warpstream_agent_segment_batcher_flush_file_size_uncompressed_bytes and warpstream_agent_segment_batcher_flush_file_size_compressed_bytes
• Fixed a bug where we returned the wrong error code when a produced record was larger than the maximum allowed size (32MiB)

Release v577

August 25th, 2024

• Fix bug in backpressure system that would cause Agents to get stuck in backpressure state.

Release v576

August 22nd, 2024

• Fix bug affecting librdkafka library’s cooperative rebalance, when adding more clients to a consumer group.
• Add new startup flags to enable mutex and block profiling:
  • -enableSetMutexProfileFraction/WARPSTREAM_ENABLE_SET_MUTEX_PROFILE_FRACTION: enable this flag to call ‘runtime.SetMutexProfileFraction’ with the value passed along -mutexProfileFraction.
  • -mutexProfileFraction/WARPSTREAM_MUTEX_PROFILE_FRACTION: tune the value passed to call ’runtime.SetMutexProfileFraction
  • -enableSetBlockProfileRate/WARPSTREAM_ENABLE_SET_BLOCK_PROFILE_RATE: enable this flag to call ‘runtime.SetBlockProfileRate’ with the value passed along -blockProfileRate
  • -blockProfileRate/WARPSTREAM_BLOCK_PROFILE_RATE: tune the value passed to call ‘runtime.SetBlockProfileRate’

Release v575

August 18th, 2024

• Report CPU usage using a moving average instead of point-in-time to improve accuracy.
• Improve performance when a cluster has many clients polling partitions that are written to infrequently.

Release v574

August 15th, 2024

• Increase default limits for backpressuring produce requests by 400%

Release v573

August 14th, 2024

• Disable automatic retries in underlying blob storage clients (AWS / GCP) so we can control blob storage retries at the application layer.
• Switch to C++ library for ZSTD instead of pure Go (2-3x faster).

Release v572

August 6th, 2024

• Improved the Agents ability to backpressure Produce requests.
• Added back-pressure support for Fetch requests in addition to Produce requests.

Release v571

August 2nd, 2024

• Improved Agent backpressuring for Produce requests so that Agents will begin throttling Produce request and individual connections when they have too much producer data buffered in memory instead of OOMing.
• Agents now return KafkaStorageError instead of RequestTimedOut for some timeout-related errors in the Fetch path. This improves behavior with Java consumer clients which treat KafkaStorageError as retryable, but not RequestedTimedOut.
• Fixed a bug in demo / playground mode where the requested hostname strategy was not being used.
• Tagged all Agent metrics with the agent_roles , agent_group , and virtual_cluster_id tags to make debugging advanced deloyments easier.

Release v570

July 17th, 2024

• Fixed a bug that would cause idempotent producer out of sequence errors when idempotent producer was enabled on some clients. Data was never committed out of order, but the previous implementation would force the client to retry more than necessary, resulting in high latency or reduced throughput
• Added Schema Validation Support:
  • Check out documentation for more details

Release v569

July 5th, 2024

• Add a new warpstream_max_offset metric tagged by topic/partition (that can be controlled with the existing disableConsumerGroupsMetricsTags flag).
• The existing warpstream_consumer_group_max_offset metric is deprecated as it shares the same value across consumer groups and it can be replaced with the new metric mentioned above.
• Add the topic label/tag to warpstream_agent_kafka_produce_uncompressed_bytes_counter that was missing it.

Release v568

July 3rd, 2024

• Various small performance improvements (batching, allocations, etc).
• Batch Metadata and FindCoordinator request RPCs between the Agents and the control plane. Helpful for workloads with a large number of consumer/producer clients.

Release v567

June 19th, 2024

• Allow individual fetch requests to fetch up to 128MiB of uncompressed data per topic-partition, per fetch request, increased from 32 MiB.
• Add -kafkaMaxFetchRequestBytesUncompressedOverride flag and WARPSTREAM_KAFKA_MAX_FETCH_REQUEST_BYTES_UNCOMPRESSED_OVERRIDE env variable to override maximum number of uncompressed bytes that can be fetched in a single fetch request.
• Add -kafkaMaxFetchPartitionBytesUncompressedOverride and WARPSTREAM_KAFKA_MAX_FETCH_PARTITION_BYTES_UNCOMPRESSED_OVERRIDE env variable to override maximum number of uncompressed bytes that can be fetched for a single topic-partition in a single fetch request.
• Introduced the metric warpstream_consumer_group_generation_id with the tags consumer_group. This metric indicates the generation number of the consumer group, incrementing by one with each rebalance. It serves as an effective indicator for detecting occurrences of rebalances.
• Added metric warpstream_consumer_group_estimated_lag_very_coarse_do_not_use_to_measure_e2e_seconds (tags: consumer_group, topic, partition). Provides an estimated lag in seconds (equivalent to warpstream_consumer_group_lag but in time units instead of offsets), calculated via interpolation/extrapolation. Caution: Very coarse; unsuitable for precise end-to-end latency measurement.
• Renamed Flags and Environment Variables:
  • benthosBucketURL (WARPSTREAM_BENTHOS_BUCKET_URL) is now bentoBucketURL (WARPSTREAM_BENTO_BUCKET_URL): Bucket URL to use when fetching the Bento configuration.
  • benthosConfigPath (WARPSTREAM_BENTHOS_CONFIG_PATH) is now bentoConfigPath (WARPSTREAM_BENTO_CONFIG_PATH): Path in the bucket to fetch the Bento configuration.

Release v566

June 5th, 2024

• Fix broken formatting of logs that were sometimes logfmt when they should’ve been JSON.

Release v565

June 5th, 2024

• Improve the Agent’s ability to handle certain pathological compactions where an input stream is not read from for so long that the object store closes the connection. The Agent will now detect this scenario and issue a new GET request to resume the compaction instead of failing it entirely and never making progress.

Release v564

May 26th, 2024

• Improve performance of file cache for high throughput workloads by improving how batching is handled.
• Add support for more Benthos components: [awk, jsonpath, lang, msgpack, parquet, protobuf, xml, zstd].
• Fix rare memory leak.
• Fix rare bug that would cause circuit breakeres to get stuck in open / half-open state forever.

Release v563

May 22nd, 2024

• Fixed a bug in the Agent that prevented users from starting playgrounds or demos.

Release v562

May 19th, 2024

Agent

• Make Agent batch size configurable using -batchMaxSizeBytes flag and WARPSTREAM_BATCH_MAX_SIZE_BYTES environment variable.
• Metrics
  • Add new Agent metric (gauge) that indicates state of consumer group: warpstream_consumer_group_state. Metric has two tags: consumer_group and group_state.
  • Add new Agent metric (gauge) that indicates the number of members in each consumer group: warpstream_consumer_group_num_members. Metric has one primary tag: consumer_group.
  • Add new Agent metric (gauge) that indicates the number of topics in each consumer group: warpstream_consumer_group_num_topics. Metric has one primary tag: consumer_group.
  • Add new Agent metric (gauge) that indicates the number of partitions in each consumer group: warpstream_consumer_group_num_partitions. Metric has one primary tag: consumer_group.
  • Add new Agent metric (gauge) that indicates the total number of topics in the cluster: warpstream_topic_count.
  • Add new Agent metric (gauge) that indicates the limit for the total number of topics in the cluster: warpstream_topic_count_limit.
  • Add new Agent metric (gauge) that indicates the total number of partitions in the cluster: warpstream_partition_count.
  • Add new Agent metric (gauge) that indicates the limit for the total number of partitions in the cluster: warpstream_partition_count_limit.

Release v559

May 13th, 2024

Agent

• Add support for WarpStream Managed Data Pipelines
• Change the default number of uncompressed bytes that will be be buffered before flushing a file to object storage from 8MiB to 4MiB

Release v558

May 10th, 2024

Agent

• Support adopting AWS roles for providing access to S3 via environment variables instead of a query argument in the bucket URL.
• Tuned circuit breakers to be less aggressive by default, and configured the retry mechanisms to avoid retrying circuit breaker errors.
• New Metrics
  • warpstream_circuit_breaker_open: Records the number of times the circuit breaker is opened, tagged by the circuit breaker name to identify the associated operations. - tags: name
  • warpstream_circuit_breaker_close: Records the number of times the circuit breaker is closed, tagged by the circuit breaker name to identify the associated operations.
    • tags: name
  • warpstream_circuit_breaker_halfopen: Records the number of times the circuit breaker is half-opened, tagged by the circuit breaker name to identify the associated operations.
    • tags: name
  • warpstream_circuit_breaker_error: Records the number of times the circuit breaker blocks an operation, tagged by the circuit breaker name to identify the associated operations.
    • tags: name

Release v557

May 9th, 2024

Agent

• Enhanced Buckets: Refined the buckets for warpstream_agent_kafka_fetch_uncompressed_bytes, warpstream_agent_kafka_fetch_compressed_bytes, warpstream_agent_kafka_produce_compressed_bytes and warpstream_agent_kafka_produce_uncompressed_bytes to ensure they are more precise. The previous bucket configuration was too sparse.
• Fixed Prometheus Counters: Resolved an issue with Prometheus counters previously were not appropriately incrementing the value.

Release v556

May 9th, 2024

Agent

• Tune circuit breakers to be less aggressive, and don’t retry requests that failed due to a circuit breaker error.
• New metrics:
  • warpstream_circuit_breaker_open: Records the number of times the circuit breaker is opened, tagged by the circuit breaker name to identify the associated operations.
    • tags: name
  • warpstream_circuit_breaker_close: Records the number of times the circuit breaker is closed, tagged by the circuit breaker name to identify the associated operations.
    • tags: name
  • warpstream_circuit_breaker_halfopen: Records the number of times the circuit breaker is half-opened, tagged by the circuit breaker name to identify the associated operations.
    • tags: name
  • warpstream_circuit_breaker_error: Records the number of times the circuit breaker blocks an operation, tagged by the circuit breaker name to identify the associated operations.
    • tags: name

Release v555

May 8th, 2024

Agent

• New metrics:
  • warpstream_agent_kafka_fetch_uncompressed_bytes_counter: Tracks the count of uncompressed bytes fetched. Although a histogram version (warpstream_agent_kafka_fetch_uncompressed_bytes) already exists, it may not provide an accurate count in some metrics systems. That’s why we’ve made this data directly available as a counter.
    • tags: topic (requires enabling high-cardinality metrics)
  • agent_kafka_produce_uncompressed_bytes_counter: Tracks the count of uncompressed bytes produced. While the histogram version (warpstream_agent_kafka_produce_uncompressed_bytes) is available, some metrics systems might not accurately count it. So, we’re also offering this data directly as a counter.
    • tags: topic (requires enabling high-cardinality metrics)

Release v554

April 30th, 2024

Agent

• Replaces all instances of Ristretto cache with a simple LRU cache, dramatically reduces the number of caches misses in topic metadata related caches.

Release v553

April 30th, 2024

Agent

• Rename metrics:
  • warpstream_agent_kafka_inflight_conn has been renamed to warpstream_agent_kafka_inflight_connections
  • warpstream_agent_kafka_inflight_request has been renamed to warpstream_agent_kafka_inflight_requests

Release v552

April 29th, 2024

Agent

• Dramatically improve the performance of fetch requests that query 100s or 1000s of partitions in a single fetch request.
• Add speculative retries for blob storage reads in the fetch path (in addition to existing speculative retries for blob storage writes in the write path).

Release v551

April 26th, 2024

Agent

• Convert an error log (that did not actually represent an error) to an info log to reduce error spam.

Release v550

April 23nd, 2024

Agent

• High-Cardinality Metrics: Introduced a new agent configuration for enabling high-cardinality metrics. To activate this feature, use the command-line flag -kafkaHighCardinalityMetrics or set the environment variable WARPSTREAM_KAFKA_HIGH_CARDINALITY_METRICS=true.
• Deprecated metrics:
  • warpstream_agent_kafka_fetch_bytes_sent
  • warpstream_agent_segment_batcher_flush_file_size_counter_type
  • warpstream_agent_segment_batcher_flush_file_size
  • blob_store_list_latency
  • blob_store_list_count
  • blob_store_delete_latency
  • blob_store_delete_count
  • blob_store_put_bytes_latency
  • blob_store_put_bytes_count
  • blob_store_put_stream_latency
  • blob_store_put_stream_count
  • blob_store_get_bytes_latency
  • blob_store_get_bytes_count
  • blob_store_get_stream_latency
  • blob_store_get_stream_count
  • blob_store_get_bytes_range_latency
  • blob_store_get_bytes_range_count
  • blob_store_get_stream_range_latency
  • blob_store_get_stream_range_count
• New metrics:
  • warpstream_agent_kafka_fetch_uncompressed_bytes: Tracks the total uncompressed bytes fetched, replacing warpstream_agent_kafka_fetch_bytes_sent.
    • tags: topic (requires enabling high-cardinality metrics)
  • warpstream_agent_kafka_produce_uncompressed_bytes: Tracks the number of uncompressed bytes produced.
    • tags: topic (requires enabling high-cardinality metrics)
  • warpstream_agent_segment_batcher_flush_file_size_uncompressed_bytes: Tracks the uncompressed size of files stored after batching, serving as a replacement for warpstream_agent_segment_batcher_flush_file_size.
  • warpstream_blob_store_operation_latency: Tracks the latency and count of individual object storage operations. It’s a replacement for all deprecated blob_store_ metrics. Now all the different operations are tagged within the same metric.
    • tags: operation,outcome

Release v549

April 22nd, 2024

Agent

1. Adds support for authenticating Kafka clients using mTLS. A -requireMTLSAuthentication flag is added, and the previous -tlsVerifyClientCert flag has been deprecated. A new -requireSASLAuthentication flag is added, and the previous -requireAuthentication flag is deprecated. When authenticating using mTLS, the Distinguished Name(DN) from the client certificate is used as the Kafka Principal. The -tlsPrincipalMappingRule flag can be used to specify a Regex to extract a principal from the DN. For example, the rule CN=([^,]+) will extract the Common Name(CN) from the DN, and use that as the ACL principal.

Release v548

April 17th, 2024

Agent

1. Fix a rare bug where some compactions could fail if you had tombstone expiration enabled and very little data to compact.

Release v547

April 11nd, 2024

Agent

1. Fix playground mode against latest control plane signup constraints

Release v545

April 4nd, 2024

Agent

1. Added support to new regions. There is a new “-region” (or WARPSTREAM_REGION env variable) that you can leverage to pick among the supported regions - you can use the console to see the current list.

Release v544

April 2nd, 2024

Agent

1. Added support for TLS/mTLS for Warpstream Agent and kafka client connections. The -kafkaTLS(env WARPSTREAM_TLS_ENABLED=true), -tlsServerCertFile(env WARPSTREAM_TLS_SERVER_CERT_FILE=<filepath>), tlsServerPrivateKeyFile(env WARPSTREAM_TLS_SERVER_PRIVATE_KEY_FILE=<filepath>), -tlsVerifyClientCert(env WARPSTREAM_TLS_VERIFY_CLIENT_CERT=true), -tlsClientCACertFile(env WARPSTREAM_TLS_CLIENT_CA_CERT_FILE=<filepath>) flags were added which respectively are used to enable tls, pass the TLS server certificate to the Agent, pass the TLS server private key to the Agent, optionally enable client certificate verification, and to optionally pass the TLS root certificate authority certificate file to the server.

Release v543

March 29th, 2024

Agent

1. Adding a new “availabilityZoneRequired” (or env variable “WARPSTREAM_AVAILABILITY_ZONE_REQUIRED”) flag. When enabled, the agent will synchronously try to resolve its availability zone during startup for 1 min, and will not start serving its /v1/status health check until it succeeds. The process will exit early if it did not manage to resolve the availability zone.

Release v542

March 26th, 2024

Agent

1. Fixed a race condition during startup that could make some agents advertise their availability zone as “WARPSTREAM_UNSET_AZ”

Release v541

March 25th, 2024

Agent

1. Added beta support for benthos in the WarpStream Agents

Release v539

March 15th, 2024

Agent

1. Adding a new “agentGroup” parameter to define the name of the ‘group’ that the Agent belongs to. This feature is used to isolate groups of Agents that belong to the same logical cluster, but should not communicate with each other because they’re deployed in separate cloud accounts, vpcs, or regions. By default the agent belongs to the default group.

Release v538

March 6th, 2024

Agent

1. Fixed a rare panic in the Agents caught by Antithesis

Release v537

March 1st, 2024

Agent

1. Kafka “compacted topics” are generally available.

Release v536

February 28th, 2024

Agent

1. Fix a bug with the warpstream playground command, introduced in v535.

Release v535

February 26th, 2024

Agent

1. Make agent pool name argument completely optional, even when using non-default clusters.
2. Fixed a memory leak that would cause some workloads to use an excessive amount of memory over time.
3. Add support for S3 express and separating the “ingestion” bucket from the “compaction” bucket so data can be landed into low-latency storage and then immediately compacted into lower cost storage.
4. Add speculative retries to file flushing which dramatically reduces outlier latency for Produce requests.

Release v534

Agent

Bug fixes

1. Fixed another bug in the fetch logic that would result in the Agent returning empty batches in some scenarios when transient network errors occurred. This did not cause any correctness issues, but would make librdkafka refuse to proceed in some scenarios and block consumption of some partitions.

Release v533

Agent

Bug fixes

1. Fixed a bug in the fetch logic that would result in the Agent returning empty batches in some scenarios depending on the client configuration. This did not cause any correctness issues, but would make librdkafka refuse to proceed in some scenarios and block consumption of some partitions.

Release v532

Agent

New Features

1. Improves roles reporting to Warpstream control plane so that they can be properly rendered in our console.

Release v531

Agent

New Features

1. Supports automatic availability zone detection in kubernetes reading the node zone label (requires version 0.10.0 of our helm charts).

Release v530

Agent

New Features

1. Compatibility with CreatePartitions Kafka API for updating partitions count.

Release v529

Agent

New Features

1. Treat grade nat as private IP addresses, and allow advertising it.

Performance improvements

1. Improve caching in the agent.
2. Improve batching of requests to the warpstream metadata backend.
3. Other general performance improvements.

Release v526

Agent

New Features

1. Full-support for ACLs using SASL credentials.

Release v525

Agent

Bug Fixes and performance Improvements

1. Improved object pooling in RPC layer to reduce memory usage.
2. Add transparent batching to the file cache RPCs to dramatically reduce the number of inter-agent RPCs for high partition workloads.
3. Enhanced Apache Kafka compatibility by refining the handling of watermarks in Fetch responses.

Release v524

Agent

Dynamic Consumer Group Rebalance Timeout

1. Partially handle consumer group requests (JoinGroup and SyncGroup) in the agent, to use the clients’ rebalance timeout, instead of the previous 10s default. This enhancement will minimize unnecessary rebalance attempts in larger consumer groups with numerous members, due to short timeouts.

Release v523

Agent

Bug Fixes and performance Improvements

1. Fixed a bug in the Fetch() code that was not setting the correct topic ID in error responses which made some Kafka clients emit warning logs when this happened.
2. Fixed a bug in the “roles” feature that was causing Agents with the “produce” role to still participate in the distributed file cache. Now only Agents with the “consume” role will participate in the file cache, as expected.

Release v522

Agent

Bug Fixes and performance Improvements

1. Circuit breakers will now return example errors for clarity.
2. Fetch() code path will now handle failures more gracefully by returning incremental results in more scenarios which improves the system’s ability to recover under load.
3. Fix a memory leak in the in-memory file cache implementation.

Release v521

Agent

New Features

1. Docker images are now multi-arch, our documentation and official kubernetes charts has been updated accordingly.
2. Introduced circuit breakers around object storage access.
3. Finer control over agent roles: it is now possible to split between the proxy-consume and proxy-produce roles, our documentation has been updated as well.

Release v520

Agent

This release is the first phase of a two-phase upgrade to WarpStream’s internal file format. This release adds support for reading the upgraded file format. You MUST upgrade all Agents to this version before moving from any version < v520 to any version > than v520.

Release v518

Agent

New Features

1. Support kafka Headers: if you produce messages containing Kafka headers, they will now be automatically persisted to your cloud object storage, and will be read when fetching.
2. Revisit the flags and configuration knobs to choose how the agents advertise themselves in Warpstream service discovery. Our documentation has been updated accordingly.
3. Agent nodes can now be configured to run dedicated roles - see splitting roles documentation.

Control Plane

New Features

1. Fully support kafka ListOffsets protocol: you can now look for partition offsets based on timestamps.

Release v517

Agent

Bug Fixes and performance Improvements

1. Fixed a bug related to the handling of empty (but not null) values in records in the Fetch implementation.

Release v516

Agent

New Features

1. The agent will now report a sample of its error logs back to Warpstream control plane. It should ease troubleshooting and help us identify issues earlier. This can be disabled with the flag disableLogsCollection or the environment variable WARPSTREAM_DISABLE_LOGS_COLLECTION.

Bug Fixes and performance Improvements

1. Added batching in the metadata calls made during Kafka Fetch, improving memory usage along the way.

Control Plane

New Features

1. Added support for Kafka’s InitProducerID protocol message, and the idempotent producer functionality in general. Requires upgrading to a version of the Agents that is >= v515.
2. Added support for Kafka ListOffsets with positive timestamps value (until now only negative values for special cases were supported)

Release v515

Agent

New Features

1. The Agents will now report the lag / max offsets for every active consumer group as standard metrics. The metrics can be found as warpstream_consumer_group_lag and warpstream_consumer_group_max_offset .
2. The Agents will now report the number of files at each compaction level so that user’s can monitor whether they are experiencing compaction lag. These metrics can be found as warpstream_files_count and the level is tagged with the name compaction_level.

Bug Fixes and performance Improvements

1. File cache is now partitioned by <file_id, 16MiB extent> instead of just <file_id>. This spreads the load for fetching data for large files more evenly amongst all the Agents.
2. Added some logic in the file cache to detect when certain parts of the cache are experiencing high churn and reduce the default IO size for paging in data from object storage. This helps avoid filling the cache with data that won’t be read.
3. Fixed a bug in the file cache that was causing it to significantly *over* fetch data in some scenarios. This did not cause any correctness problems, but it wasted network bandwidth and CPU cycles.
4. Modified the implementation of the Kafka Fetch method to return incremental results when it experiences a retryable error mid-fetch. This makes the Agents much better at recovering from disruption and catching consumer lag incrementally.
5. Added some pre-fetching logic into the Kafka Fetch method so that when data for a single partition is spread amongst many files the Agent doesn’t get bottlenecked making many single-threaded RPCs. This mostly helps increase the speed at which individual partitions can be “caught up” when lagging.
6. Increased the default maximum file size created at ingestion time from 4MiB to 8MiB. This improves performance for extremely high volume workloads.
7. Added replication to the Agent file cache so that if an error is experienced trying to load data from the file cache on the Agent node that is “responsible” for a chunk of data, the client can retry on a different node. This helps minimize disruption when Agents shutdown ungracefully.
8. Agents now report their CPU utilization to the control plane. We will use this information in the future to improve load balancing decisions. CPU utilization can be view in the WarpStream Admin console now as well.
9. Improved the performance of deserializing file footers.
10. Standardized prometheus metric names prefixes.
11. Added a lot more metrics and instrumentation, especially around the blob storage library and file cache.

Control Plane

New Features

1. Added support for the Kafka protocol message DeleteTopics .

Bug Fixes and Performance Improvements

1. We added intelligent throttling / scheduling to the deadscanner scheduler. This scheduler is responsible for scheduling jobs that run in the Agent to scan for “dead files” in object storage and delete them. Previously these jobs could run with high frequency and rates which would interfere with live workloads. In addition, they could also result in very high object storage API requests costs due to excessive amounts of LIST requests. The new implementation is much more intelligent and automatically tunes the frequency to avoid disrupting the live workload and incurring high API request fees.