Topic Configuration Reference

This page lists the supported configuration for WarpStream topics. Configuration not listed here will be silently ignored by the WarpStream Kafka API.

Configuration items that start with warpstream. are WarpStream specific configurations, native Kafka tooling like the kafka-configs CLI may not be able to set these configurations. It is recommended to set these configurations using our REST API for topics, WarpStream terraform provider, or a non-Java library like franz-go to manage these configurations.

message.timestamp.type

Define whether the timestamp in the message is message create time or log append time. The value should be either CreateTime or LogAppendTime

Type:	string
Default:	CreateTime
Valid Values:	[CreateTime, LogAppendTime]

cleanup.policy

This config designates the retention policy to use on topics. The “delete” policy (which is the default) will discard old message when their retention time or size limit has been reached. The “compact” policy will enable topic compaction, which retains the latest value for each key. It is also possible to specify both policies in a comma-separated list (e.g. “delete,compact”). In this case, old messages will be discarded per the retention time and size configuration, while retained messages will be compacted.

Note

In WarpStream cleanup.policy can be changed after topic creation but with restrictions. Specifically, a non-compacted topic cannot be made compacted, and a compacted topic cannot be made non-compacted. For example, going from compact to compact,delete or delete,compact is allowed, so is going from compact,delete to compact. However, going from compact,delete to delete is not allowed, and neither is going from delete to compact,delete.

Type:	list
Default:	delete
Valid Values:	[compact, delete]

retention.ms

This configuration controls the maximum time we will retain messages in a topic partition. This represents an SLA on how soon consumers must read their data. If set to -1, no time limit is applied. The minimum retention is 1 minute. If you set it to a positive value that’s lower than 1 minute, WarpStream will set it to 1 minute.

Type:	long
Default:	86400000 (1 day)
Valid Values:	[-1,60000…,3153600000000]

min.compaction.lag.ms

The minimum time a message will remain uncompacted in the topic partition. Only applicable for topics that are being compact.

Type:	long
Default:	0
Valid Values:	[0,…,3153600000000]

delete.retention.ms

The amount of time to retain delete tombstone markers for compacted topics. This setting also gives a bound on the time in which a consumer must complete a read if they begin from offset 0 to ensure that they get a valid snapshot of the final stage (otherwise delete tombstones may be collected before they complete their scan).

Type:	long
Default:	86400000 (1 day)
Valid Values:	[-1,3600000,…,3153600000000]

warpstream.topic.type

The type of this topic. There are currently two types of warpstream topics: classic topics and lightning topics (see

Type:	long
Default:	classic
Valid Values:	[classic, lightning]

warpstream.compression.type.fetch

The compression algorithm that is used to return compressed batches to consumers for this topic.

Type:	string
Default:	agent (lz4)
Valid Values:	[agent, lz4, zstd]

warpstream.schema.registry.type

The schema registry type to use when schema validation is enabled.

Type:	string
Default:	STANDARD
Valid Values:	[STANDARD, AWS_GLUE]

warpstream.schema.validation.warning.only

Should schema validations only return warnings and not block producing of messages to the topic.

Type:	string
Default:	true
Valid Values:	[true, false]

warpstream.key.schema.validation

Should message key schemas be validated when messages are being produced to the topic.

Type:	string
Default:	false
Valid Values:	[true, false]

warpstream.key.subject.name.strategy

Specifies how to construct the subject name for message keys. This determines the subject name allowed during schema validation.

Type:	string
Default:	TopicNameStrategy
Valid Values:	[TopicNameStrategy, RecordNameStrategy, TopicRecordNameStrategy]

warpstream.value.subject.name.strategy

Specifies how to construct the subject name for message values. This determines the subject name allowed during schema validation.

Type:	string
Default:	TopicNameStrategy
Valid Values:	[TopicNameStrategy, RecordNameStrategy, TopicRecordNameStrategy]

warpstream.value.schema.validation

Should message value schemas be validated when messages are being produced to the topic.

Type:	string
Default:	false
Valid Values:	[true, false]

warpstream.partitions_auto_scaler.enabled

Should the partition autoscaler be enabled for the topic.

Type:	string
Default:	false
Valid Values:	[true, false]

warpstream.partitions_auto_scaler.per_partition_throughput_uncompressed_bytes_per_second

The maximum thoughput per partition a topic should have before more partitions are added through autoscaling.

Type:	long
Default:	0
Valid Values:	[0,…]

warpstream.partitions_auto_scaler.max_partition_count

The maximum number of partitions that the topic can autoscale to.

Type:	long
Default:	0
Valid Values:	[0,…,3153600000000]

warpstream.partitions_auto_scaler.partition_count_multiple

Constrains the partitions auto-scaler to only ever scale the topic to multiples of this number. A value of 0 or 1 disables the constraint.

Type:	long
Default:	0
Valid Values:	[0,…]

warpstream.deletion.protection.enabled

Should the topic have delete protection enabled. If enabled the topic cannot be deleted until disabled.

Type:	string
Default:	false
Valid Values:	[true, false]