Consumer Group Protocol (KIP-848) Migration Guide
KIP-848 is the next generation of Kafka’s consumer group protocol. WarpStream has full support for this protocol including migration of groups on the older protocol to the newer protocol.
Configuration
Section titled “Configuration”Broker configs
Section titled “Broker configs”| Setting | Default | Description |
|---|---|---|
group.consumer.session.timeout.ms |
45000 |
How long a member can go without heartbeating before it’s removed from the group. |
group.consumer.heartbeat.interval.ms |
5000 |
How often members send heartbeats. |
group.consumer.max.size |
32000 |
Maximum number of members in a group. |
group.consumer.migration.policy |
bidirectional |
Whether a group with active members can switch protocols. See Migration policy. |
These can be altered using the alter config or incremental alter config kafka apis.
Group configs
Section titled “Group configs”You can override the session timeout and heartbeat interval for an individual group with consumer.session.timeout.ms and consumer.heartbeat.interval.ms consumer group configs using the incremental
alter configs kafka api.
Migrating from the older classic protocol
Section titled “Migrating from the older classic protocol”Migrated consumers will resume from the last committed offset for a partition.
1. Upgrade your clients
Section titled “1. Upgrade your clients”Upgrade to a client version that supports KIP-848. Opting in is a client-side setting: for the Java client and librdkafka, set group.protocol=consumer. See Client migration guides for each client’s minimum version and instructions.
2. Update your consumer configuration
Section titled “2. Update your consumer configuration”Remove settings that only apply to the classic protocol: session.timeout.ms, heartbeat.interval.ms, and partition.assignment.strategy. Some clients, including the Java client, refuse to start if they’re set together with group.protocol=consumer. Use the cluster-wide or per-group settings in Configuration instead.
3. Rolling restart your consumers
Section titled “3. Rolling restart your consumers”Rolling migration. The first client which connects with WarpStream on the new protocol will cause the group to automatically convert to the modern protocol. Existing members still heartbeating on the old protocol are supported. As all clients finish rotating, all members of the group will be on the new protocol. This requires the migration policy to be upgrade or bidirectional.
WarpStream won’t convert a group that still has classic members if any of the following are true:
- The migration policy doesn’t allow upgrades.
- The group uses a custom assignor which isn’t supported in the old protocol.
Migration policy
Section titled “Migration policy”group.consumer.migration.policy controls whether a group with active members can switch protocols. Empty groups can always switch.
| Value | Classic to KIP-848 | KIP-848 to classic |
|---|---|---|
disabled |
Offline only | Offline only |
upgrade |
Offline or rolling | Offline only |
downgrade |
Offline only | Offline or rolling |
bidirectional |
Offline or rolling | Offline or rolling |
The default is bidirectional.
Rolling back
Section titled “Rolling back”To roll back, switch clients back to the classic protocol, for example with group.protocol=classic. The group will only roll back once every member on the newer protocol has left the group.
Client migration guides
Section titled “Client migration guides”Each client library documents its own KIP-848 migration steps. This table covers the clients in Tuning for Performance.
| Client | KIP-848 support | Migration guide |
|---|---|---|
| Java client | 4.0 and later | Consumer Rebalance Protocol: Upgrade & Downgrade |
| librdkafka | 2.12.0 and later | Next Generation Consumer Group Protocol (KIP-848) |
| confluent-kafka-go | 2.12.0 and later | KIP-848 Migration Guide |
| confluent-kafka-python | 2.12.0 and later | KIP-848 Migration Guide |
| confluent-kafka-javascript | 1.6.0 and later | Follow the librdkafka guide. Set group.protocol outside the kafkaJS config block. |
| franz-go | 1.22.0 and later | Enable kgo.ServerSideBalancer, then follow the upgrade and downgrade flow. |
| Segment kafka-go | Not supported | None |
| Sarama | Not supported | Tracking issue |
| KafkaJS | Not supported | Migrate to confluent-kafka-javascript, then follow the KIP-848 guide. |
| Kafka Connect | Not supported. Connect workers use their own group protocol. | None |
| Spark | Not applicable. Spark tracks offsets in its own checkpoints. | Kafka-specific configurations |