benchmark-consumer

Command Info

This tool is used to run a Kafka Consumer Benchmark. For more information on benchmarking see Benchmark.

Command Usage

Usage of benchmark-consumer:
  -bootstrap-host string
      kafka bootstrap host (default "localhost")
  -bootstrap-port int
      kafka bootstrap port (default 9092)
  -client-id string
      client-id to pass along to kafka (default "warpstream-cli")
  -consumer-group string
      the consumer group to use to consume, if unset (default) no consumer group is used
  -enable-tls
      dial with TLS or not
  -fetch-max-bytes int
      the maximum amount of bytes a broker will try to send during a fetch, this corresponds to the java fetch.max.bytes setting (default 50000000)
  -fetch-max-partition-bytes int
      the maximum amount of bytes that will be consumed for a single partition in a fetch request, this corresponds to the java max.partition.fetch.bytes setting (default 25000000)
  -from-beginning
      start with the earliest message present in the topic partition rather than the latest message, when enabled e2e latency can't be calculated
  -kafka-log-level string
      the log level to set on the kafka client, accepted values are DEBUG, INFO, WARN, ERROR (default "WARN")
  -num-clients int
      number of kafka clients (default 3)
  -prometheus-port int
      the port to serve promethes metrics on, -1 to disable (default 8082)
  -sasl-password string
      password for SASL authentication
  -sasl-scram
      uses sasl scram authentication (sasl plain by default)
  -sasl-username string
      username for SASL authentication
  -tls-client-cert-file string
      path to the X.509 certificate file in PEM format for the client
  -tls-client-key-file string
      path to the X.509 private key file in PEM format for the client
  -tls-disable-hostname-verification
      whether to disable hostname verification or not
  -tls-server-ca-cert-file string
      path to the X.509 certificate file in PEM format for the server certificate authority. If not specified, the host's root certificate pool will be used for server certificate verification.
  -topic string
      the topic to consume from

Example

This is an example of running the consumer benchmark tool against a local playground WarpStream Cluster with a single client.

The consumer is consuming data in real-time that is being produced from the producer benchmark tool.

$ warpstream cli benchmark-consumer -topic ws-benchmark -num-clients 1

45784 records consumed (436.00 MiB), 9156.80 records/sec (87.33 MiB/sec), 305.785ms min e2e latency, 423.749632ms avg e2e latency, 552.51ms max e2e latency.
49570 records consumed (472.00 MiB), 9914.00 records/sec (94.55 MiB/sec), 245.006ms min e2e latency, 437.278189ms avg e2e latency, 649.385ms max e2e latency.
49070 records consumed (467.00 MiB), 9814.00 records/sec (93.59 MiB/sec), 238.257ms min e2e latency, 428.520332ms avg e2e latency, 628.591ms max e2e latency.
49550 records consumed (472.00 MiB), 9910.00 records/sec (94.51 MiB/sec), 229.308ms min e2e latency, 445.467432ms avg e2e latency, 642.138ms max e2e latency.
49663 records consumed (473.00 MiB), 9932.60 records/sec (94.72 MiB/sec), 307.591ms min e2e latency, 422.433481ms avg e2e latency, 539.169ms max e2e latency.
49697 records consumed (473.00 MiB), 9939.40 records/sec (94.79 MiB/sec), 310.196ms min e2e latency, 425.985862ms avg e2e latency, 620.136ms max e2e latency.
49678 records consumed (473.00 MiB), 9935.60 records/sec (94.75 MiB/sec), 307.071ms min e2e latency, 435.650686ms avg e2e latency, 658.087ms max e2e latency.