OpenTelemetry Collector

The Kafka exporteris no longer bundled as part of the standard otel collector, so you’ll need to use one of the contrib releases.

Unauthenticated / BYOC WarpStream Cluster (Example)

The sample configuration file below demonstrates how to configure the OpenTelemetry collector with a Kafka exporter that writes to an unauthenticated WarpStream cluster. This setup makes the most sense when you’re self-hosting the WarpStream Agents in your own cloud account / VPC using the BYOC product.

receivers:
  otlp:
    protocols:
      http:
        cors:
          allowed_origins:
            - "*"

processors:
  batch:

exporters:
  logging:
    loglevel: debug
  kafka:
    brokers: ["<YOUR_WARPSTREAM_BOOTSTRAP_URL>:9092"]
    topic: "traces"
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging, kafka]

Authenticated WarpStream Cluster (Example)

The sample configuration file below demonstrates how to configure the OpenTelemetry collector with a Kafka exporter that writes to an authenticated WarpStream cluster.

receivers:
  otlp:
    protocols:
      http:
        cors:
          allowed_origins:
            - "*"

processors:
  batch:

exporters:
  logging:
    loglevel: debug
  kafka:
    brokers: ["<YOUR_WARPSTREAM_BOOTSTRAP_URL>:9092"]
    topic: "traces"
    auth:
      sasl:
        username: $USERNAME
        password: $PASSWORD
        mechanism: PLAIN
        version: 1
      tls:
        insecure: true
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging, kafka]

Configuring Zone Aware Routing

v0.92.0 of the collector supports specifying Kafka client IDs like so:

receivers:
  otlp:
    protocols:
      http:
        cors:
          allowed_origins:
            - "*"

processors:
  batch:

exporters:
  logging:
    loglevel: debug
  kafka:
    brokers: ["<YOUR_WARPSTREAM_BOOTSTRAP_URL>:9092"]
    client_id: "warpstream_az=us-east-1a"
    topic: "traces"
    auth:
      sasl:
        username: $USERNAME
        password: $PASSWORD
        mechanism: PLAIN
        version: 1
      tls:
        insecure: true
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging, kafka]

See our documentation on configuring WarpStream features via the Kafka Client ID for more details.