Diagnostics

Diagnostics are Warpstream’s feedback system, designed to provide actionable insights about the health, cost, and performance of your cluster. Each diagnostic measures specific conditions, evaluates their impact, and provides suggestions to address potential issues.

Check out this video to learn more about Diagnostics:

Dimensions of a Diagnostic

Each diagnostic has the following dimensions:

• Type: The category of the diagnostic, such as Health or Cost.
• Name: The component or system that the diagnostic evaluates.
• Successful: Indicates whether the diagnostic has passed.
• Severity: The impact level of the diagnostic, ranging from Low to Critical.
• Muted: Specifies whether the diagnostic is muted and should not generate alerts.

Viewing Diagnostics

Diagnostics can be accessed in two ways:

1. Console UI: The diagnostics are displayed in the Warpstream console interface, providing a visual overview of the cluster’s health and performance.

2. Metrics: Diagnostics are also exposed as agent metrics, facilitating integration with existing monitoring systems. Each diagnostic generates a warpstream_diagnostic_failure gauge metric, where a value of 1 signifies a failing diagnostic and 0 indicates a healthy status.

   This metric includes the following descriptive tags:

   • diagnostic_name: The specific name of the diagnostic check, in snake case.
   • diagnostic_type: The functional category of the diagnostic (e.g., health, cost).
   • severity_low: true if the diagnostic failure severity is low, false otherwise.
   • severity_medium: true if the severity is medium, false otherwise.
   • severity_high: true if the severity is high, false otherwise.
   • severity_critical: true if the severity is critical, false otherwise.
   • muted: Indicates whether the diagnostic check has been temporarily suppressed (muted).
   • diagnostic_agent_group: The agent group that reported the diagnostic. Cluster-wide checks and diagnostics with no applicable or known reporting group use no_group.

   Agent-pushed diagnostics (for example ACL denied, stuck consumer, bucket access, Cross-AZ clients, and Tableflow ingestion issues) emit one series per live agent group. A departed group keeps a failing series until the diagnostic recovers or ages out. Cluster-wide / control-plane diagnostics (for example agent version, missing roles, and partition limits) emit a single series tagged diagnostic_agent_group="no_group".

   Use max by (diagnostic_name) (warpstream_diagnostic_failure) to monitor overall cluster health, and sum by (diagnostic_name) (warpstream_diagnostic_failure) to count how many agent groups are failing a given diagnostic.

Note

warpstream_diagnostic_failure == 1 indicates a failing series. sum(warpstream_diagnostic_failure) counts failing agent groups.

Tip

Diagnostics metrics are available both in the agents and exposed via the Prometheus endpoint for easy scraping by monitoring tools.

Examples of Diagnostics Catalog

Below is some of our diagnostics with a brief description of what each check covers. For the full list go to our console and check the Health tab for any cluster

Diagnostic	Description
ACL Denied	Detects access denials by ACLs (principal/resource/operation/API); verify privileges.
Agent Version	Detects agents running older versions; recommends upgrading to stay within supported and optimized releases.
Cluster Load per Group Role	Checks average and P90 CPU load per agent group/role combination to spot hotspots.
Consumer Groups Configs	Detects risky consumer group timeouts (rebalance/session/heartbeat) that can trigger rebalances.
Cross AZ Kafka Clients	Detects clients connecting across AZs (missing AZ hints); increases latency and cross‑AZ costs.
Embedded Pipeline	Detects pipelines running on agents that also run other roles; recommends dedicated pipeline agents.
Instance Networking	Detects non network‑optimized instance types; recommends network‑optimized for better throughput/latency.
Kafka Client Version	Detects clients using old API versions; upgrade to modern Kafka client versions.
Known Bad Agent Versions	Detects agents running versions with known issues; recommends upgrading beyond affected range(s).
Known Bad Kafka Clients	Detects client libraries/versions with known issues or poor idempotent performance.
Mixed Agent Versions	Detects multiple agent versions co‑existing for too long; standardize versions across agents.
Missing Roles	Ensures at least one agent runs critical roles: jobs, proxy-produce, and proxy-consume.
Partitions Limit	Warns when approaching the cluster partitions limit.
Pipeline Not Runnable	Pipelines are running but there are no agents with the pipelines role in the relevant agent group(s).
Produce Batches	Warns when batches per second are very high, indicating suboptimal client batching/idempotence.
Stuck Consumer	Detects consumers that have not progressed for a while (lagging at a fixed offset).
Tableflow tables Limit	Warns when approaching the Tableflow tables limit in a Datalake cluster.

To investigate a failing diagnostic, it’s often helpful to introspect your cluster using the Events Explorer. Related agent-side activity (for example ACL denials or pipeline errors) appears in event types such as acl_logs and pipeline_logs. Diagnostic state-change events themselves are stored as diagnostics_logs. On Kafka clusters those control-plane events require Events to be enabled and at least one Agent with the jobs role running v829 or higher. You can also access diagnostics and query events from your IDE using an AI assistant via the MCP Server.