# Important Metrics and Logs
On this page, we include a sample list of the most important logs and metrics emitted by the Agents.
Before reading this documentation page, please familiarize yourself with [how logs and metrics are emitted from the Agents](/warpstream/agent-setup/monitor-the-warpstream-agents).
:::caution
**Enable High-Cardinality Tags**
Some metrics support the use of tags with potentially high cardinality, such as tags based on topics. **This feature is disabled by default**.
**To enable high-cardinality tags:**
* Use the command-line flag `-kafkaHighCardinalityMetrics`.
* Alternatively, set the environment variable `WARPSTREAM_KAFKA_HIGH_CARDINALITY_METRICS=true`.
Tags that require enabling are clearly marked with "(requires enabling high-cardinality tags)" next to their name.
Furthermore, per-topic distribution / histogram metrics have 10x to 20x higher cardinality than even the regular high cardinality metrics, so any high cardinality (per-topic) metrics with type `histogram` will not be emitted with per-topic tags unless the `-kafkaHighCardinalityDistributionMetrics` flag or `WARPSTREAM_KAFKA_HIGH_CARDINALITY_DISTRIBUTION_METRICS` environment variable is set to true.
:::
:::note
**Datadog metrics**
Starting from Warpstream Agent `v679` all metrics on Datadog will start with `warpstream.` and no longer `warpstream_` . All references to metrics in our doc will keep mentioning metrics starting with `warpstream_` so you have to do the conversion when you are using Datadog and a Warpstream Agent recent enough.
You can fall back to the previous behavior by setting the `WARPSTREAM_DATADOG_NORMALIZER_PREFIX_WITH_DOT` environment variable to `false`.
This change comes along the official release of our Datadog integration, making all the Warpstream Agent metrics free if you install the integration (and use the new naming convention).
:::
## Overview
System performance metrics and logs, focusing on error detection and resource consumption.
**\[logs]**** Error Logs**
* query: `status:error`
* metric: \*
* note that some amount of error logs can be normal depending on the situation. Please contact the WarpStream team if you think a particular error log is too noisy!
**\[metrics]**** Memory usage by host**
* metric: `container.memory.usage`
* group\_by: `host`
**\[metrics]**** Used Cores**
* metric: `container.cpu.usage`
* group\_by:
**\[metrics]**** Used Cores By Host**
* metric: `container.cpu.usage`
* group\_by: `host`
## Kafka
Metrics and logs associated with the Kafka protocol provide insights into message handling, latency, and throughput.
**\[metrics]**** Produce Throughput (uncompressed bytes)**
* metric: `warpstream_agent_kafka_produce_uncompressed_bytes_counter`
* group\_by: `topic` (requires enabling high-cardinality tags)
* type: counter
* unit: bytes
* description: number of uncompressed bytes that were produced.
**\[metrics]**** Produce Throughput (compressed bytes)**
* metric: `warpstream_agent_kafka_produce_compressed_bytes_counter`
**\[metrics]**** Produce Throughput (records)**
* metric: `warpstream_agent_segment_batcher_flush_num_records_counter`
* group\_by:
* type: counter
* unit: bytes
* description: number of records that were produced.
**\[metrics]**** Fetch Throughput (uncompressed bytes)**
* metric: `warpstream_agent_kafka_fetch_uncompressed_bytes_counter`
* group\_by: `topic` (requires enabling high-cardinality tags)
* type: counter
* unit: bytes
* description: number of uncompressed bytes that were fetched.
**\[metrics]**** Fetch Throughput (compressed bytes)**
* metric: `warpstream_agent_kafka_fetch_compressed_bytes_counter`
* group\_by: `topic` (requires enabling high-cardinality tags)
* type: counter
* unit: bytes
* description: number of compressed bytes that were fetched.
**\[metrics]**** Consumer Groups Lag**
* metric: `warpstream_consumer_group_lag`
* group\_by: `virtual_cluster_id`, `topic`, `consumer_group`, `partition`
* tags: `virtual_cluster_id`, `topic`, `consumer_group`, `partition`
* Partition is disabled by default. Set `-disableConsumerGroupsMetricsTags ""` flag or `WARPSTREAM_DISABLE_CONSUMER_GROUPS_METRICS_TAGS=""` environment variable on the Agents to enable the partition tag.
* See these docs for more details: [https://docs.warpstream.com/warpstream/agent-setup/monitor-the-warpstream-agents/monitoring-consumer-groups#metrics](/warpstream/agent-setup/monitor-the-warpstream-agents/monitoring-consumer-groups#metrics)
* type: gauge
* unit: Kafka offsets
* description: consumer group lag measured in _offsets_.
**\[metrics]**** Kafka Inflight Connections**
* metric: `warpstream_agent_kafka_inflight_connections`
* group\_by:
* type: gauge
* unit: n/a
* description: number of currently inflight / active connections.
**\[metrics]**** Kafka Inflight Requests (per Connection)**
* metric: `warpstream_agent_kafka_inflight_request_per_connection`
* group\_by:
* type: histogram
* unit: n/a
* description: number of currently in-flight Kafka protocol requests for individual connections.
**\[metrics]**** Kafka Request Outcome**
* metric: `warpstream_agent_kafka_request_outcome`
* group\_by: `kafka_key,outcome`
* type: counter
* unit: n/a
* description: outcome (success, error, etc) for each Kafka protocol request.
**\[metrics]**** Kafka Request Latency**
* metric: `warpstream_agent_kafka_request_latency`
* group\_by: `kafka_key`
* type: histogram
* unit: seconds
* description: latency for processing each Kafka protocol request.
**\[metrics]**** Max Offset**
* metric: `warpstream_max_offset`
* group\_by: `virtual_cluster_id`, `topic`, `partition`
* tags: `virtual_cluster_id`, `topic`, `partition`
* Partition is disabled by default. Set `-disableConsumerGroupsMetricsTags ""` flag or `WARPSTREAM_DISABLE_CONSUMER_GROUPS_METRICS_TAGS=""` environment variable on the Agents to enable the partition tag.
* See these docs for more details: [https://docs.warpstream.com/warpstream/agent-setup/monitor-the-warpstream-agents/monitoring-consumer-groups#metrics](/warpstream/agent-setup/monitor-the-warpstream-agents/monitoring-consumer-groups#metrics)
* type: gauge
* unit: Kafka offset
* description: max offset for every topic-partition. Can be useful for monitoring lag for applications that don't use consumer groups and manage offsets externally.
**\[metrics]**** Kafka Topic Count**
* metric: `warpstream_topics_count`
* group\_by: `virtual_cluster_id`
* type: gauge
* unit: n/a
* description: how many topics are currently in your cluster
**\[metrics]**** Kafka Topic Limit**
* metric: `warpstream_topics_count_limit`
* group\_by: `virtual_cluster_id`
* type: gauge
* unit: n/a
* description: how many topics are allowed in your cluster. Upgrade your cluster tier or contact the WarpStream team if more are needed
**\[metrics]**** Kafka Partition Count**
* metric: `warpstream_partitions_count`
* group\_by: `virtual_cluster_id`
* type: gauge
* unit: n/a
* description: how many partitions are currently in your cluster
**\[metrics]**** Kafka Partition Limit**
* metric: `warpstream_partitions_count_limit`
* group\_by: `virtual_cluster_id`
* type: gauge
* unit: n/a
* description: how many partitions are allowed in your cluster. Upgrade your cluster tier or contact the WarpStream team if more are needed
**\[metrics]**** Kafka Records Count**
* metric: `warpstream_num_records`
* group\_by: `topic`, `virtual_cluster_id`
* type: gauge
* unit: n/a
* description: how many records are currently in a given topic or cluster
* note this number might not match the number of active keys if you are using compacted topics
## Control Plane
Visualizing WarpStream control plane latency and error rates can be useful for debugging.
**\[metrics]**** Operations Outcome**
* metric: `warpstream_agent_control_plane_operation_counter`
* group\_by: `virtual_cluster_id`, `outcome`, `operation`
* tags: `virtual_cluster_id`, `outcome`, `operation`
* type: counter
* unit: request
* description: count of RPCs between Agents and control plane, groupable by outcome (success/error) and operation (RPC type).
**\[metrics]**** Operations Latency**
* metric: `warpstream_agent_control_plane_operation_latency`
* group\_by: `virtual_cluster_id`, `outcome`, `operation`
* tags: `virtual_cluster_id`, `outcome`, `operation`
* type: histogram
* unit: seconds
* description: latency of RPCs between Agents and control plane, groupable by outcome (success/error) and operation (RPC type).
## Auto Migration
**\[metrics]**** Proxy Throughput (compressed bytes)**
* metric: `warpstream_agent_kafka_produce_forwarded_compressed_bytes_counter`
* group\_by: `topic` (requires enabling high-cardinality tags), `outcome`
* type: counter
* unit: bytes
* description: number of compressed bytes that were proxied to the source cluster.
**\[metrics] Proxy Throughput (records)**
* metric: `warpstream_agent_kafka_produce_forwarded_records_counter`
* group\_by: `topic` (requires enabling high-cardinality tags), `outcome`
* type: counter
* unit: records
* description: number of records that were proxied to the source cluster.
**\[metrics] Unproxied Proxy Throughput (records)**
* metric: `warpstream_orbit_auto_migration_unproxied_source_writes_num_records`
* group\_by: `topic`
* type: counter
* unit: records
* description: the number of detected unproxied records to the source cluster that bypassed the WarpStream proxy.
## Schema Registry
Metrics and logs associated with the hosted schema registry provide insights into request handling, latency, and throughput.
:::caution
**Enable Schema Registry Request Logs**
Logging schema registry requests is **disabled by default**.
**To enable request logs:**
* Use the command-line flag `-schemaRegistryEnableLogRequest`.
* Alternatively, set the environment variable `WARPSTREAM_SCHEMA_REGISTRY_ENABLE_LOG_REQUEST=true`.
:::
**\[metrics]**** Schema Registry Requests**
* metric: `warpstream_agent_schema_registry_outcome`
* group\_by: `schema_registry_operation,outcome`
* type: counter
* unit: n/a
* description: outcome (success, error, etc) for each Schema Registry request.
**\[metrics]**** Schema Registry Latency**
* metric: `warpstream_agent_schema_registry_request_latency`
* group\_by: `schema_registry_operation,outcome`
* type: histogram
* unit: seconds
* description: latency for processing each Schema protocol request.
**\[metrics]**** Schema Registry Inflight Connections**
* metric: `warpstream_agent_schema_registry_inflight_connections`
* group\_by:
* type: gauge
* unit: n/a
* description: number of currently inflight / active connections.
**\[logs]**** Schema Registry Request Logs**
* query: `service:sr-agent schema_registry_request`
* group\_by: `outcome`, `request_type`, `schema_id`, `subject`, `version`
* description: every schema registry request will emit a log with the following attributes: `request_type` and `outcome`. Some requests will also emit additional attributes such as `schema_id`, `subject`, etc if applicable. This is disabled by default and requires setting the command line flag `-schemaRegistryEnableLogRequest` or setting the environment variable `WARPSTREAM_SCHEMA_REGISTRY_ENABLE_LOG_REQUEST=true`
**\[metrics]**** Num Invalid Records**
* metric: `warpstream_schema_validation_invalid_record_count`
* group\_by: `topic`, `reason`
* type: counter
* unit: n/a
* description: counter of the number of invalid records that the agent detects when schema validation is enabled.
* note that the topic would only be a tag if high cardinality is enabled.
**\[metrics]**** Schema Linking Number of Source subject versions**
* metric: `warpstream_schema_linking_source_subject_versions_count`
* group\_by: `sync_id`, `config_id`
* type: gauge
* unit: n/a
* description: number of source subject versions that Schema Linking is currently managing
**\[metrics]**** Schema Linking Number of Newly Migrated Subject Versions**
* metric: `warpstream_schema_linking_newly_migrated_subject_versions`
* group\_by: `sync_id`, `config_id`
* type: gauge
* unit: n/a
* description: number of newly migrated subject versions performed by the latest sync, this should usually be zero unless new schemas are found. Schemas are only new for one sync and the frequency is configurable with a default of 5m
**\[metrics]**** Schema Versions Count**
* metric: `warpstream_schema_versions_count`
* group\_by:
* type: gauge
* unit: n/a
* description: number of schemas currently in your registry.
**\[metrics]**** Schema Versions Limit**
* metric: `warpstream_schema_versions_limit`
* group\_by:
* type: gauge
* unit: n/a
* description: number of schemas allowed in your registry.
## Background Jobs
The control plane assigns agents background jobs for things like compaction or retention. These are the metrics and logs on the efficiency and status of background operations, with a focus on compaction processes and the scanning of obsolete files.
**\[logs]**** Compactions by Status and Level**
* query: `service:warp-agent @stream_job_input.type:COMPACTION_JOB_TYPE status:info`
* metric: \*
* group\_by: `status`,`@stream_job_input.compaction.compaction_level`
* description: number of compactions by compaction level and status (success, error). Occasional compaction errors are normal and expected, but most compactions should succeed.
**\[metrics]**** Executed Jobs**
* metric: `warpstream_agent_run_and_ack_job_outcome`
* group\_by: `job_type`
* tags: `job_type` `outcome`
* type: counter
* unit: n/a
* description: outcome (and successful acknowledgement back to the control plane) of all jobs, groupable by job type and outcome. Jobs may fail intermittently, but most jobs should complete successfully.
**\[metrics]**** Compaction Files per Level (Indicator of Compaction Lag)**
* metric: `warpstream_files_count`
* group\_by: `compaction_level`
* type: gauge
* unit: n/a
* description: number of files in the LSM for each compaction level. The number of L2 files may be high for high volume / long retention workloads, but the number of files at L0 and L1 should always be low (< 1000 each).
**\[metrics]**** Dead Files Scanner: Checked vs Deleted Files**
* metric: `warpstream_agent_deadscanner_outcomes`
* group\_by: `outcome`
* tags: `outcome`
* type: counter
* unit: n/a
* description: the "deadscanner" is a job type in WarpStream that instructs the Agents to scan the object storage bucket for files that are "dead". A file is considered dead if it exists in the object store, but the WarpStream control plane / metadata store has no record of it, indicating it failed to be committed or was deleted by data expiration / compaction.
**\[logs]**** P99 Compaction Duration by Level**
* query: `service:warp-agent @stream_job_input.type:COMPACTION_JOB_TYPE status:info`
* metric: `@duration_ms`
* group\_by: `status`,`@stream_job_input.compaction.compaction_level`
* description: duration of compactions by compaction level. L1 and L2 compaction duration will vary based on workload, but L0 compactions should always be fast (<20 seconds).
**\[logs]**** Compaction File Output Size**
* query: `status:info`
* metric: `@stream_job_output.compaction.file_metadatas.index_offset`
* group\_by: `source`,`@stream_job_input.compaction.compaction_level`
* description: compressed size of files generated by compaction. Useful for understanding the size of different files at different levels, but not something that needs to be closely monitored or paid attention to.
## Object Storage
Metrics and logs on object storage operations' performance and usage patterns, offering insights into data retrieval, storage efficiency, and caching mechanisms.
**\[metrics]**** S3 Operations (PUT)**
* metric: `warpstream_blob_store_operation_latency`
* filter\_tag: `operation:put_bytes`, `operation:put_stream`
* group\_by: operation
* type: histogram
* unit: seconds
* description: latency to perform PUT requests. Spikes of this value can indicate issues with the underlying object store.
* note this metric is a histogram, so even if it emits latency, you can count the number of items emitted and get the number of operations.
**\[metrics]**** S3 Operations (GET)**
* metric: `warpstream_blob_store_operation_latency`
* filter\_tag: `operation:get_stream`, `operation:get_stream_range`
* group\_by: operation
* type: histogram
* unit: seconds
* description: latency for time to first byte for GET requests. Spikes of this value can indicate issues with the underlying object store.
* note this metric is a histogram, so even if it emits latency, you can count the number of items emitted and get the number of operations.