Essential Metrics for CockroachDB Self-Hosted Deployments

These essential CockroachDB metrics enable you to build custom dashboards with the following tools:

• Grafana
• Datadog Integration - The Datadog Integration Metric Name column lists the corresponding Datadog metric which requires the cockroachdb. prefix.

The Usage column explains why each metric is important to visualize in a custom dashboard and how to make both practical and actionable use of the metric in a production deployment.

Platform

CockroachDB Metric Name	Datadog Integration Metric Name (add cockroachdb. prefix)	Description	Usage
sys.cpu.combined.percent-normalized	sys.cpu.combined.percent.normalized	Current user+system CPU percentage consumed by the CRDB process, normalized by number of cores	This metric gives the CPU utilization percentage by the CockroachDB process. If it is equal to 1 (or 100%), then the CPU is overloaded. The CockroachDB process should not be running with over 80% utilization for extended periods of time (hours). This metric is used in the DB Console CPU Percent graph.
sys.cpu.host.combined.percent-normalized	NOT AVAILABLE	Current user+system CPU percentage consumed by all processes on the host OS, normalized by number of cores. If the CRDB process is run in a containerized environment, the host OS is the container since the CRDB process cannot inspect CPU usage beyond the container.	This metric gives the CPU utilization percentage of the underlying server, virtual server or container hosting the CockroachDB process. It includes CockroachDB process and non-CockroachDB process usage. If it is equal to 1 (or 100%), then the CPU is overloaded. The CockroachDB process should not be running in an environment with an overloaded state for extended periods of time (hours). This metric is used in the DB Console Host CPU Percent graph.
sys.cpu.user.percent	sys.cpu.user.percent	Current user CPU percentage consumed by the CRDB process	This metric gives the CPU usage percentage at the user level by the CockroachDB process only. This is similar to the Linux top command output. The metric value can be more than 1 (or 100%) on multi-core systems. It is best to combine user and system metrics.
sys.cpu.sys.percent	sys.cpu.sys.percent	Current system CPU percentage consumed by the CRDB process	This metric gives the CPU usage percentage at the system (Linux kernel) level by the CockroachDB process only. This is similar to the Linux top command output. The metric value can be more than 1 (or 100%) on multi-core systems. It is best to combine user and system metrics.
sys.rss	sys.rss	Current process memory (RSS)	This metric gives the amount of RAM used by the CockroachDB process. Persistently low values over an extended period of time suggest there is underutilized memory that can be put to work with adjusted settings for --cache or --max_sql_memory or both. Conversely, a high utilization, even if a temporary spike, indicates an increased risk of Out-of-memory (OOM) crash (particularly since the swap is generally disabled).
sql.mem.root.current	sql.mem.root.current	Current sql statement memory usage for root	This metric shows how memory set aside for temporary materializations, such as hash tables and intermediary result sets, is utilized. Use this metric to optimize memory allocations based on long term observations. The maximum amount is set with --max_sql_memory. If the utilization of sql memory is persistently low, perhaps some portion of this memory allocation can be shifted to --cache.
sys.host.disk.write.bytes	sys.host.disk.write.bytes	Bytes written to all disks since this process started	This metric reports the effective storage device write throughput (MB/s) rate. To confirm that storage is sufficiently provisioned, assess the I/O performance rates (IOPS and MBPS) in the context of the sys.host.disk.iopsinprogress metric.
sys.host.disk.write.count	sys.host.disk.write	Disk write operations across all disks since this process started	This metric reports the effective storage device write IOPS rate. To confirm that storage is sufficiently provisioned, assess the I/O performance rates (IOPS and MBPS) in the context of the sys.host.disk.iopsinprogress metric.
sys.host.disk.read.bytes	sys.host.disk.read.bytes	Bytes read from all disks since this process started	This metric reports the effective storage device read throughput (MB/s) rate. To confirm that storage is sufficiently provisioned, assess the I/O performance rates (IOPS and MBPS) in the context of the sys.host.disk.iopsinprogress metric.
sys.host.disk.read.count	sys.host.disk.read	Disk read operations across all disks since this process started	This metric reports the effective storage device read IOPS rate. To confirm that storage is sufficiently provisioned, assess the I/O performance rates (IOPS and MBPS) in the context of the sys.host.disk.iopsinprogress metric.
sys.host.disk.iopsinprogress	sys.host.disk.iopsinprogress	IO operations currently in progress on this host	This metric gives the average queue length of the storage device. It characterizes the storage device's performance capability. All I/O performance metrics are Linux counters and correspond to the avgqu-sz in the Linux iostat command output. You need to view the device queue graph in the context of the actual read/write IOPS and MBPS metrics that show the actual device utilization. If the device is not keeping up, the queue will grow. Values over 10 are bad. Values around 5 mean the device is working hard trying to keep up. For internal (on chassis) NVMe devices, the queue values are typically 0. For network connected devices, such as AWS EBS volumes, the normal operating range of values is 1 to 2. Spikes in values are OK. They indicate an I/O spike where the device fell behind and then caught up. End users may experience inconsistent response times, but there should be no cluster stability issues. If the queue is greater than 5 for an extended period of time and IOPS or MBPS are low, then the storage is most likely not provisioned per Cockroach Labs guidance. In AWS EBS, it is commonly an EBS type, such as gp2, not suitable as database primary storage. If I/O is low and the queue is low, the most likely scenario is that the CPU is lacking and not driving I/O. One such case is a cluster with nodes with only 2 vcpus which is not supported sizing for production deployments. There are quite a few background processes in the database that take CPU away from the workload, so the workload is just not getting the CPU. Review storage and disk I/O.
sys.host.net.recv.bytes	sys.host.net.recv.bytes	Bytes received on all network interfaces since this process started	This metric gives the node's ingress/egress network transfer rates for flat sections which may indicate insufficiently provisioned networking or high error rates. CockroachDB is using a reliable TCP/IP protocol, so errors result in delivery retries that create a "slow network" effect.
sys.host.net.send.bytes	sys.host.net.send.bytes	Bytes sent on all network interfaces since this process started	This metric gives the node's ingress/egress network transfer rates for flat sections which may indicate insufficiently provisioned networking or high error rates. CockroachDB is using a reliable TCP/IP protocol, so errors result in delivery retries that create a "slow network" effect.
clock-offset.meannanos	clock.offset.meannanos	Mean clock offset with other nodes	This metric gives the node's clock skew. In a well-configured environment, the actual clock skew would be in the sub-millisecond range. A skew exceeding 5 ms is likely due to a NTP service mis-configuration. Reducing the actual clock skew reduces the probability of uncertainty related conflicts and corresponding retires which has a positive impact on workload performance. Conversely, a larger actual clock skew increases the probability of retries due to uncertainty conflicts, with potentially measurable adverse effects on workload performance.

Storage

CockroachDB Metric Name	Datadog Integration Metric Name (add cockroachdb. prefix)	Description	Usage
capacity	capacity.total	Total storage capacity	This metric gives total storage capacity. Measurements should comply with the following rule: CockroachDB storage volumes should not be utilized more than 60% (40% free space).
capacity.available	capacity.available	Available storage capacity	This metric gives available storage capacity. Measurements should comply with the following rule: CockroachDB storage volumes should not be utilized more than 60% (40% free space).
capacity.used	capacity.used	Used storage capacity	This metric gives used storage capacity. Measurements should comply with the following rule: CockroachDB storage volumes should not be utilized more than 60% (40% free space).
storage.write-stalls	storage.write.stalls	Number of instances of intentional write stalls to backpressure incoming writes	This metric reports actual disk stall events. Ideally, investigate all reports of disk stalls. As a pratical guideline, one stall per minute is not likely to have a material impact on workload beyond an occasional increase in response time. However one stall per second should be viewed as problematic and investigated actively. It is particularly problematic if the rate persists over an extended period of time, and worse, if it is increasing.
rocksdb.compactions	rocksdb.compactions.total	Number of SST compactions	This metric reports the number of a node's LSM compactions. If the number of compactions remains elevated while the LSM health does not improve, compactions are not keeping up with the workload. If the condition persists for an extended period, the cluster will initially exhibit performance issues that will eventually escalate into stability issues.
rocksdb.block.cache.hits	rocksdb.block.cache.hits	Count of block cache hits	This metric gives hits to block cache which is reserved memory. It is allocated upon the start of a node process by the --cache flag and never shrinks. By observing block cache hits and misses, you can fine-tune memory allocations in the node process for the demands of the workload.
rocksdb.block.cache.misses	rocksdb.block.cache.misses	Count of block cache misses	This metric gives misses to block cache which is reserved memory. It is allocated upon the start of a node process by the --cache flag and never shrinks. By observing block cache hits and misses, you can fine-tune memory allocations in the node process for the demands of the workload.

Health

CockroachDB Metric Name	Datadog Integration Metric Name (add cockroachdb. prefix)	Description	Usage
sys.uptime	sys.uptime	Process uptime	This metric measures the length of time, in seconds, that the CockroachDB process has been running. Monitor this metric to detect events such as node restarts, which may require investigation or intervention.
admission.io.overload	admission.io.overload	1-normalized float indicating whether IO admission control considers the store as overloaded with respect to compaction out of L0 (considers sub-level and file counts).	If the value of this metric exceeds 1, then it indicates overload. You can also look at the metrics storage.l0-num-files, storage.l0-sublevels or rocksdb.read-amplification directly. A healthy LSM shape is defined as “read-amp < 20” and “L0-files < 1000”, looking at cluster settings admission.l0_sub_level_count_overload_threshold and admission.l0_file_count_overload_threshold respectively.
admission.wait_durations.kv-p75	admission.wait.durations.kv	Wait time durations for requests that waited	This metric shows if CPU utilization-based admission control feature is working effectively or potentially overaggressive. This is a latency histogram of how much delay was added to the workload due to throttling by CPU control. If observing over 100ms waits for over 5 seconds while there was excess CPU capacity available, then the admission control is overly aggressive.
admission.wait_durations.kv-stores-p75	admission.wait.durations.kv_stores	Wait time durations for requests that waited	This metric shows if I/O utilization-based admission control feature is working effectively or potentially overaggressive. This is a latency histogram of how much delay was added to the workload due to throttling by I/O control. If observing over 100ms waits for over 5 seconds while there was excess I/O capacity available, then the admission control is overly aggressive.
sys.runnable.goroutines.per.cpu	sys.runnable.goroutines.per_cpu	Average number of goroutines that are waiting to run, normalized by number of cores	If this metric has a value over 30, it indicates a CPU overload. If the condition lasts a short period of time (a few seconds), the database users are likely to experience inconsistent response times. If the condition persists for an extended period of time (tens of seconds, or minutes) the cluster may start developing stability issues. Review CPU planning.

Network

CockroachDB Metric Name	Datadog Integration Metric Name (add cockroachdb. prefix)	Description	Usage
rpc.connection.avg_round_trip_latency	rpc.connection.avg_round_trip_latency	Sum of exponentially weighted moving average of round-trip latencies, as measured through a gRPC RPC. Dividing this gauge by rpc.connection.healthy gives an approximation of average latency, but the top-level round-trip-latency histogram is more useful. Instead, users should consult the label families of this metric if they are available (which requires Prometheus and the cluster setting server.child_metrics.enabled); these provide per-peer moving averages. This metric does not track failed connection. A failed connection's contribution is reset to zero.	This metric is helpful in understanding general network issues outside of CockroachDB that could be impacting the user’s workload.
rpc.connection.failures	rpc.connection.failures.count	Counter of failed connections. This includes both the event in which a healthy connection terminates as well as unsuccessful reconnection attempts. Connections that are terminated as part of local node shutdown are excluded. Decommissioned peers are excluded.	See Description.
rpc.connection.healthy	rpc.connection.healthy	Gauge of current connections in a healthy state (i.e., bidirectionally connected and heartbeating).	See Description.
rpc.connection.healthy_nanos	rpc.connection.healthy_nanos	Gauge of nanoseconds of healthy connection time. On the Prometheus endpoint scraped when the cluster setting server.child_metrics.enabled is set, this gauge allows you to see the duration for which a given peer has been connected in a healthy state.	This can be useful for monitoring the stability and health of connections within your CockroachDB cluster.
rpc.connection.heartbeats	rpc.connection.heartbeats.count	Counter of successful heartbeats.	See Description.
rpc.connection.unhealthy	rpc.connection.unhealthy	Gauge of current connections in an unhealthy state (not bidirectionally connected or heartbeating).	If the value of this metric is greater than 0, this could indicate a network partition.
rpc.connection.unhealthy_nanos	rpc.connection.unhealthy_nanos	Gauge of nanoseconds of unhealthy connection time. On the Prometheus endpoint scraped when the cluster setting server.child_metrics.enabled is set, this gauge allows you to see the duration for which a given peer has been unreachable.	If this duration is greater than 0, this could indicate how long a network partition has been occurring.

Expiration of license and certificates

CockroachDB Metric Name	Datadog Integration Metric Name (add cockroachdb. prefix)	Description	Usage
seconds.until.enterprise.license.expiry	seconds.until.enterprise.license.expiry	Seconds until enterprise license expiry (0 if no license present or running without enterprise features)	See Description.
security.certificate.expiration.ca	security.certificate_expiration.ca	Expiration for the CA certificate. 0 means no certificate or error	See Description.
security.certificate.expiration.client-ca	security.certificate_expiration.client_ca	Expiration for the client CA certificate. 0 means no certificate or error	See Description.
security.certificate.expiration.ui	security.certificate_expiration.ui	Expiration for the UI certificate. 0 means no certificate or error	See Description.
security.certificate.expiration.ui-ca	security.certificate_expiration.ui_ca	Expiration for the UI CA certificate. 0 means no certificate or error	See Description.
security.certificate.expiration.node	security.certificate_expiration.node	Expiration for the node certificate. 0 means no certificate or error	See Description.
security.certificate.expiration.node-client	security.certificate_expiration.node_client	Expiration for the node's client certificate. 0 means no certificate or error	See Description.

KV distributed

CockroachDB Metric Name	Datadog Integration Metric Name (add cockroachdb. prefix)	Description	Usage
liveness.heartbeatlatency	liveness.heartbeatlatency-p90	Node liveness heartbeat latency	If this metric exceeds 1 second, it is a sign of cluster instability.
liveness.livenodes	liveness.livenodes	Number of live nodes in the cluster (will be 0 if this node is not itself live)	This is a critical metric that tracks the live nodes in the cluster.
distsender.rpc.sent.nextreplicaerror	distsender.rpc.sent.nextreplicaerror	Number of replica-addressed RPCs sent due to per-replica errors	RPC errors do not necessarily indicate a problem. This metric tracks remote procedure calls that return a status value other than "success". A non-success status of an RPC should not be misconstrued as a network transport issue. It is database code logic executed on another cluster node. The non-success status is a result of an orderly execution of an RPC that reports a specific logical condition.
distsender.errors.notleaseholder	distsender.errors.notleaseholder	Number of NotLeaseHolderErrors encountered from replica-addressed RPCs	Errors of this type are normal during elastic cluster topology changes when leaseholders are actively rebalancing. They are automatically retried. However they may create occasional response time spikes. In that case, this metric may provide the explanation of the cause.

KV replication

CockroachDB Metric Name	Datadog Integration Metric Name (add cockroachdb. prefix)	Description	Usage
leases.transfers.success	leases.transfers.success	Number of successful lease transfers	A high number of lease transfers is not a negative or positive signal, rather it is a reflection of the elastic cluster activities. For example, this metric is high during cluster topology changes. A high value is often the reason for NotLeaseHolderErrors which are normal and expected during rebalancing. Observing this metric may provide a confirmation of the cause of such errors.
rebalancing.queriespersecond	rebalancing.queriespersecond	Number of kv-level requests received per second by the store, considering the last 30 minutes, as used in rebalancing decisions.	This metric shows hotspots along the queries per second (QPS) dimension. It provides insights into the ongoing rebalancing activities.
ranges	ranges	Number of ranges	This metric provides a measure of the scale of the data size.
replicas	replicas.total	Number of replicas	This metric provides an essential characterization of the data distribution across cluster nodes.
replicas.leaseholders	replicas.leaseholders	Number of lease holders	This metric provides an essential characterization of the data processing points across cluster nodes.
ranges.underreplicated	ranges.underreplicated	Number of ranges with fewer live replicas than the replication target	This metric is an indicator of replication issues. It shows whether the cluster has data that is not conforming to resilience goals. The next step is to determine the corresponding database object, such as the table or index, of these under-replicated ranges and whether the under-replication is temporarily expected. Use the statement SELECT table_name, index_name FROM [SHOW RANGES WITH INDEXES] WHERE range_id = {id of under-replicated range};
ranges.unavailable	ranges.unavailable	Number of ranges with fewer live replicas than needed for quorum	This metric is an indicator of replication issues. It shows whether the cluster is unhealthy and can impact workload. If an entire range is unavailable, then it will be unable to process queries.
queue.replicate.replacedecommissioningreplica.error	queue.replicate.replacedecommissioningreplica.error.count	Number of failed decommissioning replica replacements processed by the replicate queue	Refer to Decommission the node.
range.splits	range.splits.total	Number of range splits	This metric indicates how fast a workload is scaling up. Spikes can indicate resource hot spots since the split heuristic is based on QPS. To understand whether hot spots are an issue and with which tables and indexes they are occurring, correlate this metric with other metrics such as CPU usage, such as sys.cpu.combined.percent-normalized, or use the Hot Ranges page.
range.merges	range.merges.count	Number of range merges	This metric indicates how fast a workload is scaling down. Merges are Cockroach's optimization for performance. This metric indicates that there have been deletes in the workload.

SQL

CockroachDB Metric Name	Datadog Integration Metric Name (add cockroachdb. prefix)	Description	Usage
sql.conns	sql.conns	Number of active SQL connections	This metric shows the number of connections as well as the distribution, or balancing, of connections across cluster nodes. An imbalance can lead to nodes becoming overloaded. Review Connection Pooling.
sql.new_conns	sql.new_conns.count	Number of new connection attempts.	The rate of this metric shows how frequently new connections are being established. This can be useful in determining if a high rate of incoming new connections is causing additional load on the server due to a misconfigured application.
sql.txns.open	sql.txns.open	Number of currently open user SQL transactions	This metric should roughly correspond to the number of cores * 4. If this metric is consistently larger, scale out the cluster.
sql.statements.active	sql.statements.active	Number of currently active user SQL statements	This high-level metric reflects workload volume.
sql.failure.count	sql.failure	Number of statements resulting in a planning or runtime error	This metric is a high-level indicator of workload and application degradation with query failures. Use the Insights page to find failed executions with their error code to troubleshoot or use application-level logs, if instrumented, to determine the cause of error.
sql.full.scan.count	sql.full.scan	Number of full table or index scans	This metric is a high-level indicator of potentially suboptimal query plans in the workload that may require index tuning and maintenance. To identify the statements with a full table scan, use SHOW FULL TABLE SCAN or the SQL Activity Statements page with the corresponding metric time frame. The Statements page also includes explain plans and index recommendations. Not all full scans are necessarily bad especially over smaller tables.
sql.insert.count	sql.insert.count	Number of SQL INSERT statements successfully executed	This high-level metric reflects workload volume. Monitor this metric to identify abnormal application behavior or patterns over time. If abnormal patterns emerge, apply the metric's time range to the SQL Activity pages to investigate interesting outliers or patterns. For example, on the Transactions page and the Statements page, sort on the Execution Count column. To find problematic sessions, on the Sessions page, sort on the Transaction Count column. Find the sessions with high transaction counts and trace back to a user or application.
sql.update.count	sql.update.count	Number of SQL UPDATE statements successfully executed	This high-level metric reflects workload volume. Monitor this metric to identify abnormal application behavior or patterns over time. If abnormal patterns emerge, apply the metric's time range to the SQL Activity pages to investigate interesting outliers or patterns. For example, on the Transactions page and the Statements page, sort on the Execution Count column. To find problematic sessions, on the Sessions page, sort on the Transaction Count column. Find the sessions with high transaction counts and trace back to a user or application.
sql.delete.count	sql.delete.count	Number of SQL DELETE statements successfully executed	This high-level metric reflects workload volume. Monitor this metric to identify abnormal application behavior or patterns over time. If abnormal patterns emerge, apply the metric's time range to the SQL Activity pages to investigate interesting outliers or patterns. For example, on the Transactions page and the Statements page, sort on the Execution Count column. To find problematic sessions, on the Sessions page, sort on the Transaction Count column. Find the sessions with high transaction counts and trace back to a user or application.
sql.select.count	sql.select.count	Number of SQL SELECT statements successfully executed	This high-level metric reflects workload volume. Monitor this metric to identify abnormal application behavior or patterns over time. If abnormal patterns emerge, apply the metric's time range to the SQL Activity pages to investigate interesting outliers or patterns. For example, on the Transactions page and the Statements page, sort on the Execution Count column. To find problematic sessions, on the Sessions page, sort on the Transaction Count column. Find the sessions with high transaction counts and trace back to a user or application.
sql.ddl.count	sql.ddl.count	Number of SQL DDL statements successfully executed	This high-level metric reflects workload volume. Monitor this metric to identify abnormal application behavior or patterns over time. If abnormal patterns emerge, apply the metric's time range to the SQL Activity pages to investigate interesting outliers or patterns. For example, on the Transactions page and the Statements page, sort on the Execution Count column. To find problematic sessions, on the Sessions page, sort on the Transaction Count column. Find the sessions with high transaction counts and trace back to a user or application.
sql.txn.begin.count	sql.txn.begin.count	Number of SQL transaction BEGIN statements successfully executed	This metric reflects workload volume by counting explicit transactions. Use this metric to determine whether explicit transactions can be refactored as implicit transactions (individual statements).
sql.txn.commit.count	sql.txn.commit.count	Number of SQL transaction COMMIT statements successfully executed	This metric shows the number of transactions that completed successfully. This metric can be used as a proxy to measure the number of successful explicit transactions.
sql.txn.rollback.count	sql.txn.rollback.count	Number of SQL transaction ROLLBACK statements successfully executed	This metric shows the number of orderly transaction rollbacks. A persistently high number of rollbacks may negatively impact the workload performance and needs to be investigated.
sql.txn.abort.count	sql.txn.abort.count	Number of SQL transaction abort errors	This high-level metric reflects workload performance. A persistently high number of SQL transaction abort errors may negatively impact the workload performance and needs to be investigated.
sql.service.latency-p90, sql.service.latency-p99	sql.service.latency	Latency of SQL request execution	These high-level metrics reflect workload performance. Monitor these metrics to understand latency over time. If abnormal patterns emerge, apply the metric's time range to the SQL Activity pages to investigate interesting outliers or patterns. The Statements page has P90 Latency and P99 latency columns to enable correlation with this metric.
sql.txn.latency-p90, sql.txn.latency-p99	sql.txn.latency	Latency of SQL transactions	These high-level metrics provide a latency histogram of all executed SQL transactions. These metrics provide an overview of the current SQL workload.
txnwaitqueue.deadlocks_total	txnwaitqueue.deadlocks.count	Number of deadlocks detected by the transaction wait queue	Alert on this metric if its value is greater than zero, especially if transaction throughput is lower than expected. Applications should be able to detect and recover from deadlock errors. However, transaction performance and throughput can be maximized if the application logic avoids deadlock conditions in the first place, for example, by keeping transactions as short as possible.
sql.distsql.contended_queries.count	sql.distsql.contended.queries	Number of SQL queries that experienced contention	This metric is incremented whenever there is a non-trivial amount of contention experienced by a statement whether read-write or write-write conflicts. Monitor this metric to correlate possible workload performance issues to contention conflicts.
sql.conn.latency-p90, sql.conn.latency-p99	sql.conn.latency	Latency to establish and authenticate a SQL connection	These metrics characterize the database connection latency which can affect the application performance, for example, by having slow startup times.
txn.restarts.serializable	txn.restarts.serializable	Number of restarts due to a forwarded commit timestamp and isolation=SERIALIZABLE	This metric is one measure of the impact of contention conflicts on workload performance. For guidance on contention conflicts, review transaction contention best practices and performance tuning recipes. Tens of restarts per minute may be a high value, a signal of an elevated degree of contention in the workload, which should be investigated.
txn.restarts.writetooold	txn.restarts.writetooold	Number of restarts due to a concurrent writer committing first	This metric is one measure of the impact of contention conflicts on workload performance. For guidance on contention conflicts, review transaction contention best practices and performance tuning recipes. Tens of restarts per minute may be a high value, a signal of an elevated degree of contention in the workload, which should be investigated.
txn.restarts.writetoooldmulti	txn.restarts.writetoooldmulti.count	Number of restarts due to multiple concurrent writers committing first	This metric is one measure of the impact of contention conflicts on workload performance. For guidance on contention conflicts, review transaction contention best practices and performance tuning recipes. Tens of restarts per minute may be a high value, a signal of an elevated degree of contention in the workload, which should be investigated.
txn.restarts.unknown	txn.restarts.unknown.count	Number of restarts due to a unknown reasons	This metric is one measure of the impact of contention conflicts on workload performance. For guidance on contention conflicts, review transaction contention best practices and performance tuning recipes. Tens of restarts per minute may be a high value, a signal of an elevated degree of contention in the workload, which should be investigated.
txn.restarts.txnpush	txn.restarts.txnpush.count	Number of restarts due to a transaction push failure	This metric is one measure of the impact of contention conflicts on workload performance. For guidance on contention conflicts, review transaction contention best practices and performance tuning recipes. Tens of restarts per minute may be a high value, a signal of an elevated degree of contention in the workload, which should be investigated.
txn.restarts.txnaborted	txn.restarts.txnaborted.count	Number of restarts due to an abort by a concurrent transaction	The errors tracked by this metric are generally due to deadlocks. Deadlocks can often be prevented with a considered transaction design. Identify the conflicting transactions involved in the deadlocks, then, if possible, redesign the business logic implementation prone to deadlocks.

Table Statistics

CockroachDB Metric Name	Datadog Integration Metric Name (add cockroachdb. prefix)	Description	Usage
jobs.auto_create_stats.resume_failed	jobs.auto.create.stats.resume_failed.count	Number of auto_create_stats jobs which failed with a non-retryable error	This metric is a high-level indicator that automatically generated table statistics is failing. Failed statistic creation can lead to the query optimizer running with stale statistics. Stale statistics can cause suboptimal query plans to be selected leading to poor query performance.
jobs.auto_create_stats.currently_running	jobs.auto.create.stats.currently_running	Number of auto_create_stats jobs currently running	This metric tracks the number of active automatically generated statistics jobs that could also be consuming resources. Ensure that foreground SQL traffic is not impacted by correlating this metric with SQL latency and query volume metrics.
jobs.auto_create_stats.currently_paused	jobs.auto.create.stats.currently_paused	Number of auto_create_stats jobs currently considered Paused	This metric is a high-level indicator that automatically generated statistics jobs are paused which can lead to the query optimizer running with stale statistics. Stale statistics can cause suboptimal query plans to be selected leading to poor query performance.
jobs.create_stats.currently_running	jobs.create.stats.currently_running	Number of create_stats jobs currently running	This metric tracks the number of active create statistics jobs that may be consuming resources. Ensure that foreground SQL traffic is not impacted by correlating this metric with SQL latency and query volume metrics.

Backup and Restore

CockroachDB Metric Name	Datadog Integration Metric Name (add cockroachdb. prefix)	Description	Usage
jobs.backup.currently_running	jobs.backup.currently_running	Number of backup jobs currently running	See Description.
jobs.backup.currently_paused	jobs.backup.currently_paused	Number of backup jobs currently considered Paused	Monitor and alert on this metric to safeguard against an inadvertent operational error of leaving a backup job in a paused state for an extended period of time. In functional areas, a paused job can hold resources or have concurrency impact or some other negative consequence. Paused backup may break the recovery point objective (RPO).
schedules.BACKUP.failed	schedules.backup.failed	Number of BACKUP jobs failed	Monitor this metric and investigate backup job failures.
schedules.BACKUP.last-completed-time	schedules.backup.last_completed_time	The unix timestamp of the most recently completed backup by a schedule specified as maintaining this metric	Monitor this metric to know that backups are meeting the recovery point objective (RPO).

Changefeeds

If changefeeds are created in a CockroachDB cluster, monitor these additional metrics in your custom dashboards:

CockroachDB Metric Name	Datadog Integration Metric Name (add cockroachdb. prefix)	Description	Usage
changefeed.running	changefeed.running	Number of currently running changefeeds, including sinkless	This metric tracks the total number of all running changefeeds.
jobs.changefeed.currently_paused	jobs.changefeed.currently_paused	Number of changefeed jobs currently considered Paused	Monitor and alert on this metric to safeguard against an inadvertent operational error of leaving a changefeed job in a paused state for an extended period of time. Changefeed jobs should not be paused for a long time because the protected timestamp prevents garbage collection.
changefeed.failures	changefeed.failures	Total number of changefeed jobs which have failed	This metric tracks the permanent changefeed job failures that the jobs system will not try to restart. Any increase in this counter should be investigated. An alert on this metric is recommended.
changefeed.error_retries	changefeed.error.retries	Total retryable errors encountered by all changefeeds	This metric tracks transient changefeed errors. Alert on "too many" errors, such as 50 retries in 15 minutes. For example, during a rolling upgrade this counter will increase because the changefeed jobs will restart following node restarts. There is an exponential backoff, up to 10 minutes. But if there is no rolling upgrade in process or other cluster maintenance, and the error rate is high, investigate the changefeed job.
changefeed.emitted_messages	changefeed.emitted.messages	Messages emitted by all feeds	This metric provides a useful context when assessing the state of changefeeds. This metric characterizes the rate of changes being streamed from the CockroachDB cluster.
changefeed.emitted_bytes	changefeed.emitted_bytes.count	Bytes emitted by all feeds	This metric provides a useful context when assessing the state of changefeeds. This metric characterizes the throughput bytes being streamed from the CockroachDB cluster.
changefeed.commit_latency	changefeed.commit.latency	The difference between the event MVCC timestamp and the time it was acknowledged by the downstream sink. If the sink batches events, then the difference between the oldest event in the batch and acknowledgement is recorded. Latency during backfill is excluded.	This metric provides a useful context when assessing the state of changefeeds. This metric characterizes the end-to-end lag between a committed change and that change applied at the destination.
jobs.changefeed.protected_age_sec	jobs.changefeed.protected_age_sec	The age of the oldest PTS record protected by changefeed jobs	Changefeeds use protected timestamps to protect the data from being garbage collected. Ensure the protected timestamp age does not significantly exceed the GC TTL zone configuration. Alert on this metric if the protected timestamp age is greater than 3 times the GC TTL.

Row-Level TTL

If Row-Level TTL is configured for any table in a CockroachDB cluster, monitor these additional metrics in your custom dashboards:

CockroachDB Metric Name	Datadog Integration Metric Name (add cockroachdb. prefix)	Description	Usage
jobs.row_level_ttl.resume_completed	jobs.row.level.ttl.resume_completed.count	Number of row_level_ttl jobs which successfully resumed to completion	If Row Level TTL is enabled, this metric should be nonzero and correspond to the ttl_cron setting that was chosen. If this metric is zero, it means the job is not running
jobs.row_level_ttl.resume_failed	jobs.row.level.ttl.resume_failed.count	Number of row_level_ttl jobs which failed with a non-retryable error	This metric should remain at zero. Repeated errors means the Row Level TTL job is not deleting data.
jobs.row_level_ttl.rows_selected	jobs.row.level.ttl.rows_selected.count	Number of rows selected for deletion by the row level TTL job.	Correlate this metric with the metric jobs.row_level_ttl.rows_deleted to ensure all the rows that should be deleted are actually getting deleted.
jobs.row_level_ttl.rows_deleted	jobs.row.level.ttl.rows_deleted.count	Number of rows deleted by the row level TTL job.	Correlate this metric with the metric jobs.row_level_ttl.rows_selected to ensure all the rows that should be deleted are actually getting deleted.
jobs.row_level_ttl.currently_paused	jobs.row.level.ttl.currently_paused	Number of row_level_ttl jobs currently considered Paused	Monitor this metric to ensure the Row Level TTL job does not remain paused inadvertently for an extended period.
jobs.row_level_ttl.currently_running	jobs.row.level.ttl.currently_running	Number of row_level_ttl jobs currently running	Monitor this metric to ensure there are not too many Row Level TTL jobs running at the same time. Generally, this metric should be in the low single digits.
schedules.scheduled-row-level-ttl-executor.failed	schedules.scheduled.row.level.ttl.executor_failed.count	Number of scheduled-row-level-ttl-executor jobs failed	Monitor this metric to ensure the Row Level TTL job is running. If it is non-zero, it means the job could not be created.
jobs.row_level_ttl.span_total_duration	NOT AVAILABLE	Duration for processing a span during row level TTL.	See Description.
jobs.row_level_ttl.select_duration	NOT AVAILABLE	Duration for select requests during row level TTL.	See Description.
jobs.row_level_ttl.delete_duration	NOT AVAILABLE	Duration for delete requests during row level TTL.	See Description.
jobs.row_level_ttl.num_active_spans	NOT AVAILABLE	Number of active spans the TTL job is deleting from.	See Description.
jobs.row_level_ttl.total_rows	NOT AVAILABLE	Approximate number of rows on the TTL table.	See Description.
jobs.row_level_ttl.total_expired_rows	NOT AVAILABLE	Approximate number of rows that have expired the TTL on the TTL table.	See Description.

See also

• Available Metrics
• Monitor CockroachDB with Prometheus
• Visualize metrics in Grafana
• Custom Chart Debug Page
• Cluster API
• Events to alert on
• CockroachDB Source Code - DB Console metrics to graphs mappings (in *.tsx files)