Prometheus Metrics

Set HARDHATDB_METRICS_ADDR to an host:port. The process serves three routes there. Leave the variable unset and the port stays closed.

export HARDHATDB_METRICS_ADDR=127.0.0.1:9090
curl http://127.0.0.1:9090/healthz
curl http://127.0.0.1:9090/readyz
curl http://127.0.0.1:9090/metrics

/healthz returns 200 once this port is serving. /readyz returns 200 when a standalone store is open. On a cluster node it returns 200 when this node knows a leader and has applied the commit index, and 503 during an election or while it is behind. hardhatdb-proxy (opens in a new tab) polls /readyz and sends a new client connection to a node that returns 200.

Series

The exposition format is Prometheus text. Counts have no labels. The role series has one label, and hardhatdb_query_seconds has a kind label.

Metric	Type	Labels	Description
`hardhatdb_queries_total`	Counter	—	Queries accepted by the server
`hardhatdb_query_errors_total`	Counter	—	Queries that returned an error
`hardhatdb_raft_commit_index`	Gauge	—	Raft commit index on this node
`hardhatdb_raft_applied_index`	Gauge	—	Raft index applied on this node
`hardhatdb_raft_lag`	Gauge	—	Commit index minus applied index
`hardhatdb_raft_role`	Gauge	`role`	`1` for this node's role (`leader`, `follower`, `candidate`, or `standalone`)
`hardhatdb_connections`	Gauge	—	Sessions admitted on this process
`hardhatdb_apply_wait_timeouts_total`	Counter	—	Local reads that gave up waiting for apply
`hardhatdb_replica_seconds_behind`	Gauge	—	Upstream SQL lag. `-1` when the SQL thread is not running
`hardhatdb_has_leader`	Gauge	—	`1` when this node knows a leader. A standalone process is `1`
`hardhatdb_commit_seconds`	Histogram	—	Time a leader waits for a recorded batch to apply
`hardhatdb_record_seconds`	Histogram	—	Time a leader spends building a batch before it is queued
`hardhatdb_apply_seconds`	Histogram	—	Time a committed batch takes to land in Badger
`hardhatdb_forward_seconds`	Histogram	—	Time a follower waits for the leader RPC
`hardhatdb_lock_wait_seconds`	Histogram	—	Time a session blocks on a row or table lock
`hardhatdb_query_seconds`	Histogram	`kind`	Time in the server handler. `kind` is `read`, `write`, `ddl`, or `admin`
`hardhatdb_lsm_bytes`	Gauge	—	On-disk LSM size of this node's Badger directory
`hardhatdb_txn_decisions_open`	Gauge	—	Cross-range decisions the meta leader has not finished. `-1` means recovery has not read the list. `0` means a new cross-range write may start
`hardhatdb_split_peers_remaining`	Gauge	—	Spare peers the size watcher can still place on new machines. `0` means the next split stays on the current hosts

Each histogram has _bucket, _sum, and _count series. The buckets are 1ms, 5ms, 10ms, 25ms, 50ms, 100ms, 250ms, 1s, and 5s.

A standalone process still exports the Raft series. The role is standalone and the indexes stay at zero.

Scrape configuration

scrape_configs:
  - job_name: hardhatdb
    static_configs:
      - targets: ['10.116.0.2:9090', '10.116.0.3:9090', '10.116.0.4:9090']
    scrape_interval: 15s

Bind HARDHATDB_METRICS_ADDR on an address the scraper can reach. The MySQL and Raft ports are separate.

Example queries

# Nodes that are behind the commit index
hardhatdb_raft_lag > 0

# Query error ratio
rate(hardhatdb_query_errors_total[5m]) / rate(hardhatdb_queries_total[5m])

# Which node is leader
hardhatdb_raft_role{role="leader"} == 1

# Recording time versus quorum wait
rate(hardhatdb_record_seconds_sum[5m]) / rate(hardhatdb_record_seconds_count[5m])
rate(hardhatdb_commit_seconds_sum[5m]) / rate(hardhatdb_commit_seconds_count[5m])

Operations Reference