Operations

Shutdown

SIGTERM and SIGINT stop the MySQL listener, wait up to HARDHATDB_SHUTDOWN_TIMEOUT (default 15s) for sessions to finish, then shut Raft down and sync Badger.

A committed Raft entry that cannot be written to Badger, or queued to the binlog, logs the node id and the Raft index and exits. The process does not advance past that index. The next start replays the log from the last index that reached disk.

Accounts

The first boot creates one caching_sha2_password superuser. HARDHATDB_BOOTSTRAP_PASSWORD is required then and is not saved for later boots. The account lives in the data directory. The default user is root and the default host is %, so a published Docker port and other containers can connect. That is what a private Docker network needs. Do not use that account on an address other hosts can reach. CREATE USER and GRANT add further accounts, and those accounts replicate with the group.

A follower that has not received the accounts yet rejects every login.

TLS

Leave HARDHATDB_TLS_CERT and HARDHATDB_TLS_KEY unset and MySQL stays on plaintext. Set both and the listener requires TLS.

Raft is stricter. A non-loopback HARDHATDB_RAFT_ADDR requires HARDHATDB_RAFT_TLS_CERT, HARDHATDB_RAFT_TLS_KEY, and HARDHATDB_RAFT_TLS_CA. The certificate needs serverAuth and clientAuth. Peers present it and verify it with that CA. The same files can be the MySQL certificate.

Create a CA and a server certificate that names every address clients and peers dial. For the Compose cluster that is 127.0.0.1, the node names, and 10.116.0.2, 10.116.0.3, 10.116.0.4. The Demo has the commands. A DigitalOcean deployment uses the same plan with the VPC addresses; see droplets/README.md in the repository.

Membership

SHOW RAFT STATUS;
RAFT ADD VOTER 'n4' '10.116.0.5:7001';
RAFT ADD NONVOTER 'n4' '10.116.0.5:7001';
RAFT REMOVE SERVER 'n4';

SHOW RAFT STATUS returns one row: role, leader, commit_index, applied_index, lag, and suffrage. A standalone process reports standalone and an empty suffrage. On a follower the statement runs locally, so lag is how far that node is behind the commit index. suffrage is voter, nonvoter, or staging.

A nonvoter copies the log and serves reads. It does not vote, so quorum stays the voters. A write sent to it is forwarded to the leader. A read on this node waits until the commit index is applied. A different node can still be behind by one Raft heartbeat. HARDHATDB_READ_FROM=leader forwards ordinary reads to the leader. RAFT REMOVE SERVER drops a voter or a nonvoter. A follower forwards the membership statements to the leader.

Writes still commit on the leader. A faster machine for the voters raises that ceiling. A table whose writes do not fit that leader is placed with SHARD TABLE on a second set of peers. A size-based split stays on the current hosts and does not add a machine.

HARDHATDB_RAFT_BOOTSTRAP=1 bootstraps only when the Raft directory has no state and every other peer answers that it has none either. A peer that already has a group, or one that does not answer, refuses the new group and the process exits. HARDHATDB_RAFT_BOOTSTRAP_FORCE=1 is the override that creates a group anyway. The meta group uses HARDHATDB_META_BOOTSTRAP_FORCE the same way. A restart that still has Raft state ignores the bootstrap flag, logs that bootstrap is ignored, and catches up as a follower instead of waiting to become leader. Restored data is the exception to the empty-directory rule: an empty Raft directory and a data directory that already has a Raft index start the group from that index. See Backup.

A volume created with an older peer list still has that list. Remove it before the first start on a new address plan: docker compose down -v.

Listener limits

HARDHATDB_MAX_CONNECTIONS (default 151), HARDHATDB_NET_READ_TIMEOUT, HARDHATDB_NET_WRITE_TIMEOUT, and HARDHATDB_MAX_EXECUTION_TIME are the boot values. SET GLOBAL for those four names updates this process. A new session is refused once the process list reaches max_connections. A connection keeps the socket timeout it was accepted with. max_execution_time is a per-statement deadline in milliseconds. 0 sets no deadline. A bare net timeout is seconds. Unset means no socket deadline.

Backups

BACKUP TO '/var/backups/1' writes that directory on the leader, or on a standalone process. It holds state.bin, meta, and, in a cluster, binlog/. The directory is refused when it already holds files. With the server stopped, hardhatdb restore --from /var/backups/1 --data /data/hardhatdb loads state.bin into an empty data directory and refuses one that already exists. Start that process with a fresh Raft directory. On the new leader, RESTORE BINLOG FROM '/var/backups/1/binlog' AFTER n applies binlog events whose sequence is above the index in meta. The full procedure is in SQL Reference.

Each process backs up the directory it stores. A sharded process also accepts SHOW META STATUS, BACKUP META TO, RESTORE META BINLOG FROM ... AFTER n, META ADD VOTER, and META ADD NONVOTER. Status is local. The others run on the meta catalog and are forwarded to its leader.

BACKUP TO writes one directory when you ask. Set HARDHATDB_BACKUP_INTERVAL and HARDHATDB_BACKUP_DIR and the leader writes a new subdirectory on that interval. A follower does nothing. HARDHATDB_BACKUP_KEEP is how many of those directories to retain. The default is 7. The interval is a Go duration or a number of seconds.

Data directory

A directory opens when its format matches this binary. The first open of a directory from this layout stamps format 1. A later layout bump refuses the old directory. There is no in-place rewrite. Moving to a new layout is a dump and load.

Health

Set HARDHATDB_METRICS_ADDR (for example 127.0.0.1:9090) to serve GET /healthz, GET /readyz, and GET /metrics. Leave it unset and that port stays closed. /healthz returns 200 once the process is serving that port. /readyz returns 200 when a standalone store is open, and on a cluster node only when it knows a leader and has applied the commit index. During an election it returns 503. The series are listed in Prometheus Metrics.