Replication

HardhatDB replicates with Raft. There is one leader. Followers apply the same log and can serve reads that are behind the commit index.

Writes

A write on the leader commits through Raft, then returns. A write on a follower is forwarded to the current leader. An explicit transaction is forwarded as a whole. A read on a node waits until that node has applied its commit index, so another connection on the same node sees the write. A different node can still be behind by one Raft heartbeat. HARDHATDB_READ_FROM=leader forwards ordinary reads to the leader instead.

Row and table locks exist only on the process that executes the statement. SELECT ... FOR UPDATE and a SERIALIZABLE read run on the leader, so they meet the locks taken by writes. BEGIN sends the session's transaction_isolation with the transaction. A follower does not lock rows for those statements.

SHOW RAFT STATUS returns one row: role, leader, commit_index, applied_index, lag, and suffrage. On a follower the statement runs locally, so lag is how far that node is behind the commit index. suffrage is voter, nonvoter, or staging, and it is empty on a standalone process.

Binlog

Every node writes a binlog beside the Raft directory. HARDHATDB_SERVER_UUID is shared by the group and is the GTID server id. HARDHATDB_BINLOG_MAX_SIZE rolls the binlog after a transaction crosses that many bytes. The default is 1 GiB.

FLUSH BINARY LOGS on the leader rolls every node's binlog together. Any node can stream that binlog. After a promotion the GTID stream is the same.

Upstream MySQL

Set the same source on every node and the current primary replicates from it. A follower does not open the upstream connection. After a failover the new primary continues from the GTID stored with the applied rows.

CHANGE REPLICATION SOURCE TO
  SOURCE_HOST = 'mysql.example',
  SOURCE_PORT = 3306,
  SOURCE_USER = 'repl',
  SOURCE_PASSWORD = 'secret',
  SOURCE_AUTO_POSITION = 1;
START REPLICA;

SOURCE_AUTO_POSITION cannot be turned off. STOP REPLICA stops the job. The password is stored in a local file beside the Raft directory, so a promoted node can dial only if it already has that password. REPLICATE_REWRITE_DB is rejected.

The same values can be set in the environment before the first start: HARDHATDB_SOURCE_HOST, HARDHATDB_SOURCE_PORT, HARDHATDB_SOURCE_USER, and HARDHATDB_SOURCE_PASSWORD.

Moving an existing MySQL database, including when a dump is the right path, is in Migration.

Membership

Membership changes are leader statements. A follower forwards them.

RAFT ADD VOTER 'n4' '10.116.0.5:7001';
RAFT ADD NONVOTER 'n4' '10.116.0.5:7001';
RAFT REMOVE SERVER 'n4';

Start the new process with the same peers and certificate names, then add it. A nonvoter copies the log and serves MySQL reads. It does not vote, so quorum stays the voters. Clients add that MySQL address for reads. A write sent there is still forwarded to the leader. A read on another connection can lag. RAFT REMOVE SERVER drops a voter or a nonvoter.

Writes still commit on the leader. A faster machine and volume for the voters raises that ceiling. A table whose writes do not fit that leader is placed with SHARD TABLE on a second set of peers. A size-based split stays on the current hosts and does not add a machine.

HARDHATDB_RAFT_BOOTSTRAP=1 bootstraps only when the Raft directory has no state and every other peer reports that it has none either. A later start with the flag still set logs that bootstrap is ignored, then catches up as a follower. A wiped volume with the flag left on refuses to start when a peer already has the group, unless HARDHATDB_RAFT_BOOTSTRAP_FORCE=1.

Architecture Migration