Sharding

Range placement is alpha.

A ranged cluster keeps the catalog on a meta group and places each table's rows on data groups by key range. A commit waits for the quorum of the group that owns the row.

Meta group

HARDHATDB_META_ADDR turns the meta catalog on. It is the Raft address of this node's meta group. The HARDHATDB_RAFT_* variables are the data group this process stores.

Variable	Role
`HARDHATDB_META_ADDR`	Raft address of this node's meta group
`HARDHATDB_META_PEERS`	Meta voters, `id=host:port`, comma-separated
`HARDHATDB_META_NONVOTERS`	Meta replicas that copy the catalog and do not vote
`HARDHATDB_META_BOOTSTRAP`	`1` on one meta voter, the first time the group starts
`HARDHATDB_META_DATA`	Badger directory for the catalog. Default `data/meta`
`HARDHATDB_META_DIR`	Raft directory for the catalog. Default `data/meta-raft`
`HARDHATDB_META_FORWARD`	Meta write-forward address. Required when the shard forward port would otherwise collide with it
`HARDHATDB_SHARD_FORWARD`	`index=host:port` for one member of each data group. Schema statements are sent to every address. The index is a label

Placing a range

SHARD TABLE records a key range in meta. START and END are exclusive integer bounds. CHECK names an extra column that must travel with the shard key. GROUP and PEER name a data group that already exists. Without GROUP, the range covers this process's data group.

SHARD TABLE app.accounts BY id CHECK email
  RANGE END 2500
  GROUP 33333333-3333-3333-3333-333333333331
  PEER r1 10.119.0.2:7101 10.119.0.2:7102
  PEER r2 10.119.0.3:7101 10.119.0.3:7102
  PEER r3 10.119.0.4:7101 10.119.0.4:7102;
 
SHARD TABLE app.accounts BY id CHECK email
  RANGE START 2500
  GROUP 44444444-4444-4444-4444-444444444441
  PEER r4 10.119.0.5:7101 10.119.0.5:7102
  PEER r5 10.119.0.6:7101 10.119.0.6:7102
  PEER r6 10.119.0.7:7101 10.119.0.7:7102;

Each PEER is id, Raft address, then write-forward address.

A sharded insert includes the shard column. A write whose keys sit on one range runs there. A write whose keys sit on several ranges is split into one statement per range and committed with prepare, a meta decision, then commit. A write that does not name the shard column is refused. A read that does not name the shard column is sent to every range that holds the table. Joins still scatter and merge.

A size-based split stays on the current hosts and does not add a machine, unless HARDHATDB_SPLIT_PEERS names spare processes. Each entry is id=raft=forward, comma-separated. The size watcher uses that list for one split onto those machines, then keeps running. Later splits stay on the current hosts. RANGE ADD VOTER still adds a voter to a group that already exists.

SPLIT RANGE app.accounts AT 2500
  PEER r4 10.119.0.5:7101 10.119.0.5:7102
  PEER r5 10.119.0.6:7101 10.119.0.6:7102
  PEER r6 10.119.0.7:7101 10.119.0.7:7102;

Those processes are already running. The range leader sends OPEN GROUP to each forward address, copies the rows above the split key, and publishes the right range. A table whose writes do not fit its leader is placed here, with PEER naming the other machines.

Migration Compatibility