1# @ajax/mesh-peer-relay
2
3The fleet thread: N machines run as one fleet with partial membership.
4
5- **Links.** A full mesh of TCP links (`@ajax/mesh-peer-link`). Every member beats every other each
6  `ping_ms`. A member is up while its link is up and it was heard within `dead_ms`; it is suspect after
7  a third of that. Nothing ends when a member drops: only a lockstep group that contains it pauses.
8- **The leader and the desired state** (`@ajax/mesh-fleet-core`).
9  - A change asked for on any member (`mpr_change`) goes to the leader as F_CTL, and is answered once
10    the reachable members have reconciled it. Without a leader the change is refused.
11  - The leader commits the change and waits for the members. If the change failed everywhere, the
12    leader reverts it.
13  - Each committed document goes to the daemon's reconciler (`committed`).
14  - Images and migrated views are fetched from members for the reconciler (`mpr_fetch_image`,
15    `mpr_fetch_view`).
16- **The leader's policies.**
17  - Lockstep groups: pause, resume, and re-form after `auto_ms`.
18  - The scheduler (`@ajax/mesh-sched-*`): placement passes, rebalancing, and owner-busy members.
19  - Migrations, with the view moved in 1 MiB F_VIEW chunks.
20  - All of it works from a model of every member, built from F_JOIN, F_LOADREP and F_PERF (the perf
21    board).
22- **Lockstep groups.** Members send F_HELLO for each epoch and cycle. Rounds are mirrored point to point
23  by a sender thread that writes under each link's writer lock.
24- **Published.**
25  - The status block's fleet, member and scheduler regions.
26  - The `fleet`, `groups` and `sched` texts, the dry run, and the fleet view's records in views.rs's
27    text form (`kind k=v ... -- text`).
28  - The jobs dispatcher's scheduling model.
29
30Everything daemon-specific comes through `mpr_host`: the status block, the mailbox, the reconciler,
31the driver, the catalog, and jobs and blobs. This package holds no daemon state.
32
33## Using it
34
35```c
36mpr_host h = {.ctx = d, .host_u32 = ..., .committed = ..., .recon = ..., /* every callback */};
37mcc_topology t;
38mcc_topology_parse(config_text, me, &t, &err);
39mpr_relay *r = mpr_start(&t, &caps, "/var/lib/meshd/fleet-state", &h, &err);
40mpr_change(r, "sched cordon 2", (mpr_reply){answered, ctx});
41...
42mpr_stop(r);
43```
44
45Callbacks must be thread-safe. The fleet thread calls most of them, but not all:
46
47- the link reader threads call `place_frame`, `jobs_frame`, `blobs_frame`, F_LOADREP's `host_bytes`,
48  and a lockstep round's `mb_write`, `mb_publish64` and `round_stats`, as well as `uptime_ms`;
49- the rounds sender thread reads the mailbox;
50- a migration's source runs `stop_and_view` on its own thread.
51
52## Ported from
53
54[alpha-rust-mesh](https://github.com/rsafier/alpha-rust-mesh), `crates/mesh-host/src/`:
55
56- `peer.rs`: `run`, `Relay`, the events, links, beats, changes, the leader, policies, images, the
57  lockstep rounds, `publish`, `fleet_text`, `groups_text`, `set_member` and `group_line`.
58- `peer_sched.rs`: the member model, `perf_map`, `sched_pass`, `schedule`, migrations, the view
59  transfer, `sched_publish` and `fleet_recs`.
60- `shared.rs`: `PerfBoard`.
61
62The frame codecs live in `@ajax/mesh-peer-codec` and `@ajax/mesh-place-wire`, and their Rust tests are
63ported there.
64
65## Tests
66
67`tests/loopback.c` runs three relays over real loopback TCP, each with a fake daemon (status block,
68mailbox, and a reconciler that applies each document at once):
69
70- they elect one leader;
71- a change asked for on a follower is forwarded, committed, reconciled everywhere and answered;
72- a refused change carries its reason;
73- a stopped member is seen gone (fleet core, status block, `fleet` text);
74- the majority left keeps committing;
75- the member comes back and catches up;
76- when the leader stops, the two left elect another;
77- with one member left there is no leader, and changes are refused.
78
79## Licence
80
81MIT. See the repository's LICENSE.