1# @ajax/mesh-jobd
2
3meshd's jobs thread and blob thread, running over the daemon's shared state (`@ajax/mesh-shared`).
4
5- **The jobs thread** runs this member's job node (`@ajax/mesh-jobs-node`) over four things: the
6  peer links (F_JOB_* frames, F_LEASE and F_RESULT), the driver (`MESH_CMD_JOB_LEASE`, `JOB_CUT` and
7  `JOB_PROGS`, and `MESH_JEV_DONE` coming back), the job log (`@ajax/mesh-jobd-store`) and the blob
8  thread.
9  - The node's outputs go through `mj_durable`, so the log is fsynced before any send or answer that
10    depends on it. If the log cannot be made durable, the member fail-stops (`mesh_shared_fail_stop`).
11  - Table frames (F_JOB_STATE, F_JOB_HAVE) wait up to 2 s for room in a link's queue. Once one has
12    waited that long, the rest of the batch to that member is dropped. Other frames are dropped when
13    the queue is full.
14  - The job programs are asked of the driver every 250 ms and the answer is never waited for. While
15    blobd holds more than `MESH_BLOBD_BLOB_MEM_MAX` of this member's payloads in memory, the member
16    reports no programs, so it takes no new leases (backpressure).
17  - Item payloads are hashed here and handed to blobd before their results go to the node.
18  - Every 250 ms it publishes three things: the host block's `S_JOB_*` fields, the jobs view's records
19    in `sh->jobs_view_recs`, and blobd's news.
20  - It compacts the log when the log asks for it.
21- **The control requests.** `jobs`, `job status` and `job results` are answered from this member's
22  committed table. Changes (`job submit|cancel|forget|tone`) go to the node when this member leads,
23  or when it is a single meshd. Otherwise they are forwarded to the fleet thread (`MESH_PEV_CHANGE`).
24  A remote request (F_CTL) is answered with an F_CTL_REPLY frame.
25- **The blob thread** runs `@ajax/mesh-blobd` on `sh->blobs`. It has three writer threads, and each
26  writer makes a batch of references durable with one directory fsync per batch.
27
28## Wiring
29
30```c
31mjd_config jc = {a.jobs_state, a.blobs, (uint64_t)a.blobs_cap_mb << 20, dead_ms /* MESH_PEER_DEAD_MS single */};
32mjd *jd = mjd_open(sh, &jc, &err);   /* after mesh_shared_new: replays the log, indexes the store */
33if (!jd) fail(err);
34mesh_control_render = mjd_render;    /* control's and http's `render` */
35...
36mjd_start(jd);                       /* instead of mesh_chan_close_rx(sh->jobs) */
37```
38
39The fleet thread (`@ajax/mesh-peer-relay`'s `mpr_host`) reaches the jobs area through one-line
40wrappers:
41
42- `jobs_view` → `mjd_set_view(sh, (mj_view){term, has_leader, leader, alive, linked})`.
43- `jobs_sched` → `mjd_jobs_sched(sh, members, n, doc, perf, w)`. It copies all of them; each
44  program's needs come from the committed document, as in peer_sched.rs.
45- `jobs_frame` → `mjd_jobs_frame(sh, from, type, body, n)`.
46- `blobs_frame` → `mjd_blobs_frame(sh, from, type, body, n)`.
47- `jobs_ctl` → a `MESH_JEV_CTL` event on `sh->jobs`. For a local request, `reply_to` is LOCAL with a
48  `mesh_reply` whose answer goes to the `mpr_reply`. For a request from member `from`, `reply_to` is
49  REMOTE with `remote_from = from` and `remote_id = id`; the jobs thread then sends the F_CTL_REPLY
50  frame itself.
51
52`mjd_route_frame(sh, from, frame, n)` takes a whole frame (type, length, body) instead.
53
54## Ported from
55
56[alpha-rust-mesh](https://github.com/rsafier/alpha-rust-mesh) (Richard Safier, MIT):
57
58- `crates/mesh-host/src/jobd.rs`: `run`, `Jobd` (`apply`, `answer`, `ctl`, `publish`, `news`,
59  `jobs_recs`).
60- `crates/mesh-host/src/blobd.rs`: `run`, `writer`, `render`.
61- `crates/mesh-host/src/shared.rs`: `jobs_view`, `jobs_sched`, `render_watch`. In C these live in
62  `sh->jobs_ext`.
63
64## Tests
65
66`tests/jobd.c` runs both threads in-process over a `mesh_shared` with a fake driver:
67
68- **Single meshd.** A job with payloads runs to the end. The test checks its leases and results,
69  that its payloads are durable in the blob store, the commit, the host block and the jobs view.
70- **Control.** The control answers and their errors, and `render` errors.
71- **Frame routing.**
72- **Backpressure** on payload memory.
73- **Restart.** The job log is replayed (a forgotten job stays forgotten) and the blob store is
74  indexed.
75- **Follower.** A follower in a fleet of 3 forwards a change to the fleet thread and answers a remote
76  request with F_CTL_REPLY.
77
78## Licence
79
80MIT. See the repository's LICENSE.