A member's blob store: job payloads content-addressed by sha256 and referenced per job, bounded, evicting only forgotten jobs' references, verified on read
$ sexc add @ajax/mesh-blobs-store- total 3
- drwxr-xr-x-../
- drwxr-xr-x-Oct 27 05:23v/# 1 version
- drwxr-xr-x-Oct 27 05:23files/# the files of 0.1.0
- latest
- 0.1.0, published Oct 27 05:23 by ajax
- owners
- ajax
- license
- MIT
- dependencies
- @ajax/mesh-check ^0.1.0, @ajax/mesh-strbuf ^0.1.0, @ajax/mesh-omap ^0.1.0, @ajax/mesh-jobs-id ^0.1.0, @ajax/mesh-ref-sha256d ^0.1.0, @ajax/mesh-blobs-durable ^0.1.0
- dependents
- @ajax/mesh-blobd, @ajax/mesh-jobd
- keywords
- mesh alphaos blobs content-addressed storage
- homepage
- https://github.com/thomasdavis/alpha-c-mesh/tree/main/packages/mesh-blobs-store
- repository
- https://github.com/thomasdavis/alpha-c-mesh
1# @ajax/mesh-blobs-store 2 3A member's blob store: job items' payloads, kept off the job log, content-addressed by their sha256 4and **referenced per job**. 5 6- **Layout.** A job's blob is the file `<dir>/<job>/<sha256 hex>`, either whole under its name or 7 absent (`@ajax/mesh-blobs-durable` writes it). Every read checks the bytes against the name. A 8 corrupt or missing file is removed, reads as absent, and is reported lost. 9- **One blob, several jobs.** Two jobs that render the same scene compute byte-identical payloads. 10 Each job holds its own reference, a hard link of the file another job's reference names, so the 11 bytes are on disk once. Forgetting or evicting one job removes only its references. A blob goes with 12 its last reference. (In the live fleet, forgetting job 1.2 once took job 2.1's committed blobs with 13 it; this layout prevents that.) 14- **Bounded.** At most `cap` bytes, each blob counted once. Room is made only by evicting *forgotten* 15 jobs' references. A store full of listed jobs' blobs refuses: `job forget` a finished job to make 16 room. 17- **In memory.** Without a directory, blobs are kept in memory under the same cap. 18 19## Ported from 20 21[alpha-rust-mesh](https://github.com/rsafier/alpha-rust-mesh) (Richard Safier, MIT): 22`crates/mesh-host/src/blobs.rs`: `Live`, `Make`, `Reserved`, `Store` (`open`, `reserve`, `written`, 23`put_mem`, `get`, `take_lost`, `drop_job`, `jobs`, `all`, the counts). 24 25## Differences 26 27The Rust keeps its index in a `HashMap`, so `all()` and `drop_job()` list hashes in arbitrary order. 28Here they come in hash order. 29 30## Tests 31 32`tests/store.c`: the Rust's four tests: 33- `write_rename_durability`; 34- `hash_verification`; 35- `a_blob_of_two_jobs_outlives_either`, on disk and in memory, with the link race; 36- `eviction`, on disk and in memory. 37 38## Licence 39 40MIT. See the repository's LICENSE.