meshd's side of an inference request: which members serve it, the call to their gpt2 worker, the answer
$ sexc add @ajax/mesh-inferd- total 3
- drwxr-xr-x-../
- drwxr-xr-x-Oct 27 05:23v/# 1 version
- drwxr-xr-x-Oct 27 05:23files/# the files of 0.1.0
- latest
- 0.1.0, published Oct 27 05:23 by ajax
- owners
- ajax
- license
- MIT
- dependencies
- @ajax/mesh-shared ^0.1.0, @ajax/mesh-infer-wire ^0.1.0, @ajax/mesh-infer-request ^0.1.0, @ajax/mesh-infer-svc ^0.1.0, @ajax/mesh-prog-table ^0.1.0, @ajax/mesh-manifest ^0.1.0, @ajax/mesh-log ^0.1.0, @ajax/mesh-buf ^0.1.0, @ajax/mesh-rsfmt ^0.1.0, @ajax/mesh-chan ^0.1.0, @ajax/mesh-check ^0.1.0
- dependents
- @ajax/mesh-control, @ajax/mesh-http-relay
- keywords
- mesh alphaos inference gpt2 fleet
- homepage
- https://github.com/thomasdavis/alpha-c-mesh/tree/main/packages/mesh-inferd
- repository
- https://github.com/thomasdavis/alpha-c-mesh
1# @ajax/mesh-inferd 2 3meshd's side of an inference request: which members serve it, the call to their gpt2 worker, the answer. 4 5meshd's side of an inference request (inferd.rs). 6 7The pair: explicit (`prefill=K decode=J`, `mono=K`), or the members where the fleet's desired state placed gpt2-prefill and gpt2-decode (this member's committed copy, read from the `fleet` text), or a monolithic gpt2. Then one call to the decode (or monolithic) member's worker, which answers K_RESULT: the tokens, the text, the logits digest, the timings. 8 9```c 10#include "mesh/inferd.h" 11``` 12 13## Ported from 14 15[alpha-rust-mesh](https://github.com/rsafier/alpha-rust-mesh) (Richard Safier, MIT): crates/mesh-host/src/inferd.rs: `members_of`, `param_of`, `run`. 16 17## Tests 18 19- `tests/inferd.c`: a gpt2 worker on loopback answers a mono request (the message meshd sends, the result it returns with coordinator, request_ms and program), its K_ERROR, a refused connection, and no member placed. 20 21## Licence 22 23MIT. See the repository's LICENSE.