GPT-2 124M synthetic weights from a counter hash: kcheck's checks without a checkpoint
$ sexc add @ajax/mesh-ref-gpt2-weights- total 3
- drwxr-xr-x-../
- drwxr-xr-x-Oct 27 01:59v/# 1 version
- drwxr-xr-x-Oct 27 01:59files/# the files of 0.1.0
- latest
- 0.1.0, published Oct 27 01:59 by ajax
- owners
- ajax
- license
- MIT
- dependencies
- @ajax/mesh-ftz ^0.1.0, @ajax/mesh-ref-matmul ^0.1.0, @ajax/mesh-ref-gpt2-shape ^0.1.0
- dependents
- @ajax/mesh-ref-gpt2
- keywords
- alphaos mesh reference gpt2 determinism
- homepage
- https://github.com/thomasdavis/alpha-c-mesh/tree/main/packages/mesh-ref-gpt2-weights
- repository
- https://github.com/thomasdavis/alpha-c-mesh
1# @ajax/mesh-ref-gpt2-weights 2 3Synthetic GPT-2 124M weights from a counter hash, for checks without a checkpoint: each tensor gets its 4own seed (`seed · 0x9e3779b9 + tensor index`) and `@ajax/mesh-ref-matmul`'s values scaled — matrices to 5[−0.05, 0.05), biases to [−0.01, 0.01), layer-norm gains to 1 ± 0.1 and their biases to ±0.1. 6 7**Ported from** `crates/mesh-ref/src/gpt2.rs` (`synthetic`). Host only (allocates ~500 MB). 8**Tested** by a sampled golden digest of the whole buffer from the Rust (in `@ajax/mesh-ref-gpt2`).