GPT-2 124M's shape: layers, heads, widths, vocabulary, context, and the flat weight buffer's layout in checkpoint order
$ sexc add @ajax/mesh-ref-gpt2-shape- total 3
- drwxr-xr-x-../
- drwxr-xr-x-Oct 27 01:59v/# 1 version
- drwxr-xr-x-Oct 27 01:59files/# the files of 0.1.0
- latest
- 0.1.0, published Oct 27 01:59 by ajax
- owners
- ajax
- license
- MIT
- dependencies
- none
- dependents
- @ajax/mesh-ref-gpt2, @ajax/mesh-ref-gpt2-attend, @ajax/mesh-ref-gpt2-kv, @ajax/mesh-ref-gpt2-norm, @ajax/mesh-ref-gpt2-weights
- keywords
- alphaos mesh reference gpt2 determinism
- homepage
- https://github.com/thomasdavis/alpha-c-mesh/tree/main/packages/mesh-ref-gpt2-shape
- repository
- https://github.com/thomasdavis/alpha-c-mesh
1# @ajax/mesh-ref-gpt2-shape 2 3GPT-2 124M's shape — 12 layers, 12 heads of 64, width 768, feed-forward 3072, vocabulary 50 257, context 41024, the node's 256 threads — and the flat weight buffer's layout: every tensor in checkpoint order and 5shape (`wte`, `wpe`, the twelve tensors of each block, `ln_f`), the same on the CPU and in the GPU's 6scratch. `mesh_gpt2_tensor` gives each tensor's checkpoint name, offset and count. 7 8**Ported from** `crates/mesh-ref/src/gpt2.rs` (the constants, `BlockOff`, `Layout`, `Layout::tensors`). 9Freestanding. **Tested** by gpt2.rs's `layout_covers_the_checkpoint` and a golden digest of every 10tensor's name, offset and count from the Rust. 11 12## Differences 13 14- `tensors()` returns a `Vec<(String, usize, usize)>`; here `mesh_gpt2_tensor(i, ...)` fills a buffer.