GPT-2's linear layer, one output row: sequential dots over k from +0 in Z, then bias, bias-GELU or a residual add
$ sexc add @ajax/mesh-ref-gpt2-linear- total 3
- drwxr-xr-x-../
- drwxr-xr-x-Oct 27 01:59v/# 1 version
- drwxr-xr-x-Oct 27 01:59files/# the files of 0.1.0
- latest
- 0.1.0, published Oct 27 01:59 by ajax
- owners
- ajax
- license
- MIT
- dependencies
- @ajax/mesh-ftz ^0.1.0, @ajax/mesh-ref-gpt2-scalar ^0.1.0
- dependents
- @ajax/mesh-prog-gpt2x, @ajax/mesh-ref-gpt2
- keywords
- alphaos mesh reference gpt2 determinism
- homepage
- https://github.com/thomasdavis/alpha-c-mesh/tree/main/packages/mesh-ref-gpt2-linear
- repository
- https://github.com/thomasdavis/alpha-c-mesh
1# @ajax/mesh-ref-gpt2-linear 2 3A GPT-2 linear layer's output row in the order the GPU's GEMM tile and GEMV give exactly: each output is 4the sum over k ascending from +0 of x[k]·w[k][n] in the cluster float mode, then the epilogue — `+ b`, 5GELU of it, or a residual add into the output in place. Weights are `[in][out]` as the checkpoint stores 6them. `project_cols` computes a column range (a thread's share of a row). 7 8**Ported from** `crates/mesh-ref/src/gpt2.rs` (`Epi`, `project_row`, `project_cols`). Freestanding. 9Compiled with `-O2` in every profile: a forward pass is billions of these operations. 10 11**Tested** in `@ajax/mesh-ref-gpt2`'s golden data (each epilogue, whole rows and a column range).