GPT-2's layer normalization of a row: 256-thread strided sums, then the block tree, in Z
$ sexc add @ajax/mesh-ref-gpt2-norm- total 3
- drwxr-xr-x-../
- drwxr-xr-x-Oct 27 01:59v/# 1 version
- drwxr-xr-x-Oct 27 01:59files/# the files of 0.1.0
- latest
- 0.1.0, published Oct 27 01:59 by ajax
- owners
- ajax
- license
- MIT
- dependencies
- @ajax/mesh-ftz ^0.1.0, @ajax/mesh-ref-block ^0.1.0, @ajax/mesh-ref-gpt2-shape ^0.1.0, @ajax/mesh-ref-gpt2-scalar ^0.1.0
- dependents
- @ajax/mesh-prog-gpt2x, @ajax/mesh-ref-gpt2
- keywords
- alphaos mesh reference gpt2 determinism
- homepage
- https://github.com/thomasdavis/alpha-c-mesh/tree/main/packages/mesh-ref-gpt2-norm
- repository
- https://github.com/thomasdavis/alpha-c-mesh
1# @ajax/mesh-ref-gpt2-norm 2 3GPT-2's layer normalization of a row of 768, in the GPU node's order: thread t of 256 sums x[t], 4x[t + 256], x[t + 512] from +0, then the block tree (`@ajax/mesh-ref-block`); the same for the squared 5deviations; then `((x − mean) · 1/sqrt(var + eps)) · g + b`, all in the cluster float mode. 6 7**Ported from** `crates/mesh-ref/src/gpt2.rs` (`row_reduce`, `normalize_row`). Freestanding. 8**Tested** in `@ajax/mesh-ref-gpt2`'s golden data.