gpt2x, GPT-2 124M's exact-mode GPU kernels (embed, norm, GEMM/GEMV, KV, attention, logits) bitwise equal to the CPU reference, as PTX from C
$ sexc add @ajax/mesh-prog-gpt2x- total 3
- drwxr-xr-x-../
- drwxr-xr-x-Oct 27 03:26v/# 1 version
- drwxr-xr-x-Oct 27 03:26files/# the files of 0.1.0
- latest
- 0.1.0, published Oct 27 03:26 by ajax
- owners
- ajax
- license
- MIT
- dependencies
- @ajax/mesh-rt ^0.1.0, @ajax/mesh-rt-linalg ^0.1.0, @ajax/mesh-rt-sim-mailbox ^0.1.0, @ajax/mesh-ref-block ^0.1.0, @ajax/mesh-ref-gpt2-scalar ^0.1.0, @ajax/mesh-ref-gpt2-norm ^0.1.0, @ajax/mesh-ref-gpt2-attend ^0.1.0, @ajax/mesh-ref-gpt2-linear ^0.1.0, @ajax/mesh-check ^0.1.0
- dependents
- none yet
- keywords
- mesh alphaos gpu ptx program gpt2 transformer deterministic
- homepage
- https://github.com/thomasdavis/alpha-c-mesh/tree/main/programs/gpt2x
- repository
- https://github.com/thomasdavis/alpha-c-mesh
1# @ajax/mesh-prog-gpt2x 2 3GPT-2 124M's exact-mode GPU kernels: the window (PUT, COPY), EMBED, NORM, GEMM and GEMV with their 4epilogues (bias, GELU, residual), KV, ATTN and LOGITS, every op following the CPU reference operation 5for operation (FTZ, correctly rounded, unfused, fixed sum orders; exp, tanh and sqrt from + - * /), so 6the logits and the KV cache are bitwise the reference's on every GPU. The host runs the model as 7stages of tasks. Layout: @ajax/mesh-abi-view-gpt2x. 8 9- **Ported from:** alpha-rust-mesh `programs/gpt2x/src/lib.rs`. 10- **Tests:** on the simulator, op by op against @ajax/mesh-ref-gpt2-*'s row operations, bitwise, on 11 synthetic data: the scalar functions over a sweep, the window, EMBED, NORM, GEMM and GEMV with each 12 epilogue off the tile grid, KV and causal ATTN for 12 heads, LOGITS for the first and last blocks.