Flash normalization: fast RMSNorm for LLMs
2024-07-12Code Available2· sign in to hype
Nils Graef, Matthew Clapp, Andrew Wasielewski
Code Available — Be the first to reproduce this paper.
ReproduceCode
- github.com/openmachine-ai/transformer-tricksOfficialpytorch★ 197
Abstract
RMSNorm is used by many LLMs such as Llama, Mistral, and OpenELM. This paper details FlashNorm, which is an exact but faster implementation of RMSNorm followed by linear layers. See https://huggingface.co/open-machine/FlashNorm for code and more transformer tricks.