Lightweight (nanoGPT)  implementation of hybrid norm - an intuitive normalization method that combines the strength of both pre-norm (i.e QKV-norm in MHA) and post-norm in the feed-forward network.
text
Lightweight (nanoGPT)  implementation of hybrid norm - an intuitive normalization method that combines the strength of both pre-norm (i.e QKV-norm in MHA) and post-norm in the feed-forward network.
new_line
Code:  
text
Code:  
https://github.com/Jaykef/ai-algorithms/blob/main/hybrid_normalization.ipynb
https://github.com/Jaykef/ai-algorithms/blob/main/hybrid_normalization.ipynb
link
Lightweight (nanoGPT)  implementation of hybrid norm - an intuitive normalization method that combines the strength of both pre-norm (i.e QKV-norm in MHA) and post-norm in the feed-forward network.
Code:  https://github.com/Jaykef/ai-algorithms/blob/main/hybrid_normalization.ipynb
👀
John6666 nicolay-r Fishtiks robb-0 dantezxcd
🔥
nicolay-r anthonymikinka dantezxcd