content.raw: Lightweight (nanoGPT)  implementation of hybrid norm - an intuitive normalization method that combines the strength of both pre-norm (i.e QKV-norm in MHA) and post-norm in the feed-forward network.
content.type: text
content.value: Lightweight (nanoGPT)  implementation of hybrid norm - an intuitive normalization method that combines the strength of both pre-norm (i.e QKV-norm in MHA) and post-norm in the feed-forward network.
content.type: new_line
content.raw: Code:  
content.type: text
content.value: Code:  
content.href: https://github.com/Jaykef/ai-algorithms/blob/main/hybrid_normalization.ipynb
content.raw: https://github.com/Jaykef/ai-algorithms/blob/main/hybrid_normalization.ipynb
content.type: link
rawContent: Lightweight (nanoGPT)  implementation of hybrid norm - an intuitive normalization method that combines the strength of both pre-norm (i.e QKV-norm in MHA) and post-norm in the feed-forward network.
  Code:  https://github.com/Jaykef/ai-algorithms/blob/main/hybrid_normalization.ipynb
reactions.reaction: 👀
reactions.users: John6666 nicolay-r Fishtiks robb-0 dantezxcd
reactions.reaction: 🔥
reactions.users: nicolay-r anthonymikinka dantezxcd