Good folks at 
text
Good folks at 
@nvidia
mention
nvidia
 have released exciting new research on normalized Transformers (nGPT) for faster and more efficient language modeling! 
text
 have released exciting new research on normalized Transformers (nGPT) for faster and more efficient language modeling! 
new_line
new_line
Here is what they are proposing:
text
Here is what they are proposing:
new_line
new_line
1. Remove all normalization layers, like RMSNorm or LayerNorm, from the standard Transformer architecture.
text
1. Remove all normalization layers, like RMSNorm or LayerNorm, from the standard Transformer architecture.
new_line
new_line
2. Normalize all matrices along their embedding dimension after each training step. This includes input and output embeddings, attention matrices (Q, K, V), output projection matrices, and MLP matrices.
text
2. Normalize all matrices along their embedding dimension after each training step. This includes input and output embeddings, attention matrices (Q, K, V), output projection matrices, and MLP matrices.
new_line
new_line
3. Replace the standard residual connections with normalized update equations using learnable eigen learning rates for the attention and MLP blocks.
text
3. Replace the standard residual connections with normalized update equations using learnable eigen learning rates for the attention and MLP blocks.
new_line
new_line
4. Change the softmax scaling factor in the attention mechanism from 1/sqrt of d_k to sqrt of d_k.
text
4. Change the softmax scaling factor in the attention mechanism from 1/sqrt of d_k to sqrt of d_k.
new_line
new_line
5. Implement rescaling and optional normalization of query (q) and key (k) vectors in the attention mechanism using learnable scaling factors.
text
5. Implement rescaling and optional normalization of query (q) and key (k) vectors in the attention mechanism using learnable scaling factors.
new_line
new_line
6. Rescale the intermediate states of the MLP block using learnable scaling factors.
text
6. Rescale the intermediate states of the MLP block using learnable scaling factors.
new_line
new_line
7. Implement rescaling of the output logits using learnable scaling factors.
text
7. Implement rescaling of the output logits using learnable scaling factors.
new_line
new_line
8. Remove weight decay and learning rate warmup from the optimization process.
text
8. Remove weight decay and learning rate warmup from the optimization process.
new_line
new_line
9. Initialize the eigen learning rates and scaling factors with appropriate values as specified in the paper.
text
9. Initialize the eigen learning rates and scaling factors with appropriate values as specified in the paper.
new_line
new_line
10. During training, treat all vectors and matrices as residing on a unit hypersphere, interpreting matrix-vector multiplications as cosine similarities.
text
10. During training, treat all vectors and matrices as residing on a unit hypersphere, interpreting matrix-vector multiplications as cosine similarities.
new_line
new_line
11. Implement the update equations for the hidden states using the normalized outputs from attention and MLP blocks, controlled by the eigen learning rates.
text
11. Implement the update equations for the hidden states using the normalized outputs from attention and MLP blocks, controlled by the eigen learning rates.
new_line
new_line
12. After each forward pass, normalize all parameter matrices to ensure they remain on the unit hypersphere.
text
12. After each forward pass, normalize all parameter matrices to ensure they remain on the unit hypersphere.
new_line
new_line
13. Use the Adam optimizer without weight decay for training the model.
text
13. Use the Adam optimizer without weight decay for training the model.
new_line
new_line
14. When computing loss, apply the learnable scaling factor to the logits before the softmax operation.
text
14. When computing loss, apply the learnable scaling factor to the logits before the softmax operation.
new_line
new_line
15. During inference, follow the same normalization and scaling procedures as in training.
text
15. During inference, follow the same normalization and scaling procedures as in training.
new_line
new_line
Excited to see how it scales to larger models and datasets!
text
Excited to see how it scales to larger models and datasets!
Good folks at @nvidia have released exciting new research on normalized Transformers (nGPT) for faster and more efficient language modeling! 

Here is what they are proposing:

1. Remove all normalization layers, like RMSNorm or LayerNorm, from the standard Transformer architecture.

2. Normalize all matrices along their embedding dimension after each training step. This includes input and output embeddings, attention matrices (Q, K, V), output projection matrices, and MLP matrices.

3. Replace the standard residual connections with normalized update equations using learnable eigen learning rates for the attention and MLP blocks.

4. Change the softmax scaling factor in the attention mechanism from 1/sqrt of d_k to sqrt of d_k.

5. Implement rescaling and optional normalization of query (q) and key (k) vectors in the attention mechanism using learnable scaling factors.

6. Rescale the intermediate states of the MLP block using learnable scaling factors.

7. Implement rescaling of the output logits using learnable scaling factors.

8. Remove weight decay and learning rate warmup from the optimization process.

9. Initialize the eigen learning rates and scaling factors with appropriate values as specified in the paper.

10. During training, treat all vectors and matrices as residing on a unit hypersphere, interpreting matrix-vector multiplications as cosine similarities.

11. Implement the update equations for the hidden states using the normalized outputs from attention and MLP blocks, controlled by the eigen learning rates.

12. After each forward pass, normalize all parameter matrices to ensure they remain on the unit hypersphere.

13. Use the Adam optimizer without weight decay for training the model.

14. When computing loss, apply the learnable scaling factor to the logits before the softmax operation.

15. During inference, follow the same normalization and scaling procedures as in training.

Excited to see how it scales to larger models and datasets!
🔥
DmitryRyumin prithivMLmods EquinoxElahin John6666 ThijsL202 djuna jb317 celsowm