🌟 It's been about a week since 
text
🌟 It's been about a week since 
@Google
mention
Google
 dropped Gemma 2 and now Gemma 2 27B is the highest-ranked open-source LLM on LMSYS Chatbot Arena Leaderboard, beating Meta Llama 3 70B and Alibaba Qwen 2 72B! 🚀💪
text
 dropped Gemma 2 and now Gemma 2 27B is the highest-ranked open-source LLM on LMSYS Chatbot Arena Leaderboard, beating Meta Llama 3 70B and Alibaba Qwen 2 72B! 🚀💪
new_line
new_line
🔍 Here is what a week of Gemma looked like:
text
🔍 Here is what a week of Gemma looked like:
new_line
new_line
1️⃣ First, it's a challenging model to run. The only reason I could find is soft-capping of logits within the attention for longer context optimizations. And no one was doing that before! 🤯
text
1️⃣ First, it's a challenging model to run. The only reason I could find is soft-capping of logits within the attention for longer context optimizations. And no one was doing that before! 🤯
new_line
new_line
2️⃣ Next, the technical report mentions a 2B model... where is it? 🤔
text
2️⃣ Next, the technical report mentions a 2B model... where is it? 🤔
new_line
new_line
3️⃣ Simple things like a context length of 8192 tokens, the Rotary Position Embeddings (RoPE), and the approximated GeGLU non-linearity are similar to earlier Gemma's. 📏🔄
text
3️⃣ Simple things like a context length of 8192 tokens, the Rotary Position Embeddings (RoPE), and the approximated GeGLU non-linearity are similar to earlier Gemma's. 📏🔄
new_line
new_line
4️⃣ But a lot of new stuff is here like Local Sliding Window and Global Attention: they alternate between them at every layer... don't know why! 🤷‍♂️ and of course Logit soft-capping. 💡
text
4️⃣ But a lot of new stuff is here like Local Sliding Window and Global Attention: they alternate between them at every layer... don't know why! 🤷‍♂️ and of course Logit soft-capping. 💡
new_line
new_line
5️⃣ On-Policy Distillation of Language Models - Knowledge Distillation: Leverage a larger teacher model to train a smaller model (the 9B model). 🎓➡️📚
text
5️⃣ On-Policy Distillation of Language Models - Knowledge Distillation: Leverage a larger teacher model to train a smaller model (the 9B model). 🎓➡️📚
new_line
new_line
6️⃣ Model Merging: Combined average models from experiments run with different hyperparameters. 🧪⚗️
text
6️⃣ Model Merging: Combined average models from experiments run with different hyperparameters. 🧪⚗️
new_line
new_line
7️⃣ And borrowed Grouped-Query Attention (GQA) from 
text
7️⃣ And borrowed Grouped-Query Attention (GQA) from 
@Meta
mention
Meta
 Llama-3. 🔄🤝
text
 Llama-3. 🔄🤝
new_line
new_line
📖 One of the best articles on Gemma 2: 
text
📖 One of the best articles on Gemma 2: 
https://huggingface.co/blog/gemma2
https://huggingface.co/blog/gemma2
link
new_line
new_line
📊 Technical report: 
text
📊 Technical report: 
https://storage.googleapis.com/deepmind-media/gemma/gemma-2-report.pdf
https://storage.googleapis.com/deepmind-media/gemma/gemma-2-report.pdf
link
new_line
new_line
🔗 Models: 
text
🔗 Models: 
https://huggingface.co/collections/google/gemma-2-release-667d6600fd5220e7b967f315
google/gemma-2-release-667d6600fd5220e7b967f315
collection
resource
https://huggingface.co/collections/google/gemma-2-release-667d6600fd5220e7b967f315
new_line
🌟 It's been about a week since @Google dropped Gemma 2 and now Gemma 2 27B is the highest-ranked open-source LLM on LMSYS Chatbot Arena Leaderboard, beating Meta Llama 3 70B and Alibaba Qwen 2 72B! 🚀💪

🔍 Here is what a week of Gemma looked like:

1️⃣ First, it's a challenging model to run. The only reason I could find is soft-capping of logits within the attention for longer context optimizations. And no one was doing that before! 🤯

2️⃣ Next, the technical report mentions a 2B model... where is it? 🤔

3️⃣ Simple things like a context length of 8192 tokens, the Rotary Position Embeddings (RoPE), and the approximated GeGLU non-linearity are similar to earlier Gemma's. 📏🔄

4️⃣ But a lot of new stuff is here like Local Sliding Window and Global Attention: they alternate between them at every layer... don't know why! 🤷‍♂️ and of course Logit soft-capping. 💡

5️⃣ On-Policy Distillation of Language Models - Knowledge Distillation: Leverage a larger teacher model to train a smaller model (the 9B model). 🎓➡️📚

6️⃣ Model Merging: Combined average models from experiments run with different hyperparameters. 🧪⚗️

7️⃣ And borrowed Grouped-Query Attention (GQA) from @Meta Llama-3. 🔄🤝

📖 One of the best articles on Gemma 2: https://huggingface.co/blog/gemma2

📊 Technical report: https://storage.googleapis.com/deepmind-media/gemma/gemma-2-report.pdf

🔗 Models: https://huggingface.co/collections/google/gemma-2-release-667d6600fd5220e7b967f315

🧠
louisbrulenaudet