content.raw: 🌟 It's been about a week since 
content.type: text
content.value: 🌟 It's been about a week since 
content.raw: @Google
content.type: mention
content.user: Google
content.raw:  dropped Gemma 2 and now Gemma 2 27B is the highest-ranked open-source LLM on LMSYS Chatbot Arena Leaderboard, beating Meta Llama 3 70B and Alibaba Qwen 2 72B! 🚀💪
content.type: text
content.value:  dropped Gemma 2 and now Gemma 2 27B is the highest-ranked open-source LLM on LMSYS Chatbot Arena Leaderboard, beating Meta Llama 3 70B and Alibaba Qwen 2 72B! 🚀💪
content.type: new_line
content.type: new_line
content.raw: 🔍 Here is what a week of Gemma looked like:
content.type: text
content.value: 🔍 Here is what a week of Gemma looked like:
content.type: new_line
content.type: new_line
content.raw: 1️⃣ First, it's a challenging model to run. The only reason I could find is soft-capping of logits within the attention for longer context optimizations. And no one was doing that before! 🤯
content.type: text
content.value: 1️⃣ First, it's a challenging model to run. The only reason I could find is soft-capping of logits within the attention for longer context optimizations. And no one was doing that before! 🤯
content.type: new_line
content.type: new_line
content.raw: 2️⃣ Next, the technical report mentions a 2B model... where is it? 🤔
content.type: text
content.value: 2️⃣ Next, the technical report mentions a 2B model... where is it? 🤔
content.type: new_line
content.type: new_line
content.raw: 3️⃣ Simple things like a context length of 8192 tokens, the Rotary Position Embeddings (RoPE), and the approximated GeGLU non-linearity are similar to earlier Gemma's. 📏🔄
content.type: text
content.value: 3️⃣ Simple things like a context length of 8192 tokens, the Rotary Position Embeddings (RoPE), and the approximated GeGLU non-linearity are similar to earlier Gemma's. 📏🔄
content.type: new_line
content.type: new_line
content.raw: 4️⃣ But a lot of new stuff is here like Local Sliding Window and Global Attention: they alternate between them at every layer... don't know why! 🤷‍♂️ and of course Logit soft-capping. 💡
content.type: text
content.value: 4️⃣ But a lot of new stuff is here like Local Sliding Window and Global Attention: they alternate between them at every layer... don't know why! 🤷‍♂️ and of course Logit soft-capping. 💡
content.type: new_line
content.type: new_line
content.raw: 5️⃣ On-Policy Distillation of Language Models - Knowledge Distillation: Leverage a larger teacher model to train a smaller model (the 9B model). 🎓➡️📚
content.type: text
content.value: 5️⃣ On-Policy Distillation of Language Models - Knowledge Distillation: Leverage a larger teacher model to train a smaller model (the 9B model). 🎓➡️📚
content.type: new_line
content.type: new_line
content.raw: 6️⃣ Model Merging: Combined average models from experiments run with different hyperparameters. 🧪⚗️
content.type: text
content.value: 6️⃣ Model Merging: Combined average models from experiments run with different hyperparameters. 🧪⚗️
content.type: new_line
content.type: new_line
content.raw: 7️⃣ And borrowed Grouped-Query Attention (GQA) from 
content.type: text
content.value: 7️⃣ And borrowed Grouped-Query Attention (GQA) from 
content.raw: @Meta
content.type: mention
content.user: Meta
content.raw:  Llama-3. 🔄🤝
content.type: text
content.value:  Llama-3. 🔄🤝
content.type: new_line
content.type: new_line
content.raw: 📖 One of the best articles on Gemma 2: 
content.type: text
content.value: 📖 One of the best articles on Gemma 2: 
content.href: https://huggingface.co/blog/gemma2
content.raw: https://huggingface.co/blog/gemma2
content.type: link
content.type: new_line
content.type: new_line
content.raw: 📊 Technical report: 
content.type: text
content.value: 📊 Technical report: 
content.href: https://storage.googleapis.com/deepmind-media/gemma/gemma-2-report.pdf
content.raw: https://storage.googleapis.com/deepmind-media/gemma/gemma-2-report.pdf
content.type: link
content.type: new_line
content.type: new_line
content.raw: 🔗 Models: 
content.type: text
content.value: 🔗 Models: 
content.raw: https://huggingface.co/collections/google/gemma-2-release-667d6600fd5220e7b967f315
content.id: google/gemma-2-release-667d6600fd5220e7b967f315
content.type: collection
content.type: resource
content.url: https://huggingface.co/collections/google/gemma-2-release-667d6600fd5220e7b967f315
content.type: new_line
rawContent: 🌟 It's been about a week since @Google dropped Gemma 2 and now Gemma 2 27B is the highest-ranked open-source LLM on LMSYS Chatbot Arena Leaderboard, beating Meta Llama 3 70B and Alibaba Qwen 2 72B! 🚀💪
  
  🔍 Here is what a week of Gemma looked like:
  
  1️⃣ First, it's a challenging model to run. The only reason I could find is soft-capping of logits within the attention for longer context optimizations. And no one was doing that before! 🤯
  
  2️⃣ Next, the technical report mentions a 2B model... where is it? 🤔
  
  3️⃣ Simple things like a context length of 8192 tokens, the Rotary Position Embeddings (RoPE), and the approximated GeGLU non-linearity are similar to earlier Gemma's. 📏🔄
  
  4️⃣ But a lot of new stuff is here like Local Sliding Window and Global Attention: they alternate between them at every layer... don't know why! 🤷‍♂️ and of course Logit soft-capping. 💡
  
  5️⃣ On-Policy Distillation of Language Models - Knowledge Distillation: Leverage a larger teacher model to train a smaller model (the 9B model). 🎓➡️📚
  
  6️⃣ Model Merging: Combined average models from experiments run with different hyperparameters. 🧪⚗️
  
  7️⃣ And borrowed Grouped-Query Attention (GQA) from @Meta Llama-3. 🔄🤝
  
  📖 One of the best articles on Gemma 2: https://huggingface.co/blog/gemma2
  
  📊 Technical report: https://storage.googleapis.com/deepmind-media/gemma/gemma-2-report.pdf
  
  🔗 Models: https://huggingface.co/collections/google/gemma-2-release-667d6600fd5220e7b967f315
  
reactions.reaction: 🧠
reactions.users: louisbrulenaudet