Hugging Face's logo

Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
Back to feed
wassemgtk 
posted an update 18 days ago
Post
2791
I’ve been diving into the iRoPE architecture from Llama 4—a game-changer for long-context models! It interleaves local attention (with RoPE) for short contexts and global attention (with inference-time temp scaling) for long-range reasoning, aiming for infinite context. I’m going to try writing iRoPE—who wants to help?

Code: https://github.com/wassemgtk/iRoPE-try/blob/main/iRoPE.ipynb
wassemgtk

Applied iRoPE to any model like LLaMA 3.2-3B VERY possible! Interleaved local (RoPE) & global (temp-scaled) attention boosts long-context (10M tokens) handling. With chunking & weight transfer, it’s adaptable to "any" transformer model.
Infinite context feels closer 🤯

https://github.com/wassemgtk/iRoPE-try

In this post