Hugging Face's logo
Rakshith's picture
1 1

Rakshith

rakshith-writer
wassemgtk's profile picture
·

AI & ML interests

None yet

Recent Activity

reacted to wassemgtk's post with 😎 10 days ago
I’ve been diving into the iRoPE architecture from Llama 4—a game-changer for long-context models! It interleaves local attention (with RoPE) for short contexts and global attention (with inference-time temp scaling) for long-range reasoning, aiming for infinite context. I’m going to try writing iRoPE—who wants to help? Code: https://github.com/wassemgtk/iRoPE-try/blob/main/iRoPE.ipynb
reacted to wassemgtk's post with 👀 10 days ago
I’ve been diving into the iRoPE architecture from Llama 4—a game-changer for long-context models! It interleaves local attention (with RoPE) for short contexts and global attention (with inference-time temp scaling) for long-range reasoning, aiming for infinite context. I’m going to try writing iRoPE—who wants to help? Code: https://github.com/wassemgtk/iRoPE-try/blob/main/iRoPE.ipynb
liked a dataset 2 months ago
Writer/FailSafeQA
View all activity

Organizations

Writer's profile picture

rakshith-writer's activity

reacted to wassemgtk's post with 😎👀 10 days ago
view post
Post
2791
I’ve been diving into the iRoPE architecture from Llama 4—a game-changer for long-context models! It interleaves local attention (with RoPE) for short contexts and global attention (with inference-time temp scaling) for long-range reasoning, aiming for infinite context. I’m going to try writing iRoPE—who wants to help?

Code: https://github.com/wassemgtk/iRoPE-try/blob/main/iRoPE.ipynb
  • 1 reply
·
liked a dataset 2 months ago
upvoted a paper 2 months ago