content.raw: 🔍 Today's pick in Interpretability & Analysis of LMs: In-Context Language Learning: Architectures and Algorithms by E. Akyürek, B. Wang, Y. Kim, J. Andreas
content.type: text
content.value: 🔍 Today's pick in Interpretability & Analysis of LMs: In-Context Language Learning: Architectures and Algorithms by E. Akyürek, B. Wang, Y. Kim, J. Andreas
content.type: new_line
content.type: new_line
content.raw: This work methodically evaluates of in-context learning on formal languages across several model architectures, showing that Transformers outperform all other recurrent and convolutional models, including SSMs. These results are attributed to the presence of “n-gram heads” able to retrieve
content.type: text
content.value: This work methodically evaluates of in-context learning on formal languages across several model architectures, showing that Transformers outperform all other recurrent and convolutional models, including SSMs. These results are attributed to the presence of “n-gram heads” able to retrieve
content.type: new_line
content.raw: the token following a context already seen in the current context window and copy it. This idea is further supported by a better ability of Transformer models to encode in-context n-gram frequencies for n>1, and a higher similarity of Transformer-based LM outputs with classical n-gram models trained on the same data. Finally, these insights are applied to the design static attention layers mimicking the behavior of n-gram head, which lead to lower perplexity despite the lower computational costs.
content.type: text
content.value: the token following a context already seen in the current context window and copy it. This idea is further supported by a better ability of Transformer models to encode in-context n-gram frequencies for n>1, and a higher similarity of Transformer-based LM outputs with classical n-gram models trained on the same data. Finally, these insights are applied to the design static attention layers mimicking the behavior of n-gram head, which lead to lower perplexity despite the lower computational costs.
content.type: new_line
content.type: new_line
content.raw: 📄 Paper: 
content.type: text
content.value: 📄 Paper: 
content.label: In-Context Language Learning: Architectures and Algorithms (2401.12973)
content.raw: https://huggingface.co/papers/2401.12973
content.id: 2401.12973
content.type: paper
content.type: resource
content.url: https://huggingface.co/papers/2401.12973
content.type: new_line
content.raw: 💻 Code: 
content.type: text
content.value: 💻 Code: 
content.href: https://github.com/berlino/seq_icl
content.raw: https://github.com/berlino/seq_icl
content.type: link
rawContent: 🔍 Today's pick in Interpretability & Analysis of LMs: In-Context Language Learning: Architectures and Algorithms by E. Akyürek, B. Wang, Y. Kim, J. Andreas
  
  This work methodically evaluates of in-context learning on formal languages across several model architectures, showing that Transformers outperform all other recurrent and convolutional models, including SSMs. These results are attributed to the presence of “n-gram heads” able to retrieve
  the token following a context already seen in the current context window and copy it. This idea is further supported by a better ability of Transformer models to encode in-context n-gram frequencies for n>1, and a higher similarity of Transformer-based LM outputs with classical n-gram models trained on the same data. Finally, these insights are applied to the design static attention layers mimicking the behavior of n-gram head, which lead to lower perplexity despite the lower computational costs.
  
  📄 Paper: https://huggingface.co/papers/2401.12973
  💻 Code: https://github.com/berlino/seq_icl
reactions.reaction: ❤️
reactions.users: merve osanseviero clefourrier clem
commentators._id: 6141a88b3a0ec78603c9e784
commentators.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6141a88b3a0ec78603c9e784/DJsxSmWV39M33JFheLobC.jpeg
commentators.fullname: Merve Noyan
commentators.name: merve
commentators.type: user