content.raw: 📜 𝐎𝐥𝐝-𝐬𝐜𝐡𝐨𝐨𝐥 𝐑𝐍𝐍𝐬 𝐜𝐚𝐧 𝐚𝐜𝐭𝐮𝐚𝐥𝐥𝐲 𝐫𝐢𝐯𝐚𝐥 𝐟𝐚𝐧𝐜𝐲 𝐭𝐫𝐚𝐧𝐬𝐟𝐨𝐫𝐦𝐞𝐫𝐬!
content.type: text
content.value: 📜 𝐎𝐥𝐝-𝐬𝐜𝐡𝐨𝐨𝐥 𝐑𝐍𝐍𝐬 𝐜𝐚𝐧 𝐚𝐜𝐭𝐮𝐚𝐥𝐥𝐲 𝐫𝐢𝐯𝐚𝐥 𝐟𝐚𝐧𝐜𝐲 𝐭𝐫𝐚𝐧𝐬𝐟𝐨𝐫𝐦𝐞𝐫𝐬!
content.type: new_line
content.type: new_line
content.raw: Researchers from Mila and Borealis AI just have shown that simplified versions of good old Recurrent Neural Networks (RNNs) can match the performance of today's transformers.
content.type: text
content.value: Researchers from Mila and Borealis AI just have shown that simplified versions of good old Recurrent Neural Networks (RNNs) can match the performance of today's transformers.
content.type: new_line
content.type: new_line
content.raw: They took a fresh look at LSTMs (from 1997!) and GRUs (from 2014). They stripped these models down to their bare essentials, creating "minLSTM" and "minGRU". The key changes:
content.type: text
content.value: They took a fresh look at LSTMs (from 1997!) and GRUs (from 2014). They stripped these models down to their bare essentials, creating "minLSTM" and "minGRU". The key changes:
content.type: new_line
content.raw: ❶ Removed dependencies on previous hidden states in the gates
content.type: text
content.value: ❶ Removed dependencies on previous hidden states in the gates
content.type: new_line
content.raw: ❷ Dropped the tanh that had been added to restrict output range in order to avoid vanishing gradients
content.type: text
content.value: ❷ Dropped the tanh that had been added to restrict output range in order to avoid vanishing gradients
content.type: new_line
content.raw: ❸ Ensured outputs are time-independent in scale (not sure I understood that well either, don't worry)
content.type: text
content.value: ❸ Ensured outputs are time-independent in scale (not sure I understood that well either, don't worry)
content.type: new_line
content.type: new_line
content.raw: ⚡️ As a result, you can use a “parallel scan” algorithm to train these new, minimal RNNs, in parallel, taking 88% more memory but also making them 200x faster than their traditional counterparts for long sequences
content.type: text
content.value: ⚡️ As a result, you can use a “parallel scan” algorithm to train these new, minimal RNNs, in parallel, taking 88% more memory but also making them 200x faster than their traditional counterparts for long sequences
content.type: new_line
content.type: new_line
content.raw: 🔥 The results are mind-blowing! Performance-wise, they go toe-to-toe with Transformers or Mamba.
content.type: text
content.value: 🔥 The results are mind-blowing! Performance-wise, they go toe-to-toe with Transformers or Mamba.
content.type: new_line
content.type: new_line
content.raw: And for Language Modeling, they need 2.5x fewer training steps than Transformers to reach the same performance! 🚀
content.type: text
content.value: And for Language Modeling, they need 2.5x fewer training steps than Transformers to reach the same performance! 🚀
content.type: new_line
content.type: new_line
content.raw: 🤔 Why does this matter?
content.type: text
content.value: 🤔 Why does this matter?
content.type: new_line
content.type: new_line
content.raw: By showing there are simpler models with similar performance to transformers, this challenges the narrative that we need advanced architectures for better performance!
content.type: text
content.value: By showing there are simpler models with similar performance to transformers, this challenges the narrative that we need advanced architectures for better performance!
content.type: new_line
content.type: new_line
content.raw: 💬 François Chollet wrote in a tweet about this paper:
content.type: text
content.value: 💬 François Chollet wrote in a tweet about this paper:
content.type: new_line
content.type: new_line
content.raw: “The fact that there are many recent architectures coming from different directions that roughly match Transformers is proof that architectures aren't fundamentally important in the curve-fitting paradigm (aka deep learning)”
content.type: text
content.value: “The fact that there are many recent architectures coming from different directions that roughly match Transformers is proof that architectures aren't fundamentally important in the curve-fitting paradigm (aka deep learning)”
content.type: new_line
content.type: new_line
content.raw: “Curve-fitting is about embedding a dataset on a curve. The critical factor is the dataset, not the specific hard-coded bells and whistles that constrain the curve's shape.”
content.type: text
content.value: “Curve-fitting is about embedding a dataset on a curve. The critical factor is the dataset, not the specific hard-coded bells and whistles that constrain the curve's shape.”
content.type: new_line
content.type: new_line
content.raw: It’s the Bitter lesson by Rich Sutton striking again: don’t need fancy thinking architectures, just scale up your model and data!
content.type: text
content.value: It’s the Bitter lesson by Rich Sutton striking again: don’t need fancy thinking architectures, just scale up your model and data!
content.type: new_line
content.type: new_line
content.raw: Read the paper 👉 
content.type: text
content.value: Read the paper 👉 
content.label: Were RNNs All We Needed? (2410.01201)
content.raw: https://huggingface.co/papers/2410.01201
content.id: 2410.01201
content.type: paper
content.type: resource
content.url: https://huggingface.co/papers/2410.01201
rawContent: 📜 𝐎𝐥𝐝-𝐬𝐜𝐡𝐨𝐨𝐥 𝐑𝐍𝐍𝐬 𝐜𝐚𝐧 𝐚𝐜𝐭𝐮𝐚𝐥𝐥𝐲 𝐫𝐢𝐯𝐚𝐥 𝐟𝐚𝐧𝐜𝐲 𝐭𝐫𝐚𝐧𝐬𝐟𝐨𝐫𝐦𝐞𝐫𝐬!
  
  Researchers from Mila and Borealis AI just have shown that simplified versions of good old Recurrent Neural Networks (RNNs) can match the performance of today's transformers.
  
  They took a fresh look at LSTMs (from 1997!) and GRUs (from 2014). They stripped these models down to their bare essentials, creating "minLSTM" and "minGRU". The key changes:
  ❶ Removed dependencies on previous hidden states in the gates
  ❷ Dropped the tanh that had been added to restrict output range in order to avoid vanishing gradients
  ❸ Ensured outputs are time-independent in scale (not sure I understood that well either, don't worry)
  
  ⚡️ As a result, you can use a “parallel scan” algorithm to train these new, minimal RNNs, in parallel, taking 88% more memory but also making them 200x faster than their traditional counterparts for long sequences
  
  🔥 The results are mind-blowing! Performance-wise, they go toe-to-toe with Transformers or Mamba.
  
  And for Language Modeling, they need 2.5x fewer training steps than Transformers to reach the same performance! 🚀
  
  🤔 Why does this matter?
  
  By showing there are simpler models with similar performance to transformers, this challenges the narrative that we need advanced architectures for better performance!
  
  💬 François Chollet wrote in a tweet about this paper:
  
  “The fact that there are many recent architectures coming from different directions that roughly match Transformers is proof that architectures aren't fundamentally important in the curve-fitting paradigm (aka deep learning)”
  
  “Curve-fitting is about embedding a dataset on a curve. The critical factor is the dataset, not the specific hard-coded bells and whistles that constrain the curve's shape.”
  
  It’s the Bitter lesson by Rich Sutton striking again: don’t need fancy thinking architectures, just scale up your model and data!
  
  Read the paper 👉 https://huggingface.co/papers/2410.01201
reactions.reaction: 👍
reactions.users: ajibawa-2023 YaTharThShaRma999 Q-bert deusilence ethancl1 den0620 antiven0m nnikolovskii Joseph717171 kotyKD DmitryRyumin slai1988 fblgit Ryukijano phi0112358 privategeek24 ntnq
reactions.reaction: 👀
reactions.users: YaTharThShaRma999 John6666 Joseph717171 connorads louisbrulenaudet
commentators._id: 63c1dfd4a0ffa3857eb362a9
commentators.avatarUrl: /avatars/ea4398745974d781ae9dc0e95b12cabe.svg
commentators.fullname: Joseph
commentators.name: Joseph717171
commentators.type: user
commentators._id: 6401c8c9f98fbc64bcd7dca1
commentators.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6401c8c9f98fbc64bcd7dca1/MOSgc_mPbfUZ-354osy1v.png
commentators.fullname: FBL
commentators.name: fblgit
commentators.type: user