content.raw: "The Case for Co-Designing Model Architectures with Hardware"
content.type: text
content.value: "The Case for Co-Designing Model Architectures with Hardware"
content.type: new_line
content.type: new_line
content.raw: This is a long overdue paper that we have started discussing back when training BLOOM-176.
content.type: text
content.value: This is a long overdue paper that we have started discussing back when training BLOOM-176.
content.type: new_line
content.type: new_line
content.raw: Basically this paper tells you how to design your model's dimensions for an optimal training throughput.
content.type: text
content.value: Basically this paper tells you how to design your model's dimensions for an optimal training throughput.
content.type: new_line
content.type: new_line
content.raw: Fantastic!
content.type: text
content.value: Fantastic!
content.type: new_line
content.type: new_line
content.raw: Yours truly contributed the SwiGLU section ;)
content.type: text
content.value: Yours truly contributed the SwiGLU section ;)
content.type: new_line
content.type: new_line
content.href: https://twitter.com/QuentinAnthon15/status/1752393989813375119
content.raw: https://twitter.com/QuentinAnthon15/status/1752393989813375119
content.type: link
content.type: new_line
content.type: new_line
content.href: https://arxiv.org/abs/2401.14489
content.raw: https://arxiv.org/abs/2401.14489
content.type: link
content.type: new_line
rawContent: "The Case for Co-Designing Model Architectures with Hardware"
  
  This is a long overdue paper that we have started discussing back when training BLOOM-176.
  
  Basically this paper tells you how to design your model's dimensions for an optimal training throughput.
  
  Fantastic!
  
  Yours truly contributed the SwiGLU section ;)
  
  https://twitter.com/QuentinAnthon15/status/1752393989813375119
  
  https://arxiv.org/abs/2401.14489
  
reactions.reaction: ❤️
reactions.users: clem qanthony radames osanseviero abidlabs samusenps jeffboudier efederici merve muhtasham
reactions.reaction: 🤝
reactions.users: clem osanseviero abidlabs merve muhtasham
commentators._id: 5f07383b19cb630495b812cd
commentators.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1594311341799-5f07383b19cb630495b812cd.jpeg
commentators.fullname: Stas Bekman
commentators.name: stas
commentators.type: user
commentators._id: 5fac18fb5eec0323e9470ba2
commentators.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1605114051380-noauth.jpeg
commentators.fullname: Jeff Boudier
commentators.name: jeffboudier
commentators.type: user