content.raw: Meta Researchers: How many compute hours should we use to train Llama 3.1?
content.type: text
content.value: Meta Researchers: How many compute hours should we use to train Llama 3.1?
content.type: new_line
content.raw: Mr. Zuck: Yes! 🤖💪
content.type: text
content.value: Mr. Zuck: Yes! 🤖💪
content.type: new_line
content.type: new_line
content.raw: Good folks at 
content.type: text
content.value: Good folks at 
content.raw: @AIatMeta
content.type: mention
content.user: AIatMeta
content.raw:  did not just release the models but also published a 92-page detailed paper 📄 on their findings and technical aspects of the models and their training process!
content.type: text
content.value:  did not just release the models but also published a 92-page detailed paper 📄 on their findings and technical aspects of the models and their training process!
content.type: new_line
content.type: new_line
content.raw: Generally, we just gobble up these weights and forget the compute infrastructure used to train these models. 🖥️🚀
content.type: text
content.value: Generally, we just gobble up these weights and forget the compute infrastructure used to train these models. 🖥️🚀
content.type: new_line
content.type: new_line
content.type: new_line
content.raw: Here are some interesting findings about the computing infrastructure of Llamas:
content.type: text
content.value: Here are some interesting findings about the computing infrastructure of Llamas:
content.type: new_line
content.type: new_line
content.raw: - Llama 1 and 2 models were trained on 
content.type: text
content.value: - Llama 1 and 2 models were trained on 
content.raw: @Meta
content.type: mention
content.user: Meta
content.raw:  's AI Research SuperCluster. Llama 3 was migrated to Meta’s production clusters! 📊
content.type: text
content.value:  's AI Research SuperCluster. Llama 3 was migrated to Meta’s production clusters! 📊
content.type: new_line
content.type: new_line
content.raw: - That's 16,000 H100 GPUs, with each GPU featuring 700W TDP and 80GB HBM3, arranged in Meta’s Grand Teton AI server platform. 🖥️🔋
content.type: text
content.value: - That's 16,000 H100 GPUs, with each GPU featuring 700W TDP and 80GB HBM3, arranged in Meta’s Grand Teton AI server platform. 🖥️🔋
content.type: new_line
content.type: new_line
content.raw: - What about storing checkpoints? Used Tectonic, a distributed file system, for storage, with capacities reaching 240 PB and peak throughput of 7 TB/s. 💾📈
content.type: text
content.value: - What about storing checkpoints? Used Tectonic, a distributed file system, for storage, with capacities reaching 240 PB and peak throughput of 7 TB/s. 💾📈
content.type: new_line
content.type: new_line
content.raw: - Meta's mad lads saved each GPU’s model state, ranging from 1 MB to 4 GB per GPU, for recovery and debugging. 🛠️🔍
content.type: text
content.value: - Meta's mad lads saved each GPU’s model state, ranging from 1 MB to 4 GB per GPU, for recovery and debugging. 🛠️🔍
content.type: new_line
content.type: new_line
content.type: new_line
content.raw: If this sounds big, well, they document the humungous challenges that come with it:
content.type: text
content.value: If this sounds big, well, they document the humungous challenges that come with it:
content.type: new_line
content.type: new_line
content.raw: - In the 54-day training period, there were 466 job interruptions. 🕒🔄
content.type: text
content.value: - In the 54-day training period, there were 466 job interruptions. 🕒🔄
content.type: new_line
content.type: new_line
content.raw: - About 78% of unexpected interruptions were attributed to confirmed or suspected hardware issues. Mostly GPUs! 💥🖥️
content.type: text
content.value: - About 78% of unexpected interruptions were attributed to confirmed or suspected hardware issues. Mostly GPUs! 💥🖥️
content.type: new_line
content.type: new_line
content.raw: - Saving all checkpoints is cool until you do it for the 300B+ parameters model. The bursty nature of checkpoint writes, essential for state-saving during training, periodically saturated the storage fabric, impacting performance. 📉💾
content.type: text
content.value: - Saving all checkpoints is cool until you do it for the 300B+ parameters model. The bursty nature of checkpoint writes, essential for state-saving during training, periodically saturated the storage fabric, impacting performance. 📉💾
content.type: new_line
content.type: new_line
content.raw: - With all this, effective training time—measured as the time spent on useful training over the elapsed time—was higher than 90%. ⏱️📊
content.type: text
content.value: - With all this, effective training time—measured as the time spent on useful training over the elapsed time—was higher than 90%. ⏱️📊
content.type: new_line
content.type: new_line
content.raw: I think this is the stuff that movies can be made on! 🎬🌟
content.type: text
content.value: I think this is the stuff that movies can be made on! 🎬🌟
content.type: new_line
content.type: new_line
content.raw: Paper: 
content.type: text
content.value: Paper: 
content.href: https://ai.meta.com/research/publications/the-llama-3-herd-of-models/
content.raw: https://ai.meta.com/research/publications/the-llama-3-herd-of-models/
content.type: link
rawContent: Meta Researchers: How many compute hours should we use to train Llama 3.1?
  Mr. Zuck: Yes! 🤖💪
  
  Good folks at @AIatMeta did not just release the models but also published a 92-page detailed paper 📄 on their findings and technical aspects of the models and their training process!
  
  Generally, we just gobble up these weights and forget the compute infrastructure used to train these models. 🖥️🚀
  
  
  Here are some interesting findings about the computing infrastructure of Llamas:
  
  - Llama 1 and 2 models were trained on @Meta 's AI Research SuperCluster. Llama 3 was migrated to Meta’s production clusters! 📊
  
  - That's 16,000 H100 GPUs, with each GPU featuring 700W TDP and 80GB HBM3, arranged in Meta’s Grand Teton AI server platform. 🖥️🔋
  
  - What about storing checkpoints? Used Tectonic, a distributed file system, for storage, with capacities reaching 240 PB and peak throughput of 7 TB/s. 💾📈
  
  - Meta's mad lads saved each GPU’s model state, ranging from 1 MB to 4 GB per GPU, for recovery and debugging. 🛠️🔍
  
  
  If this sounds big, well, they document the humungous challenges that come with it:
  
  - In the 54-day training period, there were 466 job interruptions. 🕒🔄
  
  - About 78% of unexpected interruptions were attributed to confirmed or suspected hardware issues. Mostly GPUs! 💥🖥️
  
  - Saving all checkpoints is cool until you do it for the 300B+ parameters model. The bursty nature of checkpoint writes, essential for state-saving during training, periodically saturated the storage fabric, impacting performance. 📉💾
  
  - With all this, effective training time—measured as the time spent on useful training over the elapsed time—was higher than 90%. ⏱️📊
  
  I think this is the stuff that movies can be made on! 🎬🌟
  
  Paper: https://ai.meta.com/research/publications/the-llama-3-herd-of-models/
reactions.reaction: 🤯
reactions.users: YaTharThShaRma999 ZachZimm palash147 Ketansomewhere cnmoro GPT007 John6666 Ruaruaruabick ajibawa-2023 adorkin Chief-Inspector louisbrulenaudet
reactions.reaction: 🤝
reactions.users: bezir
commentators._id: 65d883893a52cd9bcd8ab7cf
commentators.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65d883893a52cd9bcd8ab7cf/tRsCJlHNZo1D02kBTmfy9.jpeg
commentators.fullname: leroy Samuel Dyer
commentators.name: LeroyDyer
commentators.type: user
commentators._id: 662bf5bfe93bb73804ef9344
commentators.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/662bf5bfe93bb73804ef9344/WXYLnjjJ4SROkoveIi7If.png
commentators.fullname: Kuldeep Singh Sidhu
commentators.name: singhsidhukuldeep
commentators.type: user