Meta Researchers: How many compute hours should we use to train Llama 3.1?
text
Meta Researchers: How many compute hours should we use to train Llama 3.1?
new_line
Mr. Zuck: Yes! 🤖💪
text
Mr. Zuck: Yes! 🤖💪
new_line
new_line
Good folks at 
text
Good folks at 
@AIatMeta
mention
AIatMeta
 did not just release the models but also published a 92-page detailed paper 📄 on their findings and technical aspects of the models and their training process!
text
 did not just release the models but also published a 92-page detailed paper 📄 on their findings and technical aspects of the models and their training process!
new_line
new_line
Generally, we just gobble up these weights and forget the compute infrastructure used to train these models. 🖥️🚀
text
Generally, we just gobble up these weights and forget the compute infrastructure used to train these models. 🖥️🚀
new_line
new_line
new_line
Here are some interesting findings about the computing infrastructure of Llamas:
text
Here are some interesting findings about the computing infrastructure of Llamas:
new_line
new_line
- Llama 1 and 2 models were trained on 
text
- Llama 1 and 2 models were trained on 
@Meta
mention
Meta
 's AI Research SuperCluster. Llama 3 was migrated to Meta’s production clusters! 📊
text
 's AI Research SuperCluster. Llama 3 was migrated to Meta’s production clusters! 📊
new_line
new_line
- That's 16,000 H100 GPUs, with each GPU featuring 700W TDP and 80GB HBM3, arranged in Meta’s Grand Teton AI server platform. 🖥️🔋
text
- That's 16,000 H100 GPUs, with each GPU featuring 700W TDP and 80GB HBM3, arranged in Meta’s Grand Teton AI server platform. 🖥️🔋
new_line
new_line
- What about storing checkpoints? Used Tectonic, a distributed file system, for storage, with capacities reaching 240 PB and peak throughput of 7 TB/s. 💾📈
text
- What about storing checkpoints? Used Tectonic, a distributed file system, for storage, with capacities reaching 240 PB and peak throughput of 7 TB/s. 💾📈
new_line
new_line
- Meta's mad lads saved each GPU’s model state, ranging from 1 MB to 4 GB per GPU, for recovery and debugging. 🛠️🔍
text
- Meta's mad lads saved each GPU’s model state, ranging from 1 MB to 4 GB per GPU, for recovery and debugging. 🛠️🔍
new_line
new_line
new_line
If this sounds big, well, they document the humungous challenges that come with it:
text
If this sounds big, well, they document the humungous challenges that come with it:
new_line
new_line
- In the 54-day training period, there were 466 job interruptions. 🕒🔄
text
- In the 54-day training period, there were 466 job interruptions. 🕒🔄
new_line
new_line
- About 78% of unexpected interruptions were attributed to confirmed or suspected hardware issues. Mostly GPUs! 💥🖥️
text
- About 78% of unexpected interruptions were attributed to confirmed or suspected hardware issues. Mostly GPUs! 💥🖥️
new_line
new_line
- Saving all checkpoints is cool until you do it for the 300B+ parameters model. The bursty nature of checkpoint writes, essential for state-saving during training, periodically saturated the storage fabric, impacting performance. 📉💾
text
- Saving all checkpoints is cool until you do it for the 300B+ parameters model. The bursty nature of checkpoint writes, essential for state-saving during training, periodically saturated the storage fabric, impacting performance. 📉💾
new_line
new_line
- With all this, effective training time—measured as the time spent on useful training over the elapsed time—was higher than 90%. ⏱️📊
text
- With all this, effective training time—measured as the time spent on useful training over the elapsed time—was higher than 90%. ⏱️📊
new_line
new_line
I think this is the stuff that movies can be made on! 🎬🌟
text
I think this is the stuff that movies can be made on! 🎬🌟
new_line
new_line
Paper: 
text
Paper: 
https://ai.meta.com/research/publications/the-llama-3-herd-of-models/
https://ai.meta.com/research/publications/the-llama-3-herd-of-models/
link
Meta Researchers: How many compute hours should we use to train Llama 3.1?
Mr. Zuck: Yes! 🤖💪

Good folks at @AIatMeta did not just release the models but also published a 92-page detailed paper 📄 on their findings and technical aspects of the models and their training process!

Generally, we just gobble up these weights and forget the compute infrastructure used to train these models. 🖥️🚀


Here are some interesting findings about the computing infrastructure of Llamas:

- Llama 1 and 2 models were trained on @Meta 's AI Research SuperCluster. Llama 3 was migrated to Meta’s production clusters! 📊

- That's 16,000 H100 GPUs, with each GPU featuring 700W TDP and 80GB HBM3, arranged in Meta’s Grand Teton AI server platform. 🖥️🔋

- What about storing checkpoints? Used Tectonic, a distributed file system, for storage, with capacities reaching 240 PB and peak throughput of 7 TB/s. 💾📈

- Meta's mad lads saved each GPU’s model state, ranging from 1 MB to 4 GB per GPU, for recovery and debugging. 🛠️🔍


If this sounds big, well, they document the humungous challenges that come with it:

- In the 54-day training period, there were 466 job interruptions. 🕒🔄

- About 78% of unexpected interruptions were attributed to confirmed or suspected hardware issues. Mostly GPUs! 💥🖥️

- Saving all checkpoints is cool until you do it for the 300B+ parameters model. The bursty nature of checkpoint writes, essential for state-saving during training, periodically saturated the storage fabric, impacting performance. 📉💾

- With all this, effective training time—measured as the time spent on useful training over the elapsed time—was higher than 90%. ⏱️📊

I think this is the stuff that movies can be made on! 🎬🌟

Paper: https://ai.meta.com/research/publications/the-llama-3-herd-of-models/
🤯
YaTharThShaRma999 ZachZimm palash147 Ketansomewhere cnmoro GPT007 John6666 Ruaruaruabick ajibawa-2023 adorkin Chief-Inspector louisbrulenaudet
🤝
bezir
65d883893a52cd9bcd8ab7cf
https://cdn-avatars.huggingface.co/v1/production/uploads/65d883893a52cd9bcd8ab7cf/tRsCJlHNZo1D02kBTmfy9.jpeg
leroy Samuel Dyer
LeroyDyer
user
662bf5bfe93bb73804ef9344
https://cdn-avatars.huggingface.co/v1/production/uploads/662bf5bfe93bb73804ef9344/WXYLnjjJ4SROkoveIi7If.png
Kuldeep Singh Sidhu
singhsidhukuldeep
user