The Universal Checkpointing paper is out! 
text
The Universal Checkpointing paper is out! 
https://arxiv.org/abs/2406.18820
https://arxiv.org/abs/2406.18820
link
new_line
new_line
If you remember the Bigscience BLOOM-176B training, Tunji Ruwase and I co-invented this technology for Megatron-Deepspeed in order to enable to quickly scale up and down node topology while continuing training.
text
If you remember the Bigscience BLOOM-176B training, Tunji Ruwase and I co-invented this technology for Megatron-Deepspeed in order to enable to quickly scale up and down node topology while continuing training.
new_line
new_line
Since then the DeepSpeed team continued improving on that and it has now been fully integrated into Deepspeed.
text
Since then the DeepSpeed team continued improving on that and it has now been fully integrated into Deepspeed.
new_line
new_line
The blog post is here: 
text
The blog post is here: 
https://github.com/microsoft/DeepSpeed/blob/master/blogs/deepspeed-ucp/README.md
https://github.com/microsoft/DeepSpeed/blob/master/blogs/deepspeed-ucp/README.md
link
The Universal Checkpointing paper is out! https://arxiv.org/abs/2406.18820

If you remember the Bigscience BLOOM-176B training, Tunji Ruwase and I co-invented this technology for Megatron-Deepspeed in order to enable to quickly scale up and down node topology while continuing training.

Since then the DeepSpeed team continued improving on that and it has now been fully integrated into Deepspeed.

The blog post is here: https://github.com/microsoft/DeepSpeed/blob/master/blogs/deepspeed-ucp/README.md
👀
alielfilali01 John6666