content.raw: The Universal Checkpointing paper is out! 
content.type: text
content.value: The Universal Checkpointing paper is out! 
content.href: https://arxiv.org/abs/2406.18820
content.raw: https://arxiv.org/abs/2406.18820
content.type: link
content.type: new_line
content.type: new_line
content.raw: If you remember the Bigscience BLOOM-176B training, Tunji Ruwase and I co-invented this technology for Megatron-Deepspeed in order to enable to quickly scale up and down node topology while continuing training.
content.type: text
content.value: If you remember the Bigscience BLOOM-176B training, Tunji Ruwase and I co-invented this technology for Megatron-Deepspeed in order to enable to quickly scale up and down node topology while continuing training.
content.type: new_line
content.type: new_line
content.raw: Since then the DeepSpeed team continued improving on that and it has now been fully integrated into Deepspeed.
content.type: text
content.value: Since then the DeepSpeed team continued improving on that and it has now been fully integrated into Deepspeed.
content.type: new_line
content.type: new_line
content.raw: The blog post is here: 
content.type: text
content.value: The blog post is here: 
content.href: https://github.com/microsoft/DeepSpeed/blob/master/blogs/deepspeed-ucp/README.md
content.raw: https://github.com/microsoft/DeepSpeed/blob/master/blogs/deepspeed-ucp/README.md
content.type: link
rawContent: The Universal Checkpointing paper is out! https://arxiv.org/abs/2406.18820
  
  If you remember the Bigscience BLOOM-176B training, Tunji Ruwase and I co-invented this technology for Megatron-Deepspeed in order to enable to quickly scale up and down node topology while continuing training.
  
  Since then the DeepSpeed team continued improving on that and it has now been fully integrated into Deepspeed.
  
  The blog post is here: https://github.com/microsoft/DeepSpeed/blob/master/blogs/deepspeed-ucp/README.md
reactions.reaction: 👀
reactions.users: alielfilali01 John6666