request about deepspeed tutorial 
Dear @stas00 

You have created this great tutorial here that without you it was really very hard near impossible to be able to train these large models, thank you so much 

https://github.com/huggingface/transformers/issues/8771

Do you mind updating your comment, including how numbers would change if you add distributed training and deepspeed and how the command should be in this case? so to further speedup deepspeed on multiple GPUs 

I also get this message with deepspeed with transformer 4.3.3 running the tutorial:
[deepscale] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 131072.0, reducing to 65536.0

could you tell me if this shows any issue in training? sorry I am very unfamiliar with deepspeed.

One more question, in tutorial you did not use --fp16, could you add a few comments if we can use it with deepspeed?
this is such a great toturial and would be great if we could have all info in one place.

thank you so much for the hard work and a lot of help


@dorost1234, thank you for the kind words. I'm glad to hear it was useful.

In general to answer the bulk of your questions - you will find the full documentation here:
https://huggingface.co/transformers/master/main_classes/trainer.html#deepspeed

Please let me know if you still have any question after reading it.

> [deepscale] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 131072.0, reducing to 65536.0

I shall document this. This is just a warning that DeepSpeed prints when it delays the optimizer stepping due to fp16 dynamic scaling. I definitely want to document that since it's alarming and hard to understand. and how to get the optimizer kick on the first step.

Also I started working on a notebook, which you can try - should work on  jupyter or colab:
https://github.com/stas00/porting/blob/master/transformers/deepspeed/DeepSpeed_on_colab_CLI.ipynb
Please let me know what you'd like to be added to it.
<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.