Use latest stable PyTorch/DeepSpeed for Push & Scheduled CI
# What does this PR do?

(**As there are a few PRs waiting for review, feel free to postpone the review for this PR if things get confusing at this moment**)

Currently:
- scheduled CI uses latest stable PyTorch (OK) + nightly DeepSpeed (Not OK)
- push CI uses PyTorch 1.9 (Not OK)  + latest stable DeepSpeed (OK)

This PR fix it by using latest stable PyTorch + DeepSpeed, for both push/scheduled CIs

(**Let me run a dummy test before merge** 🙏 )
_The documentation is not available anymore as the PR was closed or merged._<|||||>@stas00 

Regarding `transformers-pytorch-gpu`:

https://github.com/huggingface/transformers/blob/7e46ec71371b5e705522821e741d7c0dac910859/docker/transformers-pytorch-gpu/Dockerfile#L17

Here we only upgrade `torch`, but not `torchvision` and `torcuaudio`. Those two are installed in a previous step

https://github.com/huggingface/transformers/blob/7e46ec71371b5e705522821e741d7c0dac910859/docker/transformers-pytorch-gpu/Dockerfile#L12

I am not sure if we need to upgrade all 3 modules at the same time. PyTorch installation instructions always install these 3 at the same time, so I guess yes ??
<|||||>Yes, usually it's the easiest to always handle all 3 packages as a single package to avoid incompatibility conflicts down the road. 

I first tried to "optimize" and only install the other 2 when it was needed, but later I switched to always installing the 3 together. the other 2 are tiny compared to the main package.<|||||>> Yes, usually it's the easiest to always handle all 3 packages as a single package to avoid incompatibility conflicts down the road.
> 
> I first tried to "optimize" and only install the other 2 when it was needed, but later I switched to always installing the 3 together. the other 2 are tiny compared to the main package.

OK, I will try to get it done. A bit not easy as here we have using an argument `$PYTORCH` so we can specify torch version.
(although we only use the default value to get the latest stable version for now.)<|||||>[Just FYI] an update:

I am running the tests before merge. So far only a subset of tests is run. I got some issues

PyTorch pipelines (single-gpu)]
https://github.com/huggingface/transformers/runs/6728867682?check_suite_focus=true

Torch CUDA test (multi GPUs)
https://github.com/huggingface/transformers/runs/6728868779?check_suite_focus=true

I will try to re-run them later, also will wait the scheduled CIs during this weekend in order to compare.
<|||||>Feel free to also spawn dummy machines to help you out if it helps getting it merged quicker<|||||>I am going to merge now.

----------------------------------

@stas00 

- `intel_extension_for_pytorch` is added
  - the version still uses the approach so far `$(python3 -c "from torch import version; print(version.__version__.split('+')[0])")` 
  - I have to remove `pip uninstall -y` that was in your original patch. The whole line wasn't working.

For the versions, let's discuss in #17586

----------------------------------
@stas00

I observed with the latest stable PyTorch/DeepSpeed, `test_can_resume_training_normal_zero2_fp16` takes quite long to run the first time: (On push CI, it will timeout)

First Run
```
63.46s call     tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_can_resume_training_normal_zero2_fp16
10.97s call     tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_can_resume_training_normal_zero3_fp16
```
Second Run
```
18.17s call     tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_can_resume_training_normal_zero2_fp16
11.01s call     tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_can_resume_training_normal_zero3_fp16
```

With previous setting (PyTorch 1.9 + DeepSpeed Recompiled)
```
6.16s call     tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_can_resume_training_normal_zero2_fp16
2.83s call     tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_can_resume_training_normal_zero3_fp16
```
<|||||>> I observed with the latest stable PyTorch/DeepSpeed, test_can_resume_training_normal_zero2_fp16 takes quite long to run the first time: (On push CI, it will timeout)

The very first deepspeed test using deepspeed JIT install will have the overhead of building deepspeed, which takes about 1min - depending on the hardware.

This doesn't happen if deepspeed was prebuilt before tests were run.

Is it possible that this test happens to be the first one to run?<|||||>@stas00 

(This is for the stable release of `DeepSpeed` + `PyTorch`)

Here is the job run page
https://github.com/huggingface/transformers/runs/6761224960?check_suite_focus=true

The tests are ran in the following order 

```
# The following 3 are OK
tests/deepspeed/test_deepspeed.py::CoreIntegrationDeepSpeed::test_init_zero3_fp16 
tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_can_resume_training_errors_zero2_fp16 
tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_can_resume_training_errors_zero3_fp16 

# This one timed out
tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_can_resume_training_normal_zero2_fp16

# This one is OK
tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_can_resume_training_normal_zero3_fp16 

tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_config_object 
...
```

Dumb question: Is stable release of `DeepSpeed` == pre-built ?<|||||>You can see that it was indeed building deepspeed during that test's run, see: https://github.com/huggingface/transformers/runs/6761224960?check_suite_focus=true#step:6:334

so need to either have a longer timeout or always prebuild deepspeed.

> Dumb question: Is stable release of DeepSpeed == pre-built ?

Hope the following makes the whole situation loud and clear:

### What is being built:

* stable release install: `pip install deepspeed==0.6.5`
* bleed/master install: `pip install git+https://github.com/microsoft/DeepSpeed` (or `git clone ...; pip install -e .`)

### How is it being built:

* `pip install deepspeed` JIT build - this will build deepspeed the first time it's used. pip just installs the source files here.
* `DS_BUILD_CPU_ADAM=1 DS_BUILD_AIO=1  DS_BUILD_UTILS=1 pip install -e . --global-option="build_ext" --global-option="-j8"` - this is the prebuiding - so that the first time it's used it's already ready to use<|||||>Thank you @stas00 , thankfully I get better understanding of the terminology now!

I will pre-build `DeepSpeed` so it will be indeed ready to be speedy!