Optuna hyperparameter does not sync trial/hyperparameters when using torchrun single-node, multi-process
### System Info

- `transformers` version: 4.22.0.dev0
- Platform: Linux-3.10.0-1160.76.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.9.12
- Huggingface_hub version: 0.9.1
- PyTorch version (GPU?): 1.12.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: yes

### Who can help?

@sgugger

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Note this requires running a script with `torchrun` on a single node but multiple processes. I ran this on a computer with 4 GPUs, so I used 1 node, 4 processes per node.

1. Install dependencies: `datasets`, `evaluate` for example script, `optuna` itself.
2. Unzip scripts: [scripts_to_reproduce.zip](https://github.com/huggingface/transformers/files/9329512/scripts_to_reproduce.zip).
3. Run `bug.sh`.
4. Observe the output mentions 4 trials, rather than 1 as the arguments specify. Note each reported learning rate is different.

Example of relevant output:

```
[INFO|trainer.py:1612] 2022-08-12 15:12:03,173 >> ***** Running training *****
[INFO|trainer.py:1613] 2022-08-12 15:12:03,173 >>   Num examples = 1024
[INFO|trainer.py:1614] 2022-08-12 15:12:03,173 >>   Num Epochs = 3
[INFO|trainer.py:1615] 2022-08-12 15:12:03,173 >>   Instantaneous batch size per device = 64
[INFO|trainer.py:1616] 2022-08-12 15:12:03,173 >>   Total train batch size (w. parallel, distributed & accumulation) = 256
[INFO|trainer.py:1617] 2022-08-12 15:12:03,173 >>   Gradient Accumulation steps = 1
[INFO|trainer.py:1618] 2022-08-12 15:12:03,174 >>   Total optimization steps = 12
  0%|          | 0/12 [00:00<?, ?it/s][W reducer.cpp:1251] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
[W reducer.cpp:1251] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
[W reducer.cpp:1251] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
[W reducer.cpp:1251] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
100%|██████████| 12/12 [00:07<00:00,  1.79it/s][INFO|trainer.py:1857] 2022-08-12 15:12:11,105 >>

Training completed. Do not forget to share your model on huggingface.co/models =)


{'train_runtime': 7.9321, 'train_samples_per_second': 387.289, 'train_steps_per_second': 1.513, 'train_loss': 4.8337141672770185, 'epoch': 3.0}
100%|██████████| 12/12 [00:07<00:00,  1.51it/s]
[INFO|trainer.py:729] 2022-08-12 15:12:11,143 >> The following columns in the evaluation set don't have a corresponding argument in `BertForQuestionAnswering.forward` and have been ignored: offset_mapping, example_id. If offset_mapping, example_id are not expected by `BertForQuestionAnswering.forward`,  you can safely ignore this message.
[INFO|trainer.py:2902] 2022-08-12 15:12:11,148 >> ***** Running Evaluation *****
[INFO|trainer.py:2904] 2022-08-12 15:12:11,148 >>   Num examples = 1024
[INFO|trainer.py:2907] 2022-08-12 15:12:11,148 >>   Batch size = 64
100%|██████████| 4/4 [00:00<00:00,  7.13it/s]08/12/2022 15:12:12 - INFO - utils_qa - Post-processing 1024 example predictions split into 1024 features.
100%|██████████| 1024/1024 [00:02<00:00, 392.92it/s]
08/12/2022 15:12:15 - INFO - utils_qa - Saving predictions to /tmp/debug_hpsearch_TrfOptunaBug_16693069.out/eval_predictions.json.
08/12/2022 15:12:15 - INFO - utils_qa - Saving nbest_preds to /tmp/debug_hpsearch_TrfOptunaBug_16693069.out/eval_nbest_predictions.json.
100%|██████████| 1024/1024 [00:02<00:00, 394.31it/s]
100%|██████████| 1024/1024 [00:02<00:00, 393.04it/s]
100%|██████████| 1024/1024 [00:03<00:00, 337.82it/s]
[I 2022-08-12 15:12:16,218] Trial 1 finished with value: 3.0207817191957127 and parameters: {'learning_rate': 5.82055234642441e-06}. Best is trial 1 with value: 3.0207817191957127.
[I 2022-08-12 15:12:16,471] Trial 2 finished with value: 3.0207817191957127 and parameters: {'learning_rate': 1.0083131394917086e-06}. Best is trial 1 with value: 3.0207817191957127.
100%|██████████| 4/4 [00:05<00:00,  1.30s/it]
08/12/2022 15:12:16 - INFO - __main__ - Rank 0: Metrics: {'eval_exact_match': 0.09765625, 'eval_f1': 3.0207817191957127, 'epoch': 3.0}
[I 2022-08-12 15:12:16,788] Trial 3 finished with value: 3.0207817191957127 and parameters: {'learning_rate': 5.780902974449181e-06}. Best is trial 1 with value: 3.0207817191957127.
08/12/2022 15:12:16 - INFO - __main__ - Best HP search run results: {'id': '1', 'value': 3.0207817191957127, 'all_metrics': None, 'hyperparameters': {'learning_rate': 5.82055234642441e-06}, 'value_name': 'brier_score', 'train_samples': 1024}
[I 2022-08-12 15:12:17,173] Trial 0 finished with value: 3.0207817191957127 and parameters: {'learning_rate': 1.1949546634616653e-06}. Best is trial 1 with value: 3.0207817191957127.
[INFO|modelcard.py:443] 2022-08-12 15:12:17,212 >> Dropping the following result as it does not have all the necessary fields:
{'task': {'name': 'Question Answering', 'type': 'question-answering'}, 'dataset': {'name': 'squad', 'type': 'squad', 'config': 'plain_text', 'split': 'train', 'args': 'plain_text'}}
```

### Expected behavior

I expected this setup to produce and report 1 trial of results, with each GPU-process using the same hyperparameters, in this case the same learning rate. I expected `trainer.hyperparameter_search()` would be consistent in this way with how `trainer.train()` and `trainer.evaluate()` work. Instead the script reports 4 results and each GPU-process apparently uses a different learning rate.
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.<|||||>Can confirm this still happens as of commit c126a239bcea9c68453cf86045a5177afbe2be6c.<|||||>Hi, I have enabled the HPO DDP for optuna, and it works for CPU, you could try it in the latest master.<|||||>Hi, @sywangyi. Are you referring to #19096 (merged as 6227078d0a95aed688578d37b319e969a1dcd30f)? It's not clear to me that this should fix the problem -- because each process calls optimize() on its own I'd expect each process to generate a different Optuna trial object and for that to cause problems when passing the trial object to `train()`. However, I did try rerunning the OP (reproduce) script on the latest commit (83dc6377d0107b462e5d804ffa72d069625bc36b). It crashed with `RuntimeError: DDP expects same model across all ranks, but Rank 0 has 199 params, while rank 1 has inconsistent 0 params.`. Unsure if that's related.<|||||>Hi @spigo900 ,I am referring to https://github.com/huggingface/transformers/pull/19002, only rank0 will generate the trial and pass the argument to other ranks<|||||>@sywangyi I see, that looks like it should solve the problem. I will try that today. ETA: Thank you.<|||||>@sywangyi Yes, #19168 solved the problem. Thanks again.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.