Understanding run_glue in distributed mode
## ❓ Questions & Help

In my own project I am building on top of `transformers` and I'd like to take advantage of DDP. For inspiration I've been looking at how different libraries implement that, as well as how `transformers` handles it. In particular, I've been looking at [`run_glue`](https://github.com/huggingface/transformers/blob/master/examples/run_glue.py). 

I guess the main issue that I have, is that I don't quite understand how the forward/backward is synced across processes. **Does it mean that before each forward and backward pass the processes are synced, and that the backward pass averages over all gradients for all processes?**

In addition, I am a bit confused about how `run_glue` presents its results. It seems that the logger is only active for the first process

https://github.com/huggingface/transformers/blob/be916cb3fb4579e278ceeaec11a6524662797d7f/examples/run_glue.py#L453-L455

**Does this mean that `tr_loss` in the following fragment is a generalization**, i.e. it's the training loss of only the first process and NOT an average over all processes. It's to give users _some_ idea of the results, but it is not a factual representation of the full training loss, since it only represents the loss of one process?

https://github.com/huggingface/transformers/blob/be916cb3fb4579e278ceeaec11a6524662797d7f/examples/run_glue.py#L493

I noticed that you do the testing with only one device:

https://github.com/huggingface/transformers/blob/be916cb3fb4579e278ceeaec11a6524662797d7f/examples/run_glue.py#L520

**What is the reasoning behind this?** Exactly what I suggested before, namely that you do not get all results back easily when running in distributed mode? In a setting with training, validating, and testing, would you recommend that only training is done distributed, and testing and validating on one device?

Thanks in advance for your time. 
To understand how DDP synchronize across processes, you can read:
- the official doc for DDP: https://pytorch.org/docs/stable/nn.html?highlight=distributed%20data%20parallel#torch.nn.parallel.DistributedDataParallel
- this detailed blog post I did a few months ago: https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255

Yes, `tr_loss` is only for the first device, this is just an information to follow the training. To get the average loss over all devices, you can add a step of synchronization of losses. It's pretty simple to do, here is an example I wrote this summer for our NAACL tutorial: https://github.com/huggingface/naacl_transfer_learning_tutorial/blob/master/utils.py#L53-L59

Doing evaluation on only one device is better since the total metrics are not always averages of metrics on each node. For instance F1 uses a non linear fonction of the samples and as thus can't be computed in a distributed setting just by averaging F1 on all devices.

As a general note: these points can be improved (and we are aware of that and how to do it). Not including more complexities in examples like `run_glue` is a conscious decision to keep the examples simple to understand (I already think distributed training makes them a bit complex but honestly we can't really do without it on our big models). <|||||>I did indeed read through the official documentation as well as your blog post and other (official and non-official) tutorials. However, many were outdated or didn't go into details about setting up the actual training loop and all the intricacies that are involved. It often seems to be explained as "wow, you can do so much cool stuff with this because it is _D I S T R I B U T E D_", but then details are absolutely lacking and often you're left to figure out how things work by going through source code. I did set up initialisation and training, but then I wasn't sure how to deal with the gathered loss and validating/testing.

That being said, thank you very much for your response, this is very helpful! I had also never heard about ignite. Great. Now I'll have to refactor all my code! (Kidding, even though it might be worth my time to look into it.)

Closing this. Thanks again for the information.<|||||>@thomwolf  As a small update: I have decided that I will still use distributed testing and validating. However, the final metric (per epoch for validating) will be calculated on the collected results of all processes. In other words, after a validating iteration where the loss for all steps is saved as a tensor, gather ALL losses, and then average those. That seems like a good compromise to me. Something like this.

```python
def gather_cat(x):
    gather = [torch.empty_like(x) for _ in range(dist.get_world_size())]
    dist.all_gather(gather, x)
    return torch.cat(gather)

# ...
# distributed:
loss = model(...)
loss = gather_cat(loss)
avg_loss = torch.mean(loss)
# ... track avg_loss only in the first process for instance
```

II think this would be especially useful when you have a lot of data and you want your validation and test to run smoothly as well. If anything's wrong with approach, I'd be happy to hear about it.