DDP performing slightly worse in terms of loss and metrics than DP
Hi,

I am running Transformers 4.3.2 and am testing DistributedDataParallel vs DataParallel. The Huggingface Trainer in both instances have been untouched and slight modifications were made to run_clm.py to fit my specific use case.
This is using GPT-2 model trained from scratch.

I am consistently seeing much faster, but slightly worse results for DistributedDataParallel and was wondering if there are any possible reasons this could occur. Convergence still occurs but the evaluation loss is often slightly worse and alternative metrics we use are also slightly worse as well. The worse results are consistent through many runs and when keeping hyper-parameters the same (learning rate, # gpus, batch size, etc.)

The DistributedDataParallel code is being launch using "-m torch.distributed.launch" as has been recommended.

I load my data into a Datasets object from the huggingface/Datasets library.

Things I have checked:
1. Padding, padding is being treated the same in both models
2. gradient averaging, some information online suggested the gradients may be summed when using DataParallel vs averaged when using Distributed but I found this was not the case looking at the code.

Apologies if this too abstract a question, but I felt I would raise it as I have not seen any discussion of possible regression when switching to DDP.

Thanks!
Hi! This thread https://github.com/huggingface/transformers/issues/10223 might be useful, it sheds light over the issue you mention.<|||||>Hi thank you for linking, it is actually worse in terms of loss, not speed. DDP is significantly faster than DP, which I believe it is supposed to be.<|||||>Oh, sorry about that, I read a bit too fast. Pinging @sgugger who might know what's up.<|||||>I have not experienced this, so no idea of what might be causing the issue. It's also a bit vague and with no reproducer, so very hard to investigate further.<|||||>Thank you and I know it is very vague and will work on something that is reproducible.

 Do you have any thoughts as to why this could occur? From my knowledge I would think these would be identical as the gradients from all GPUs are being averaged in both and there is no batch-norm which would be implemented differently.

I'd appreciate any thoughts and will work to find something more reproducible.

Thank you for the help.<|||||>This might be coming from the distributed sampler setting the random seed at each epoch, so trying to set it the same way with a run in DP might help (a bad seed could explain a small difference). It might also interfere with the random masking somehow, but that's far-fetched.

I also have no idea of how much difference you observed in the loss, one thing to try for debugging would also to double check the evaluation loss is the same in DP/DDP for a given model. There might a bug in the way one of them is computed.<|||||>The DistributedSampler looks it sets the seed in a way that would not effect the seed, its first lines of the __iter__ method are:
```
    def __iter__(self) -> Iterator[T_co]:
        if self.shuffle:
            # deterministically shuffle based on epoch and seed
            g = torch.Generator()
            g.manual_seed(self.seed + self.epoch)
            indices = torch.randperm(len(self.dataset), generator=g).tolist()  # type: ignore
```
In that case the seed is set in both cases to the default huggingface seed of 42.

For our task we don't have random masking, as we are just doing CausalLM.

The difference in eval loss for one run is around the best score being .17 for DP and .175 for DDP. So slight but consistent across many runs. There are additional task specific metrics in which it performs significantly worse, which is the main reason I are trying to fix the issue. The eval_loss is both using the default GPT2WithLMHead, no change to the way it calculates loss.<|||||>> The DistributedSampler looks it sets the seed in a way that would not effect the seed

Correct if you use the most recent version of PyTorch, but it was not always the case.<|||||>Ah, great, thanks for the clarification. Yes, using pytorch 1.7.1 so that shouldn't be an issue<|||||>If the following suggestion might help, one way I approach such problems is logging the loss on every step with its count and noticing when numbers start to diverge - and checking if there is a significant event happening around that time. 

For example with DeepSpeed's fp16 default dynamic loss scale enabled odd things were starting to happen around step 20 - until I learned that the scheduler was getting skipped until the loss scale value was small enough and only then it'd kick in - which typically happened around step 20. I'm certain this is not relevant to your situation, but I'm just giving an example. <|||||>Thank you for the suggestion! I will try this<|||||>Hi, after back-tracking on this we found that the padding issue hadn't been fully investigated and that did turn out to be the issue. I was able to use pytorch's all_reduce to communicate the max sequence length across gpus and pad to that amount. I'll paste the code here for the prepare_inputs method in case it helps anyone else bridge the gap between DDP and DP, its written for a bs per gpu of 1, would need a tweak to the torch.cat for larger batch sizes.
```
    def _prepare_inputs(self, inputs: Dict[str, Union[torch.Tensor, Any]]) -> Dict[str, Union[torch.Tensor, Any]]:
        world_size = dist.get_world_size()
        self.gpu_group = dist.new_group(list(range(world_size)))
        self.world_size = world_size
        max_len = torch.tensor(max([len(i) for i in inputs['input_ids']]))
        max_len = max_len.to(self.args.device)
        dist.all_reduce(max_len, op=dist.ReduceOp.MAX, group=self.gpu_group)
        max_len = max_len.cpu()
        for k, v in inputs.items():
            if isinstance(v, torch.Tensor):
                if k in ['input_ids', 'labels']:
                    v = torch.cat((v, torch.ones(1, max(0, max_len - len(v[0])))*self.tokenizer.pad_token_id), axis=1).long()
                    #print(f'{k}-{self.args.local_rank}-{self.state.global_step}: {len(v[0])}')
                elif k in ['attention_mask']:
                    v = torch.cat((v, torch.ones(1, max(0, max_len - len(v[0])))), axis=1).long()
                inputs[k] = v.to(self.args.device)

        if self.args.past_index >= 0 and self._past is not None:
            inputs["mems"] = self._past

        return inputs
```
I'd suggest that to be a first try for people trying to debug. We found that padding materially improved our results and not by just reducing the loss. One can also use all_reduce in place of the DDP model wrapper as in here: https://pytorch.org/tutorials/intermediate/dist_tuto.html. It may be helpful for someone else investigating DDP.
Thanks for the suggestions and apologies for misleading on the original post.<|||||>Interesting, thanks for the info and the code!