 clip loss
https://github.com/mlfoundations/open_clip/blob/37b729bc69068daa7e860fb7dbcf1ef1d03a4185/src/open_clip/loss.py#L49


In the implementation of open_clip, logits distributed across multiple gpus are gathered for calculating loss. However, I cannot find the code related to this feature in this repository. I think more negative samples are very important for contrastive learning. @younesbelkada @ydshieh 
Hi @hljjjmssyh The loss computation management is in the `Trainer` class, see

https://github.com/huggingface/transformers/blob/da005253b82395b6097623bcee44b819bfe72b87/src/transformers/trainer.py#L2649-L2650<|||||>That's only for models wrapped in DataParallel @ydshieh 

@hljjjmssyh We don't include code requiring torch.distributed as it then fails when the script is used on one GPU. However we could use the Accelerate library to have something that works in both situation. If you want to explore this and open a PR, I'll be happy to review!<|||||>I think I’m missing something, it looks like this could be done for CLIP today with accelerate’s implementation in `examples/pytorch/image-classification/run_image_classification_no_trainer.py` running it with the appropriate args? Or maybe accelerate would nonetheless be a welcome addition somewhere else for the above mentioned purpose? 

It also looks [here](https://github.com/huggingface/transformers/blob/5990743fddb4780b15b8af2ed7ab55145ab40455/src/transformers/trainer.py#L1386-L1388) like the model would in fact be wrapped in DataParallel when training on multiple gpus.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.