FLAVA: module 'torch.distributed.nn.functional' has no attribute 'all_gather_with_backprop'
https://github.com/huggingface/transformers/blob/a0e733283930bdb9ae2b1afdc53ec5f2daefb033/src/transformers/models/flava/modeling_flava.py#L1696

The following error is thrown when running FLAVA with PyTorch 2.0 and `global_backprop_contrastive=True`: `AttributeError: module 'torch.distributed.nn.functional' has no attribute 'all_gather_with_backprop'`. As far as I know, this attribute never existed in PyTorch.

 The bug might have to do with the fact that `all_gather` is renamed to `all_gather_with_backprop` in [facebookresearch/multimodal](https://github.com/facebookresearch/multimodal) and this could have been copied over: https://github.com/facebookresearch/multimodal/blob/c6f6e44ec6e0addfdf01695db860a6febeb2d88b/torchmultimodal/utils/distributed.py#L12

A rename to `all_gather` should fix this I think.
cc @younesbelkada  and @amyeroberts <|||||>Hi @amariucaitheodor 
This sounds like being the correct fix in my opinion. I can't find that method either on the PT documentation, and I guess we never flagged that issue as not many users have run the model in a distributed mode..
Would you mind opening a PR for that? If you can't, happy to do it! 