FLAVA tensors are masked twice, forward pass fails
https://github.com/huggingface/transformers/blob/cf11493dce0a1d22446efe0d6c4ade02fd928e50/src/transformers/models/flava/modeling_flava.py#L1965

The line above is the second time this tensor is masked if the previous ITM logic happens (line 1950), resulting e.g. `IndexError: The shape of the mask [14] at index 0 does not match the shape of the indexed tensor [13, 196, 768] at index 0`

The fix could be something like `if pos_mask is not None and sequence_for_image.size(0) == pos_mask.size(0)` on line 1964.
A similar thing happens at line 1969 because of line 1956 (`mim_labels` changes shape):
```python
    mim_labels[bool_masked_pos.ne(True)] = self.ce_ignore_index
IndexError: The shape of the mask [14, 196] at index 0 does not match the shape of the indexed tensor [13, 196] at index 0
```

The fix could be adding `bool_masked_pos = bool_masked_pos[pos_mask]` between lines 1969 and 1968.<|||||>Same for line 1988, fix could be `if pos_mask is not None and sequence_for_text.size(0) == pos_mask.size(0):`<|||||>Tried the fixes and FLAVA runs. I wonder how no one else noticed 🤔<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.