LXMERT cross_modality_matching logits order in code vs. documentation
Issue #7266 evolved into an entirely different issue described below.

# Possible documentation error
Independently of considering #8333 or not, it is clear that the results of the cross-modality matching in LXMERT (for example in [this demo](https://colab.research.google.com/drive/18TyuMfZYlgQ_nXo-tr8LCnzUaoX0KS-h?usp=sharing)) are better (below random chance vs. above random chance) when the first logit of the `output_lxmert['cross_relationship_score']` *– (torch.FloatTensor of shape (batch_size, 2))* represents *a mismatch* rather than *a match*.

This **contradicts the [documentation](https://huggingface.co/transformers/model_doc/lxmert.html)** which in my understanding assigns the first logit to `is_match` (True):
> cross_relationship_score – (torch.FloatTensor of shape (batch_size, 2)): Prediction scores of the textual matching objective (classification) head (scores of True/False continuation before SoftMax).

@LysandreJik, @eltoto1219: Is the documentation right or wrong? How was the model trained and which logit was intended to predict the match and which one the mismatch?

I would be very happy to have a clear answer about the order of the logits in training and code and its correspondence to the documentation. I know how easy it is to flip the logits around and choose the one delivering better results on the data at hand. But in my understanding of scientific conduct, I can not just choose this by wishful thinking and ignoring the documentation: there might be other things going on exhibiting this behavior.

Thank you in advance!
This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.