Hello @lysandre, thanks for tagging the right person. Here is my github response with a new question:

Hello @eltoto1219, thank you for the answer! I suppose it was a weird question from my part, since I was asking this to make sure that I am loading a pre-trained LXMERT model and not some random weights. Especially because I look at the  output_lxmert['cross_relationship_score']  of COCO images and captions (so not on some out of distribution images and captions), after I loaded LXMERT with the aforementioned code  lxmert_base = LxmertForPreTraining.from_pretrained("unc-nlp/lxmert-base-uncased") . It seems that  on cross-modality matching LXMERT performs with 50% accuracy  (random guessing). So I wanted to make sure that I load pre-trained weights on (4) cross-modality matching in the first place.
New question : Do you know how it can be that LXMERT randomly guesses on cross-modality matching even it was pre-trained to deliver a score (after the Softmax, of course) of smaller 0.5 if the caption is not describing the image and a score bigger 0.5 it the caption and the image match?