[Update]:  Trainer works but I see one challenge in the impl


Apparently return_loss parameter in forward() needs to be set to True for the model to return loss. So cloned the source and locally removed the if the condition (shown below) that checks for the flag to be True to test. Trainer works !!


Challenge is not sure how to set  return_loss as a part of my data loader (Dataset) so it goes as True for training.


        loss = None
        if return_loss:
            loss = clip_loss(logits_per_text)


[Original response]:
Thanks this screenshot was helpful. I will check what does the model return.
But I am simply reusing HF CLIPModel implementation (not a custom impl) and as per the code it returns a CLIPOutput (below) which is a concrete implementation of ModelOutput like you rightly pointed, in here loss is cliploss which internally uses similarity based contrastive loss.
 return CLIPOutput(
            loss=loss,
            logits_per_image=logits_per_image,
            logits_per_text=logits_per_text,
            text_embeds=text_embeds,
            image_embeds=image_embeds,
            text_model_output=text_outputs,
            vision_model_output=vision_outputs,
        )