Difference between `logit_scale` initialisation in Transformers CLIP and the original OpenAI implementation.
I tried another training code based on the OpenAI'CLIP version: I found a difference at logit_scale between them. Does it mean temperature parameter? Is it the reason for loss rising?
huggingface transformers' CLIP:

```
self.logit_scale = nn.Parameter(torch.ones([]))
```
OpenAI CLIP:

```
self.logit_scale = nn.Parameter(torch.ones([]) * np.log(1 / 0.07))
```
Hi @lycfight 

Please provide a proper title when opening an issue, I've updated the title and issue description a bit.

> I found a difference at logit_scale between them. Does it mean temperature parameter? Is it the reason for loss rising?

Great catch!  Yes `logits_scale`  is also called `temperature`. And indeed not initializing the `logits_scale` to the right value could cause issues during training.

Also, training CLIP is a bit complicated, for contrastive learning try to use as big a batch size as possible. Initializing `logits_scale` also matters a lot. You could find some info in this [report](https://huggingface.co/spaces/clip-italian/clip-italian-demo) as well (it's CLIP for Italian language), this [issue](https://github.com/openai/CLIP/issues/83)  might also help.<|||||>> Hi @lycfight
> 
> Please provide a proper title when opening an issue, I've updated the title and issue description a bit.
> 
> > I found a difference at logit_scale between them. Does it mean temperature parameter? Is it the reason for loss rising?
> 
> Great catch! Yes `logits_scale` is also called `temperature`. And indeed not initializing the `logits_scale` to the right value could cause issues during training.
> 
> Also, training CLIP is a bit complicated, for contrastive learning try to use as big a batch size as possible. Initializing `logits_scale` also matters a lot. You could find some info in this [report](https://huggingface.co/spaces/clip-italian/clip-italian-demo) as well (it's CLIP for Italian language), this [issue](https://github.com/openai/CLIP/issues/83) might also help.

Furthermore, the loss rising happens when I fine-tuned CLIP pretrained model in COCO dataset: 
`model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")`
Then, in order to find the difference, I also try to train CLIP from scratch, and the loss descents normally:
`from transformers import CLIPConfig, CLIPModel, CLIPProcessor
    config = CLIPConfig()
    model = CLIPModel(config)`
That confused me extremely. 