text: Title: Loss is constant
  Body: I'm using CLIP to train on my custom dataset with the following params: 
  
  Dataset size : 50k image-text pairs
  Batch size : 128
  Image Size : 224
  Gpus : 1 
  Epochs : 500
  
  It's been running for a while now, I'm on my 15th epoch, and the loss hasn't changed at all. It isn't a constant number, but its constantly at 4.8xxx. Should I be concerned? I'm not sure why this is happening. 
  
  ![image](https://user-images.githubusercontent.com/28048963/133154185-01bdd63f-b3bc-460b-a583-21f5f9616a02.png)
  