@Rajaram1996 ,
I just checked your model, and it seems it’s using

'classifier.dense.bias', 'classifier.out_proj.weight', 'classifier.out_proj.bias', 'classifier.dense.weight'

instead of

'classifier.bias', 'classifier.weight', 'projector.weight', 'projector.bias'

when loading (check the warings), which leads to random outputs no matter the input.
Try renaming the tensors appropriately for it to work.
So most likely, the pipeline is not broken but it just seems like the weights are incorrect so the head gives garbage output.