TypeError: add got incompatible shapes for broadcasting: (512, 50, 1024), (1, 145, 1024).
### System Info

transformers version: 4.27.4
Platform: Linux-5.15.0-52-generic-x86_64-with-glibc2.29
Python version: 3.8.10
Huggingface_hub version: 0.13.4
PyTorch version (GPU?): 1.9.0+cpu (False)
Tensorflow version (GPU?): 2.9.1 (True)
Flax version (CPU?/GPU?/TPU?): 0.6.8 (cpu)
Jax version: 0.4.8
JaxLib version: 0.4.7
Using GPU in script?:
Using distributed or parallel set-up in script?:

### Who can help?

@sanchit-gandhi

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I am running the Imagenet evaluation script [here](https://github.com/clip-italian/clip-italian/blob/master/evaluation/CLIP_italian_ImageNet_Zero_Shot_Evaluation_.ipynb) to evaluate a version of clip that uses google/vit-large-patch32-384 provided [here](https://huggingface.co/LinaAlhuri/clip-vit-large-patch32). However, around the prediction cell bellow
```
top_ns = [1, 5, 10, 100]
acc_counters = [0. for _ in top_ns]
n = 0.

for i, (images, target) in enumerate(tqdm(loader)):
    images = images
    target = target.numpy()
    # predict
    image_features = image_model(images)
    image_features = image_features / np.linalg.norm(image_features, axis=-1, keepdims=True)
    logits = 100. * image_features @ zeroshot_weights

    # measure accuracy
    accs = accuracy(logits, target, topk=top_ns)
    for j in range(len(top_ns)):
        acc_counters[j] += accs[j]
    n += images.shape[0]

tops = {f'top{top_ns[i]}': acc_counters[i] / n * 100 for i in range(len(top_ns))}
print(tops)
```
I am getting the below error
```
---------------------------------------------------------------------------
UnfilteredStackTrace                      Traceback (most recent call last)
 in 
      8     # predict
----> 9     image_features = image_model(images)
     10     image_features = image_features / np.linalg.norm(image_features, axis=-1, keepdims=True)

 in (images)
     24     language_model = lambda queries: np.asarray(model.get_text_features(*tokenize(queries)))
---> 25     image_model = lambda images: np.asarray(model.get_image_features(images.permute(0, 2, 3, 1).numpy(),))

[~/.local/lib/python3.8/site-packages/transformers/models/vision_text_dual_encoder/modeling_flax_vision_text_dual_encoder.py](https://file+.vscode-resource.vscode-cdn.net/c%3A/Users/96654/Downloads/~/.local/lib/python3.8/site-packages/transformers/models/vision_text_dual_encoder/modeling_flax_vision_text_dual_encoder.py) in get_image_features(self, pixel_values, params, dropout_rng, train)
    405 
--> 406         return self.module.apply(
    407             {"params": params or self.params},

[~/.local/lib/python3.8/site-packages/jax/_src/traceback_util.py](https://file+.vscode-resource.vscode-cdn.net/c%3A/Users/96654/Downloads/~/.local/lib/python3.8/site-packages/jax/_src/traceback_util.py) in reraise_with_filtered_traceback(*args, **kwargs)
    165     try:
--> 166       return fun(*args, **kwargs)
    167     except Exception as e:

[~/.local/lib/python3.8/site-packages/flax/linen/module.py](https://file+.vscode-resource.vscode-cdn.net/c%3A/Users/96654/Downloads/~/.local/lib/python3.8/site-packages/flax/linen/module.py) in apply(self, variables, rngs, method, mutable, capture_intermediates, *args, **kwargs)
   1484     method = _get_unbound_fn(method)
-> 1485     return apply(
   1486         method, self,
...
---> 96     return lax_fn(x1, x2) if x1.dtype != np.bool_ else bool_lax_fn(x1, x2)
     97   fn.__qualname__ = f"jax.numpy.{numpy_fn.__name__}"
     98   fn = jit(fn, inline=True)

TypeError: add got incompatible shapes for broadcasting: (512, 50, 1024), (1, 145, 1024).
```

### Expected behavior

to run smoothly and provide accuracy results
Hey @alhuri - as detailed in #22673 and #22780, the Italian CLIP repository is not maintained or affiliated with Hugging Face transformers. It is a standalone repository offering its own fine-tuning / evaluation scripts. As such, you're more likely to receive support regarding this issue by directly asking in the Italian CLIP repository: https://github.com/clip-italian/clip-italian/issues/new