msgpack.exceptions.ExtraData: unpack(b) received extra data.
### System Info

- `transformers` version: 4.9.0
- Platform: Linux-5.15.0-52-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyTorch version (GPU?): 1.9.0+cpu (False)
- Tensorflow version (GPU?): 2.9.1 (True)
- Flax version (CPU?/GPU?/TPU?): 0.3.4 (gpu)
- Jax version: 0.4.8
- JaxLib version: 0.4.7
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

-Models: FlaxHybridCLIP

### Who can help?

@sanchit-gandhi 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I'm trying to load a trained FlaxHybridCLIP model from a folder that contains the following files

config.json
flax_model.msgpack

I attempted to load it using the below:
```
 if args.run_from_checkpoint is not None:
        with open(f"{args.run_from_checkpoint}/config.json", "r") as fp:
            config_dict = json.load(fp)
        config_dict["vision_config"]["model_type"] = "clip"
        config = HybridCLIPConfig(**config_dict)
        model = FlaxHybridCLIP.from_pretrained(
            args.run_from_checkpoint,
            seed=training_args.seed,
            dtype=getattr(jnp, model_args.dtype),
            config=config,
            freeze_backbones=args.freeze_backbones
        )
```
But, I encountered the following error:
```
text_config_dict is None. Initializing the CLIPTextConfig with default values.
vision_config_dict is None. initializing the CLIPVisionConfig with default values.
loading weights file freeze/flax_model.msgpack
Traceback (most recent call last):
  File "run_hybrid_clip.py", line 832, in <module>
    main()
  File "run_hybrid_clip.py", line 529, in main
    model = FlaxHybridCLIP.from_pretrained(
  File "/home/ubuntu/.local/lib/python3.8/site-packages/transformers/modeling_flax_utils.py", line 350, in from_pretrained
    state = from_bytes(cls, state_f.read())
  File "/home/ubuntu/.local/lib/python3.8/site-packages/flax/serialization.py", line 359, in from_bytes
    state_dict = msgpack_restore(encoded_bytes)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/flax/serialization.py", line 342, in msgpack_restore
    return msgpack.unpackb(
  File "msgpack/_unpacker.pyx", line 201, in msgpack._cmsgpack.unpackb
msgpack.exceptions.ExtraData: unpack(b) received extra data.
```
I used the modified Italian hybrid CLIP scripts [here](https://github.com/clip-italian/clip-italian/tree/master/hybrid_clip)

### Expected behavior

to load successfully and fine-tune with unfrozen backbone

Thanks
Hey @alhuri! Echoing what I suggested in https://github.com/huggingface/transformers/issues/22673#issuecomment-1517714740 - this is probably best asked on the Italian Hybrid CLIP repo (since they use a custom implementation of JAX CLIP there). Unless you have a code snippet that I can use to reproduce this error with just transformers? In which case I'd be able to take a deeper dive!<|||||>Closing since this issue is related to the Italian CLIP repo (not transformers!)