[layoutlmv3] SER and RE task combined into one model
### System Info


datasets==1.15.1
transformers==4.12.5
seqeval==1.2.2
deepspeed==0.5.7
tensorboard==2.7.0
seqeval==1.2.2
sentencepiece
timm==0.4.12
Pillow
einops
textdistance
shapely

### Who can help?

@NielsRogge

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

combine SER and RE task into one model using layoutlmv3

### Expected behavior

Hi @NielsRogge,
Now we want to train SER and RE task using layoutlmv3, but these two models is more heavy for deploy with tensorrt, so to solve this problem, if we combine two tasks into one model, the performance for SER and RE will decrease more compared to the seprated two models?  we have no more experience for it, do you give us some advice? thank you very much.
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.<|||||>I think it's totally doable to share the Transformer encoder and use 2 separate heads. <|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.<|||||>Closing this issue as it seems resolved. Feel free to reopen.