[WIP] Add UDOP models
#20650 
_The documentation is not available anymore as the PR was closed or merged._<|||||>@sgugger  @NielsRogge The model weights are here https://huggingface.co/ZinengTang/Udop/tree/main , But how to get the config for these models ?  
<|||||>@raghavanone For reference, someone asked the same question on the UDOP repo: https://github.com/microsoft/i-Code/issues/17<|||||>Note: Cannot proceed further without microsoft releasing the entire weights. Currently vision decoder weights have not been released.<|||||>If I'm not mistaken, vision decoder weights should not be needed when using the text layout decoder part, only.
`vision_encoder` weights are part of the shared model weights.<|||||>@raghavanone is there anything else blocking? It sounds like we can proceed with the given weights, assuming that we notify users that the vision decoder is not trained. <|||||>@logan-markewich  Yes, I will work on closing this within couple of days . <|||||>@sgugger Need some pointers on How should this model be tested ? Can I follow the tests used for T5 model and replicate similar tests ? <|||||>@NielsRogge Any pointer here ? <|||||>I hope it gets merged soon @raghavanone . Nice work :)<|||||>Forgive my naiveté, why do all the tests call `from_pretrained()` on some variation of `t5`? The UDOP model checkpoints are [here](https://huggingface.co/ZinengTang/Udop/tree/main). Could these be used?<|||||>Ah, I see that the test script they provide also [uses T5-large](https://github.com/microsoft/i-Code/blob/main/i-Code-Doc/scripts/finetune_rvlcdip.sh), I expected it to use one of those checkpoints<|||||>@raghavanone how are things going with this so far? I'm very interested in using this model as soon as it gets integrated - if you need a hand with anything let me know! And thanks for bringing it into the library 😄 
 <|||||>> @raghavanone how are things going with this so far? I'm very interested in using this model as soon as it gets integrated - if you need a hand with anything let me know! And thanks for bringing it into the library 😄

@thefirebanks I am working on fixing last few tests. Hoping to close this PR very soon. Sorry for the delay.<|||||>@raghavanone I am currently trying to finetune `UdopUniModelForConditionalGeneration` using this PR. I ran into the following exception while training:

```
 File "/opt/conda/lib/python3.8/site-packages/transformers/models/udop/modeling_udop.py", line 2422, in forward
 encoder_outputs = self.encoder(
 TypeError: forward() got an unexpected keyword argument 'ids_keep'`
```

I explained what appears to be happening in [this comment](https://github.com/huggingface/transformers/commit/ea7e44ca37d14d24798ed938b52ce3b2a202816f#r103307941).

It looks like the `ids_keep` parameter was removed from `UdopUniStack` but not removed from the call to it in `UdopUniModelForConditionalGeneration`

**EDIT**
Looks like `output_attentions`, also needs to be removed
And in the `self.decoder()` call, `cross_attn_head_mask`, `output_attentions` 

Happy to make the changes myself with repo permissions

<|||||>> @raghavanone I am currently trying to finetune `UdopUniModelForConditionalGeneration` using this PR. I ran into the following exception while training:
> 
> ```
>  File "/opt/conda/lib/python3.8/site-packages/transformers/models/udop/modeling_udop.py", line 2422, in forward
>  encoder_outputs = self.encoder(
>  TypeError: forward() got an unexpected keyword argument 'ids_keep'`
> ```
> 
> I explained what appears to be happening in [this comment](https://github.com/huggingface/transformers/commit/ea7e44ca37d14d24798ed938b52ce3b2a202816f#r103307941).
> 
> It looks like the `ids_keep` parameter was removed from `UdopUniStack` but not removed from the call to it in `UdopUniModelForConditionalGeneration`
> 
> **EDIT** Looks like `output_attentions`, also needs to be removed And in the `self.decoder()` call, `cross_attn_head_mask`, `output_attentions`
> 
> Happy to make the changes myself with repo permissions

@plamb-viso Yes, removing those parameters were not done in all places, I have fixed it locally. I am working on fixing failing tests. This the last step pending for merging. Fixing these tests are taking more time than expected. <|||||>@raghavanone I saw you closed this PR. Skimming over your work, the PR seemed to be in a rather good state. Where there any blockers you encountered? IMO, it would be nice to add UDOP models in Hugginface at some point.<|||||>@maxjeblick @NielsRogge feels that the code original repo is bit hacky, he is working a separate PR to UDOP in better implementation, so closed this in consultation with him. He should open a PR soon .

@NielsRogge please do add more details for the benefit of folks following this PR <|||||>Thanks a lot for the fast reply!<|||||>@NielsRogge @raghavanone please link the new PR when its available for people subscribed to this one<|||||>Hi yes I'll open a PR soon! Thanks a lot for your work already @raghavanone, will ping you on the PR <|||||>Hi @NielsRogge I saw the large amount of commits on your new UDOP branch, curious if you have any idea on when you think a PR might be ready<|||||>Sorry to keep hammering on this, but again have noticed a flurry of activity on that branch then almost 2 weeks off. Curious what the plan is for it @NielsRogge <|||||>Hi @plamb-viso sorry for the late reply, the model is working, only have limited time to work on it. I'll open a PR this weekend/Monday.

For now you can already use the model if you're curious, check [this code example](https://github.com/NielsRogge/transformers/blob/14f327d1e9804aeddbe420bd44b811945a3aadd4/tests/models/udop/test_modeling_udop.py#L363) regarding usage. Model is already on the hub [here](https://huggingface.co/nielsr/udop-large).<|||||>Out of curiosity @NielsRogge : did you ever use your implementation to fine tune it on a task like CORD?<|||||>I've fine-tuned the model on a [toy dataset of RVL-CDIP](https://huggingface.co/datasets/nielsr/rvl_cdip_10_examples_per_class), works well but the model is pretty heavy, got OOM on Google Colab even with batch size = 1 so had to use a bigger GPU. The author only released large variants. <|||||>In my original work on @raghavanone 's version of the model, I also had to use a batch size of 1 to get it to not OOM on 40gb GPUs