Add missing type hints
### This issue is part of our **Great Code Cleanup 2022**. If you're interested in helping out, take a look at [this thread](https://twitter.com/carrigmat/status/1502319813510766599), or come [join us on Discord](https://t.co/kS42XBvpWH) and talk with other contributors!

# 🚀 Add missing type hints

Type hints are used inconsistently in the `transformers` repo across both TF and PT models, and it'd be nice to make them a complete, consistent thing for the core models, especially because we want to develop features that depend on them!

### Guide to contributing:

1. Ensure you've read our contributing [guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) 📜 
2. Claim your architecture(s) in this thread (ensure no one is working on it). It's 100% okay to only take the TensorFlow or PyTorch version of a model, if you're not familiar with both frameworks! It's also okay to claim multiple models and group those changes into a single PR! 🎯 
3. Implement the changes as in https://github.com/huggingface/transformers/pull/16057 or https://github.com/huggingface/transformers/pull/16074 (see the diff on the model architectures for a few examples) 💪 
4. Open the PR and tag me in it. You should run `make fixup` at the end to do a code quality check before your final commit!

### Tips for making your PR

1. The files you need to edit will be in `src/transformers/models/[model_name]/`
2. For TensorFlow, you want the `modeling_tf_[model_name].py` file. For PyTorch, you want the `modeling_[model_name].py` file.
3.  Remember, you **do not** have to cover every class in that file!. The main thing we want to cover is the `call` (for TF) or `forward` (for PT) method for user-facing classes like `TFRobertaForMaskedLM` or `RobertaForSequenceClassification`. It's not necessary to add type hints to layers or base classes like `RobertaModel` or `TFRobertaPreTrainedModel` - these are trickier to write, and generally people do not use those classes as standalone models.
4. If you're unfamiliar with how type hints work, you can read the [Python library documentation on them](https://docs.python.org/3/library/typing.html), but it's probably even easier to just look at another PR that added them. Take a look at the list of changes in the pull requests linked above!
5. The types will usually be obvious - most inputs are `Optional[Union[np.ndarray, tf.Tensor]]` for TF models and `Optional[torch.Tensor]` for PyTorch models, and boolean inputs are `Optional[bool]`. Pay attention to the first input of TF models, though, which is usually `TFModelInputType` - this is because Keras handles that first input in a special way! Other inputs to pay attention to are `past_key_values`, which can vary between models, and also the model output type. For the base model classes like `RobertaModel`, you may have to look at the corresponding `MainLayer` to figure out the right output type! Also, note that the output type may be a tuple if `return_dict` is False, in which case you should specify `Union[Tuple, ...]`. Finally, note that in TF models, `training` is never `None`, so it should be `training: bool` and not `training: Optional[bool]`.
6. Note that some code is copied across our codebase. If you see a line like `# Copied from transformers.models.bert...`, this means that the code is copied from that source, and our scripts will automatically keep that in sync. If you see that, you should not edit the copied method! Instead, edit the original method it's copied from, and run `make fixup` to synchronize that across all the copies. Be sure you installed the development dependencies with `pip install -e ".[dev"]`, as described in the contributor guidelines above, to ensure that the code quality tools in `make fixup` can run.

### How can I find models that need type hints?

I used to maintain a list here, but it got out of date, I'm sorry. Instead, you can use [this Colab notebook](https://colab.research.google.com/drive/1EvZTslb50yfRqIcXjCZFrbod4HrPdA0G?usp=sharing). If you run this, it will show you models in PyTorch or TF that are still missing type hints. Unlike my manually curated lists, it's guaranteed to be up to date - but do double-check that someone else in the thread hasn't claimed a model before you start, because the Colab code will only register type hints after the PR containing them is merged!
I would love to work on PyTorch `Albert`🚀<|||||>Hi, I would like to work on PyTorch `ImageGPT`<|||||>Hi, I would like to work on `CamemBERT` for PT & TF.

I will take a look at `LayoutLMv2` after the first one :smiley: 

Edit: Because CamemBert depends on `Roberta` I will take PyTorch `Roberta` :+1: <|||||>Hello! 

I'd like to take `Hubert` & `Wav2Vec2` for Pytorch.

Cheers!<|||||>I'll try PyTorch BERT to start!<|||||>@johnryan465 I just did it as an example, I'm sorry! I'm marking off the completed models now.
<|||||>@Rocketknight1 no worries, will try and do DistillBert instead<|||||>I'd like to work on GPT2 (TF).<|||||>@Rocketknight1 I switch to Roberta PyTorch because CamemBERT depends on Roberta modeling <|||||>Awesome! Hey @Rocketknight1 – I'd like to work on Longformer for both PyTorch & TF! <|||||>I'd like to work on BigBird<|||||>I would like to work on Clip for pytorch.<|||||>Also, will work on `BeiT`, `Deit` and `ViT` (Pytorch)<|||||>I can work on ImageGPT. <|||||>I can work on Swin (Pytorch)<|||||>I'd like to work on XLM (Tensorflow)<|||||>I'll take T5 (Tensorflow)!<|||||>I'd like to claim GPT-2 (PyTorch).<|||||>Hi @Rocketknight1,

I would like to work on  BART of  both TF and PyTorch<|||||>ELECTRA TF - https://github.com/huggingface/transformers/pull/16104 
ELECTRA PT - https://github.com/huggingface/transformers/pull/16103 
DeBERTA PT - https://github.com/huggingface/transformers/pull/16105
<|||||>XLMRobertaXL (PyTorch)
<|||||>segformer pytorch<|||||>I'll take OpenAIGPT!<|||||>> Hi @Rocketknight1,
> 
> I would like to work on BART of both TF and PyTorch

can you please confirm with emoji whether i am eligible to take these or not? @Rocketknight1 <|||||>I will work on XLM (PyTorch)<|||||>@robotjellyzone You can! Please note that we accepted a PR yesterday to add the TF decorator to BART, so make sure you're working on the most recent version of the library before you start your PR!<|||||>I'll take Distilbert (TensorFlow)<|||||>Happy to take T5 (PyTorch)

@Rocketknight1 isn't the list missing ConvNext? If so, I'm happy to take care of that one too :ok_hand: <|||||>I'll work on GPTJ<|||||>> @robotjellyzone You can! Please note that we accepted a PR yesterday to add the TF decorator to BART, so make sure you're working on the most recent version of the library before you start your PR!

OK sure! I will keep this in mind 😊👍... <|||||>I'll take Splinter and ~~Segformer~~ Rembert for torch
Edit: @p-mishra1 has Segformer. Taking Rembert instead<|||||>Looks like ImageGPT was done. I can take Luke in PyTorch. <|||||>I'd like to take PoolFormer<|||||>I'm going for `FlauBERT` now !<|||||>I'll work on FNet for PyTorch.

<|||||>Hello, I will work on SqueezeBERT for Pytorch!<|||||>I will also work on GPTNeo for Pytorch<|||||>I'd like to work on Perceiver for torch<|||||>I will also work on Pegasus for pytorch<|||||>I will work on XLMRoberta for TF<|||||>I will work on YOSO for PT<|||||>Hi, I'll take Marian (Pytorch)<|||||>Hi, I will work on RAG(pytorch).<|||||>Nevermind, XLMRoberta relies entirely on Roberta (for TF)
I will work on Reformer instead!<|||||>Hey, I would like to work on the BigBirdPegasus model of Pytorch.<|||||>Hey, I am looking into mBART model for TF and PyTorch implementations. If anyone, interested do let me know. <|||||>I will work on XLNet for TF and PT<|||||>Happy to take CTRL and MPNet for Tensorflow<|||||>I'm working on MobileBert for both TensorFlow & PyTorch.<|||||>Hi, I'd like to take OpenAIGPT (PyTorch).<|||||>Hi @DPT4, it has already been done here: https://github.com/huggingface/transformers/pull/16156<|||||>Hi @TristanBilot thanks for informing me. Then, I'd like to take MegatronBert for PyTorch.<|||||>This is Awesome! Hey, @Rocketknight1 I'd like to work on ConvBERT for both PyTorch & TF 🤗<|||||>I'd like to work on TransfoXL if available<|||||>I'll take Vilt for PyTorch seems available.<|||||> I will work on XGLM for PT<|||||>I would like to work on T5 (PyTorch)<|||||>I will work on ProphetNet, UniSpeech and UnispeechSat <|||||>I'll work on 
TF - Blenderbot 
PyTorch - Blenderbot and BlenderbotSmall<|||||>I can take up 
PyTorch: PLBart and VisualBERT<|||||>Hi, I'll take Funnel<|||||>I will take IBert, Lxmert for PyTorch.<|||||>Hi all, with the end of the sprint I probably won't be actively maintaining this issue, but there are still models outstanding! If you're curious about what's left to do, I've added [this Colab notebook](https://colab.research.google.com/drive/1EvZTslb50yfRqIcXjCZFrbod4HrPdA0G?usp=sharing) that you can use to find model classes without type hints. We'd appreciate type hints for any and all of them! <|||||>@Vaibhavs10 Are you still working on the Wav2Vec model? I'm working on UniSpeech and some ```make fixup``` tests fail because of inconsistencies with that model. I could take care of it if you want<|||||>Hi, I'd like to work on Canine.<|||||>@hiromu166 I have already worked on that model in my PR #16425 <|||||>Hey, I'd like to work on ProphetNet(Pytorch)

edit: @sgugger  Hey Sylvain. Found that the some sections in the docs (https://huggingface.co/docs/transformers/model_doc/prophetnet) are missing parameters explanations (for example in the ProphetNetConfig there's no definition for the decoder_start_token_id, and something similar happens for the ProphetNetModel and the ProphetNetEncoder). Not sure if it was done on purpose, so I'm only posting here!<|||||>Hi, I would like to work on `Pytorch VisualBERT`.<|||||>@Rocketknight1 It appears that all user facing classes in `Pytorch VisualBERT` are already annotated as mentioned in this PR #16544, so I would like to work on `Pytorch BigBirdPegasus`
<|||||>Hello @Rocketknight1  I would like to work on ConvBert tensorflow<|||||>@Rocketknight1 It seems that "ConvBert" is already done PR #16377 for both TensorFlow and Pytorch.
So I would like to work on "yoso" Pytorch<|||||>Hey there @Rocketknight1 I would like to work on MegatronBERT 😄 (edit: pytorch)<|||||>Hi @Rocketknight1 I checked the Colab notebook and I will be adding the missing type hints for CVT (Pytorch)

> Hi all, with the end of the sprint I probably won't be actively maintaining this issue, but there are still models outstanding! If you're curious about what's left to do, I've added [this Colab notebook](https://colab.research.google.com/drive/1EvZTslb50yfRqIcXjCZFrbod4HrPdA0G?usp=sharing) that you can use to find model classes without type hints. We'd appreciate type hints for any and all of them!

<|||||>Good shout @F02934 ! 

@Rocketknight1 I've checked the above Colab notebook and looks like MegatronBERT is done. I will add missing type hints for QDQBertModel <|||||>Can I take `GPTNeoxForCausalLM` and `GPTNeoXModel`?<|||||>I would like to add type hints for `RoFormer` (only in Pytorch). After running and checking the notebook Colab [notebook](https://colab.research.google.com/drive/1EvZTslb50yfRqIcXjCZFrbod4HrPdA0G), I think I can take care of the seven classes missing type hints 👍️.

Should I submit a single PR for all the seven updated classes?

> It's also okay to claim multiple models and group those changes into a single PR! dart

_Edit: I just read again the description of this issue and noticed my question was already answered_<|||||>Hi, I would like to work on `TensorFlow` : `OpenAIGPT`<|||||>Hi, I would like to work on `TensorFlow` : `CTRL`<|||||>Hello, according to the colab notebook, `XLMRobertaXL` (PyTorch) classes are still missing type annotations
They were mentioned here back on March 12th but there is no PR associated, so I can work on that if @manandey hasn't already<|||||>> I can work on that if @manandey hasn't already

@asofiaoliveira Feel free to work on this incase you are interested!

<|||||>Hi 🤗. `Vilt` (PyTorch) model remains without type hints, so I would like to work on it.<|||||>I'll take on PyTorch BigBirdPegasus and open a PR :)<|||||>M2M model missing type hints, I'll work on it and open a PR.<|||||>I'll create a PR for FSMT :)<|||||>Some type hints are missing in PyTorch UniSpeech, MPNet, and Nystromformer.
I'll take on it and group those changes into a single PR!<|||||>PyTorch SEWD model missing some type hints, I'll work on it and open a PR.<|||||>`TFConvBERTModel` does not have type hints. I would like to work on adding typing hints for it.<|||||>I would like to work on adding type hints for `TF MPNet` models.<|||||>I'm taking up `TF Rag`<|||||>I'd like to work on Lxmert for TF and PyTorch.<|||||>
Hi, I would like to work on Tensorflow LED

<|||||>HI, I would like to work on Tensorflow XLNet<|||||>I would like to work on Pytorch/BigBirdPegasus<|||||>Hey! I'll take `OpenAIGPT` both for TensorFlow & PyTorch<|||||>I would love to work on PyTorch SEWD for the remaining classes.<|||||>Hi!, I can create a PR for CTRL for Pytorch and Tensorflow.
<|||||>Hi, I'd like to work on MT5 for Pytorch and Tensorflow 🙂<|||||>Hey, I would like to work on the Swin model files!<|||||>Hi I'd love to work on ~~Reformer, Data2Vec, and RoFormer pytorch 😊~~

Seems like a lot of these have been merged already but not marked off the list for pytorch (CTRL, OpenAIGPT, BigBirdPegasus, M2M, reformer, data2vec, roformer, visualbert). If there's anything left I'll take it lol<|||||>Hi @sirmammingtonham thank you so much for pointing that out! I just realized I'd been failing to keep the list of models up to date - I made a Colab to check our codebase instead, which is linked at the bottom of the initial post and will always be up to date.<|||||>This applies to other people who recently started, cc @mariagrandury @arcAman07 @RamitPahwa @rchan26 @WhiteWolf47. If you found the model you chose already has type hints, this is why! Please double-check in the Colab to see where type hints are still needed.<|||||>Hey just saw the codebase and realised a lot of models already have type hints. I did notice dpt list of models didn't have type hints and I would love to work on that issue.<|||||>Thanks @Rocketknight1 the notebook is super helpful! I can work on adding return types to tf xlm and tf GPTJ.<|||||>> This applies to other people who recently started, cc @mariagrandury @arcAman07 @RamitPahwa @rchan26 @WhiteWolf47. If you found the model you chose already has type hints, this is why! Please double-check in the Colab to see where type hints are still needed.

Hey, just checked out the colab notebook, i have decided to work on the YolosModel<|||||>Thanks all, and sorry again about the list being out of date initially!<|||||>Hi, I would love to work on FLAVA and LEVIT Pytorch , can you assign this to me?<|||||>Hi I was going to work on MT5 for Pytorch and TF, but looking at `src/transformers/models/mt5/modeling_mt5.py` and `src/transformers/models/mt5/modelling_tf_mt5`, it looks like each of the classes in these files override other models (`T5model`, `T5ForConditionalGeneration`, `T5EncoderModel`, `TFT5Model`, `TFT5ForConditionalGeneration`, `TFT5EncoderModel`). Each of these already have type hints, so I guess there isn't anything to do for MT5 now?

Edit: Sorry, just seen that the colab is where to look for models that still need work... In that case, I'll pick up `MCTCTForCTC` and `MCTCTModel`!


<|||||>@Rocketknight1  I will work on ViTModel<|||||>Hi @Rocketknight1 , ~I will take `TFViTPreTrainedModel` :)~ change of plans, I didn't notice that `PreTrained` classes were not priority.
I will take `TFPegasusModel` instead<|||||>I would like to work on `EsmForProteinFolding`<|||||>Hi folks, 

We are almost done with this task! From the [notebook](https://colab.research.google.com/drive/1EvZTslb50yfRqIcXjCZFrbod4HrPdA0G?usp=sharing) created by RK ([Rocketknight1](https://github.com/Rocketknight1)), I'm sharing here the remaining models to gain some visibility and encourage one last push.

1. PyTorch models: 
   - ASTModel
   - EsmForProteinFolding
   - TimesformerModel

2. TF models (excluding pre-trained classes):
   - TFDeiTModel
   - TFEncoderDecoderModel (WIP by [Batese2001](https://github.com/Batese2001))
   - TFLEDForConditionalGeneration
   - TFLEDModel
   - TFLxmertForPreTraining
   - TFMarianMTModel
   - TFMarianModel
   - TFRagModel
   - TFRagTokenForGeneration
   - TFSegformerDecodeHead
   - TFTransfoXLLMHeadModel
   - TFTransfoXLModel
   - TFVisionEncoderDecoderModel

_Important_: Remember to **run** the [notebook](https://colab.research.google.com/drive/1EvZTslb50yfRqIcXjCZFrbod4HrPdA0G?usp=sharing) to check which models are available (the ones printed in each cell mean models without type hints) and look if anyone has claimed before start working on it.<|||||>Hello @Rocketknight1 I'd love to tackle TFEncoderDecoderModel if it is available <|||||>Hello @Rocketknight1 I would like to tackle TFVisionEncoderDeocderModel if possible with @miyu386  and @AdiaWu.<|||||>Hi @Rocketknight1 I'd love to take TFMarianModel with @Batese2001 and @mollerup23!<|||||>Good Morning @Rocketknight1,
I would love to help you with adding types to the following models:

1. TFAlbertPreTrainedModel
2. TFBartPretrainedModel
3. TFBertPreTrainedModel<|||||>@Rocketknight1 Are there other models that need type hints? I looked at the list generated by the Google Colab notebook, and noticed the 'TFPegasusPreTrainedModel' still needs type hints. If so, I'd like to pick this up.<|||||>Oops, yep, I didn't realize that last PR had auto-closed it. If you can find other models that need type hints, then please submit PRs to add them! Make sure to 'sync changes' in your fork before creating the PR branch so that there aren't any issues with rebasing and it should go smoothly.<|||||>Great! In that case I'd like to claim the TFPegasusPreTrainedModel. I'll make sure to sync before making any commits. Thanks!<|||||>@Rocketknight1 I would love to take TFDeiTModel if that's alright. <|||||>@Rocketknight1 Actually, looking deeper into the model, it looks like the type hints are already implemented in the call method for the TFDeiTModel class. Should I be adding the type hints to some of the other call methods in modeling_tf_deit.py?<|||||>@Batese2001 If the script is listing models as needing type hints but they already have them, it's possible that they're missing some part of the hints. Maybe the return type from the main model `call()` methods is not present?<|||||>@Rocketknight1 Hi Matt, can I work on TFBlenderbot for this issue?<|||||>@Rocketknight1 Hi Matt 👋 , i would like to take `Graphormer`  pytorch version<|||||>hi there 👋 can anybody tell me how to perform all check before comitting, make fixup- not working and github ci/cd failing, is it because i'm using python 3.11 or some other error<|||||>Hi @dewasahu2003, to run `make fixup` you will need to install some dependencies for code formatting and quality tools. The easiest way to do this is `pip install transformers[quality]` or `pip install -e .[quality]` in the directory where you cloned the `transformers` repo.<|||||>Hi 👋 @Rocketknight1,thanks for the helpful feedback
But I closed the pull request that I was facing issue with and open a new pull request  with proper code formatting and `make fixup` command.

new pull request 👉 [New Pull Request](https://github.com/huggingface/transformers/pull/23073)<|||||>If ALBERT or XLMRoBERTa (Tensorflow) are still available then I would like to work on any one of them.<|||||>@MuskaanCAIR You can use the [colab script](https://colab.research.google.com/drive/1EvZTslb50yfRqIcXjCZFrbod4HrPdA0G?usp=sharing) in the first post to see which models still need hints. Remember that models will be listed by that Colab as long as they're missing full type hints including return type hints on the `call()` method. It's less important to annotate layers and functions in the model files, although still appreciated!<|||||>Colab lists TFXLMRobertaPreTrainedModel as one of the models. Can I take that?
<|||||>Hey @Rocketknight1 hope you are doing great, I see that there is still work to do here. `ASTModel` feels like a great opportunity of learning so I will work on it if no one else has picked it up, (if that's the case feel free to yell at me :upside_down_face: )