[WIP] Add UniLM model
# Typical workflow for including a model

Here an overview of the general workflow: 

- [x] add model/configuration/tokenization classes
- [x] add conversion scripts
- [x] add tests
- [x] finalize

Let's detail what should be done at each step

## Adding model/configuration/tokenization classes

Here is the workflow for adding model/configuration/tokenization classes:

- [x] copy the python files from the present folder to the main folder and rename them, replacing `xxx` with your model name,
- [x] edit the files to replace `XXX` (with various casing) with your model name
- [x] copy-paste or create a simple configuration class for your model in the `configuration_...` file
- [x] copy-paste or create the code for your model in the `modeling_...` files (PyTorch and TF 2.0)
- [x] copy-paste or create a tokenizer class for your model in the `tokenization_...` file

# Adding conversion scripts

Here is the workflow for the conversion scripts:

- [x] copy the conversion script (`convert_...`) from the present folder to the main folder.
- [x] edit this script to convert your original checkpoint weights to the current pytorch ones.

# Adding tests:

Here is the workflow for the adding tests:

- [x] copy the python files from the `tests` sub-folder of the present folder to the `tests` subfolder of the main folder and rename them, replacing `xxx` with your model name,
- [x] edit the tests files to replace `XXX` (with various casing) with your model name
- [x] edit the tests code as needed

# Final steps

You can then finish the addition step by adding imports for your classes in the common files:

- [x] add import for all the relevant classes in `__init__.py`
- [x] add your configuration in `configuration_auto.py`
- [x] add your PyTorch and TF 2.0 model respectively in `modeling_auto.py` and `modeling_tf_auto.py`
- [x] add your tokenizer in `tokenization_auto.py`
- [x] add your models and tokenizer to `pipeline.py`
- [x] add a link to your conversion script in the main conversion utility (currently in `__main__` but will be moved to the `commands` subfolder in the near future)
- [x] edit the PyTorch to TF 2.0 conversion script to add your model in the `convert_pytorch_checkpoint_to_tf2.py` file
- [x] add a mention of your model in the doc: `README.md` and the documentation itself at `docs/source/pretrained_models.rst`.
- [x] upload the pretrained weigths, configurations and vocabulary files.

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2160?src=pr&el=h1) Report
> Merging [#2160](https://codecov.io/gh/huggingface/transformers/pull/2160?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/f061606277322a013ec2d96509d3077e865ae875?src=pr&el=desc) will **increase** coverage by `0.04%`.
> The diff coverage is `49.69%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2160/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2160?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #2160      +/-   ##
==========================================
+ Coverage   80.32%   80.37%   +0.04%     
==========================================
  Files         122      127       +5     
  Lines       18342    19000     +658     
==========================================
+ Hits        14734    15272     +538     
- Misses       3608     3728     +120
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2160?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/2160/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3V0aWxzLnB5) | `88.91% <15.38%> (-2.55%)` | :arrow_down: |
| [transformers/configuration\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/2160/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL2NvbmZpZ3VyYXRpb25fYXV0by5weQ==) | `44.68% <33.33%> (-0.78%)` | :arrow_down: |
| [transformers/tokenization\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/2160/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rva2VuaXphdGlvbl9hdXRvLnB5) | `59.18% <33.33%> (-1.69%)` | :arrow_down: |
| [transformers/modeling\_unilm.py](https://codecov.io/gh/huggingface/transformers/pull/2160/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3VuaWxtLnB5) | `36.95% <36.95%> (ø)` | |
| [transformers/modeling\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/2160/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX2F1dG8ucHk=) | `38.79% <53.84%> (+1.89%)` | :arrow_up: |
| [transformers/tests/tokenization\_unilm\_test.py](https://codecov.io/gh/huggingface/transformers/pull/2160/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rlc3RzL3Rva2VuaXphdGlvbl91bmlsbV90ZXN0LnB5) | `55% <55%> (ø)` | |
| [transformers/configuration\_unilm.py](https://codecov.io/gh/huggingface/transformers/pull/2160/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL2NvbmZpZ3VyYXRpb25fdW5pbG0ucHk=) | `87.09% <87.09%> (ø)` | |
| [transformers/tests/modeling\_unilm\_test.py](https://codecov.io/gh/huggingface/transformers/pull/2160/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rlc3RzL21vZGVsaW5nX3VuaWxtX3Rlc3QucHk=) | `93.75% <93.75%> (ø)` | |
| [transformers/tokenization\_unilm.py](https://codecov.io/gh/huggingface/transformers/pull/2160/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rva2VuaXphdGlvbl91bmlsbS5weQ==) | `94.73% <94.73%> (ø)` | |
| ... and [10 more](https://codecov.io/gh/huggingface/transformers/pull/2160/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2160?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2160?src=pr&el=footer). Last update [f061606...bbacc86](https://codecov.io/gh/huggingface/transformers/pull/2160?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
<|||||>Thank you for the PR! I edited your post to add the guideline for adding a new model; we'll check the boxes as we go. I'll have a look at the code and come back to you quickly!<|||||>> # Typical workflow for including a model
> * [ ]  add your models and tokenizer to `pipeline.py`

@rlouf  Sorry, I didn't find the `pipeline.py` file.<|||||>@sshleifer Thanks for the comments! We will merge them into the code.  @addf400 <|||||>Let's restart with a new pull request @addf400 @donglixp <|||||>Is anyone still working on this? @addf400 @donglixp @JetRunner 
also @thomwolf from #1530<|||||>I'm also looking forward to applying the UniLM model via Huggingface Transformers!
@donglixp @JetRunner @thomwolf <|||||>It seems that this pull request has lasted for a year but still not finished? Is someone still working on it? <|||||>Has this PR for UniLM model been added to Huggingface Transformers?
@donglixp @JetRunner @thomwolf @sshleifer<|||||>Hey @ontocord , I think it the "minilm" model should work out-of-the-box:

https://github.com/huggingface/transformers/issues/5777

Not sure if you're looking for this model :thinking: 

I haven't tried it yet, but the recent Microsoft papers (on language modeling) are looking really promising!<|||||>Thanks @stefan-it. I don't think MiniLM and UniLM are the same thing, altough it all falls under one project. The MS papers are promising!<|||||>I'm also looking forward to applying the unilm model via Huggingface Transformers!<|||||>2022 year, still not merged the unilm model into the master branch.<|||||>I'm still looking forward to applying the unilm model via Huggingface Transformers! 👻👻

<|||||>I'm still looking forward to applying the unilm model via Huggingface Transformers too!