DeltaLM
### Model description

DeltaLM is a multilingual encoder-decoder architecture that regards the decoder as the task layer of off-the-shelf pre-trained encoders. This architecture introduces an interleaved decoder, which has a more consistent structure with the encoder. Weights from pre-trained multilingual encoders are used to initialise both the encoder and decoder models before training on monolingual and bilingual data.

As of September 2021 DeltaLM ranks first on the [WMT21 multilingual translation task](http://www.statmt.org/wmt21/large-scale-multilingual-translation-task.html).

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation
The model implementation is available at: 
https://github.com/microsoft/unilm/tree/master/deltalm

Model weights are available:
[DeltaLM-base](https://deltalm.blob.core.windows.net/deltalm/deltalm-base.pt)
[DeltaLM-large](https://deltalm.blob.core.windows.net/deltalm/deltalm-large.pt)

Who are the authors:
@shumingma @gitnlp

I'd be happy to try work on contributing the model.
Any progress on this?<|||||>Hi,

I've noticed there are some DeltaLM available in the Hub:
- https://huggingface.co/IDEA-CCNL/Randeng-Deltalm-362M-En-Zh
- https://huggingface.co/IDEA-CCNL/Randeng-Deltalm-362M-Zh-En

The code they use seems to be here: https://github.com/IDEA-CCNL/Fengshenbang-LM/tree/main/fengshen/models/deltalm

However, I'd be interested in using the original checkpoints available in the official repository: https://github.com/microsoft/unilm/tree/master/deltalm

Do you have any idea on how to do it?

Thanks!