MPNet: Masked and Permuted Pre-training for Language Understanding
# 🌟 New model addition

## Model description

[MPNet: Masked and Permuted Pre-training for Language Understanding](https://arxiv.org/pdf/2004.09297.pdf)
use both Masked Language Model and Permuted Language Model

## Open source status

* [MPNet ] the model implementation is available: (https://github.com/microsoft/MPNet)
* [MPNet pretrain weight ] the model weights are available: (https://modelrelease.blob.core.windows.net/pre-training/MPNet/mpnet.example.pt)
* [@tobyoup@StillKeepTry ] who are the authors: (mention them, if possible by @gh-username)

Hi huggingface/transformers,

I am Xu Tan from Microsoft Research. Thanks for that you mentioned MPNet in huggingface/transformers repo.

MPNet is an advanced pre-trained models for language understanding, which inherits the advantages of BERT and XLNet, while well address their problems.  We have MPNet pre-trained models, codes (the version for huggingface/transformers) and downstream task results ready.

How can we proceed to incorporate MPNet into huggingface/transformers repo? Thanks very much!


Best,
Xu
From: RyanHuangNLP <notifications@github.com>
Sent: Tuesday, May 12, 2020 5:48 PM
To: huggingface/transformers <transformers@noreply.github.com>
Cc: Xu Tan <Xu.Tan@microsoft.com>; Mention <mention@noreply.github.com>
Subject: [huggingface/transformers] MPNet: Masked and Permuted Pre-training for Language Understanding (#4308)

🌟 New model addition
Model description

MPNet: Masked and Permuted Pre-training for Language Understanding<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Farxiv.org%2Fpdf%2F2004.09297.pdf&data=02%7C01%7CXu.Tan%40microsoft.com%7C36a49d494f604784087708d7f6598986%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248737639246486&sdata=%2BE5kPD%2F1nyY9bgRtqr%2FCFf2YzsI8E%2BcTzWX0gcUSnWs%3D&reserved=0>
use both Masked Language Model and Permuted Language Model

Open source status

  *   [MPNet ] the model implementation is available: (https://github.com/microsoft/MPNet<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmicrosoft%2FMPNet&data=02%7C01%7CXu.Tan%40microsoft.com%7C36a49d494f604784087708d7f6598986%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248737639246486&sdata=%2Fp94TR%2BQVvRNdoio0kjL%2FjmGgtQb8SlcQouAnas8gmY%3D&reserved=0>)
  *   [MPNet pretrain weight ] the model weights are available: (https://modelrelease.blob.core.windows.net/pre-training/MPNet/mpnet.example.pt<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmodelrelease.blob.core.windows.net%2Fpre-training%2FMPNet%2Fmpnet.example.pt&data=02%7C01%7CXu.Tan%40microsoft.com%7C36a49d494f604784087708d7f6598986%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248737639256488&sdata=F8O2da%2FYPXS8AjmIVt4D%2FOaNLOYmSVgfyICRQuHf0BI%3D&reserved=0>)
  *   [@tobyoup<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftobyoup&data=02%7C01%7CXu.Tan%40microsoft.com%7C36a49d494f604784087708d7f6598986%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248737639266479&sdata=hJn0ZIEEzgukAII%2FtaXEl6ZmIt2jQqRwtSP2Hu1lujs%3D&reserved=0>@StillKeepTry ] who are the authors: (mention them, if possible by @gh-username<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgh-username&data=02%7C01%7CXu.Tan%40microsoft.com%7C36a49d494f604784087708d7f6598986%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248737639266479&sdata=3ntmS0WjsV8DxOyrDyt7Qduc6jGMzU%2BfdcniizaYuLQ%3D&reserved=0>)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fhuggingface%2Ftransformers%2Fissues%2F4308&data=02%7C01%7CXu.Tan%40microsoft.com%7C36a49d494f604784087708d7f6598986%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248737639276475&sdata=t3iSTAMR54mEsK%2Blgc9pjxuYZmnYQcwJjtje%2BQ05yeU%3D&reserved=0>, or unsubscribe<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FALQV7L4JWDDOBVHDVZYRCD3RRELMPANCNFSM4M6VMZPQ&data=02%7C01%7CXu.Tan%40microsoft.com%7C36a49d494f604784087708d7f6598986%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248737639276475&sdata=4496eTEWR5xHALKY9MGu6Aal%2Fx%2BU1dn87FkpASm72ME%3D&reserved=0>.
<|||||>+ RyanHuangNLP

From: Xu Tan
Sent: Sunday, July 5, 2020 6:58 PM
To: huggingface/transformers <reply@reply.github.com>; huggingface/transformers <transformers@noreply.github.com>
Subject: RE: [huggingface/transformers] MPNet: Masked and Permuted Pre-training for Language Understanding (#4308)

Hi huggingface/transformers,

I am Xu Tan from Microsoft Research. Thanks for that you mentioned MPNet in huggingface/transformers repo.

MPNet is an advanced pre-trained models for language understanding, which inherits the advantages of BERT and XLNet, while well address their problems.  We have MPNet pre-trained models, codes (the version for huggingface/transformers) and downstream task results ready.

How can we proceed to incorporate MPNet into huggingface/transformers repo? Thanks very much!


Best,
Xu
From: RyanHuangNLP <notifications@github.com<mailto:notifications@github.com>>
Sent: Tuesday, May 12, 2020 5:48 PM
To: huggingface/transformers <transformers@noreply.github.com<mailto:transformers@noreply.github.com>>
Cc: Xu Tan <Xu.Tan@microsoft.com<mailto:Xu.Tan@microsoft.com>>; Mention <mention@noreply.github.com<mailto:mention@noreply.github.com>>
Subject: [huggingface/transformers] MPNet: Masked and Permuted Pre-training for Language Understanding (#4308)

🌟 New model addition
Model description

MPNet: Masked and Permuted Pre-training for Language Understanding<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Farxiv.org%2Fpdf%2F2004.09297.pdf&data=02%7C01%7CXu.Tan%40microsoft.com%7C36a49d494f604784087708d7f6598986%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248737639246486&sdata=%2BE5kPD%2F1nyY9bgRtqr%2FCFf2YzsI8E%2BcTzWX0gcUSnWs%3D&reserved=0>
use both Masked Language Model and Permuted Language Model

Open source status

  *   [MPNet ] the model implementation is available: (https://github.com/microsoft/MPNet<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmicrosoft%2FMPNet&data=02%7C01%7CXu.Tan%40microsoft.com%7C36a49d494f604784087708d7f6598986%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248737639246486&sdata=%2Fp94TR%2BQVvRNdoio0kjL%2FjmGgtQb8SlcQouAnas8gmY%3D&reserved=0>)
  *   [MPNet pretrain weight ] the model weights are available: (https://modelrelease.blob.core.windows.net/pre-training/MPNet/mpnet.example.pt<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmodelrelease.blob.core.windows.net%2Fpre-training%2FMPNet%2Fmpnet.example.pt&data=02%7C01%7CXu.Tan%40microsoft.com%7C36a49d494f604784087708d7f6598986%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248737639256488&sdata=F8O2da%2FYPXS8AjmIVt4D%2FOaNLOYmSVgfyICRQuHf0BI%3D&reserved=0>)
  *   [@tobyoup<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftobyoup&data=02%7C01%7CXu.Tan%40microsoft.com%7C36a49d494f604784087708d7f6598986%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248737639266479&sdata=hJn0ZIEEzgukAII%2FtaXEl6ZmIt2jQqRwtSP2Hu1lujs%3D&reserved=0>@StillKeepTry ] who are the authors: (mention them, if possible by @gh-username<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgh-username&data=02%7C01%7CXu.Tan%40microsoft.com%7C36a49d494f604784087708d7f6598986%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248737639266479&sdata=3ntmS0WjsV8DxOyrDyt7Qduc6jGMzU%2BfdcniizaYuLQ%3D&reserved=0>)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fhuggingface%2Ftransformers%2Fissues%2F4308&data=02%7C01%7CXu.Tan%40microsoft.com%7C36a49d494f604784087708d7f6598986%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248737639276475&sdata=t3iSTAMR54mEsK%2Blgc9pjxuYZmnYQcwJjtje%2BQ05yeU%3D&reserved=0>, or unsubscribe<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FALQV7L4JWDDOBVHDVZYRCD3RRELMPANCNFSM4M6VMZPQ&data=02%7C01%7CXu.Tan%40microsoft.com%7C36a49d494f604784087708d7f6598986%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248737639276475&sdata=4496eTEWR5xHALKY9MGu6Aal%2Fx%2BU1dn87FkpASm72ME%3D&reserved=0>.
<|||||>Hi @tobyoup, it's very cool that you want to contribute your architecture!

You can follow this [checklist](https://github.com/huggingface/transformers/tree/master/templates/adding_a_new_model) to add your model. Once you open a PR, we'll be very happy to help you finish the integration!<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>@tobyoup @LysandreJik Is there any update about MPNET in transformers?<|||||>We are actively preparing the integration codes, and will be ready these weeks.<|||||>@tobyoup Is MPNET. ready in transformers? Hello, any updates?<|||||>@tobyoup @LysandreJik Is there any update about MPNET in transformers?<|||||>~The PR for the MPNET integration has not seen any progress yet.~ @tobyoup, don't hesitate to ping us if you need help or pointers towards the integration.

There are ongoing efforts to implement MPNet on https://github.com/huggingface/transformers/pull/5804 by @StillKeepTry<|||||>Thanks for the reminder! We are adapting the codes these days and we expect to give the first PR by this weekend.

Thanks
Xu
From: Lysandre Debut <notifications@github.com>
Sent: Friday, November 13, 2020 1:27 AM
To: huggingface/transformers <transformers@noreply.github.com>
Cc: Xu Tan <Xu.Tan@microsoft.com>; Mention <mention@noreply.github.com>
Subject: Re: [huggingface/transformers] MPNet: Masked and Permuted Pre-training for Language Understanding (#4308)


The PR for the MPNET integration has not seen any progress yet. @tobyoup<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftobyoup&data=04%7C01%7CXu.Tan%40microsoft.com%7C9c1b4ed7fdb847535a2908d887302e38%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637407988352073356%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Q2NWAAxvh5LXn6d24FW7Uy4jBfs2%2BsodWvfPY3F%2BZVc%3D&reserved=0>, don't hesitate to ping us if you need help or pointers towards the integration.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fhuggingface%2Ftransformers%2Fissues%2F4308%23issuecomment-726222432&data=04%7C01%7CXu.Tan%40microsoft.com%7C9c1b4ed7fdb847535a2908d887302e38%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637407988352073356%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=fnG9MzRgWBxG4tGKsoRJd0GCSHEH8AMvpNZFYNhOcJs%3D&reserved=0>, or unsubscribe<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FALQV7L6OVO3TCAHLX3UDVVLSPQLGXANCNFSM4M6VMZPQ&data=04%7C01%7CXu.Tan%40microsoft.com%7C9c1b4ed7fdb847535a2908d887302e38%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637407988352083352%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=kFPFzyRHfpImvd55Kr33XodHoQ8HtMlT8LhC5WDf4qA%3D&reserved=0>.
<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
