Answers.Body: <p>There is a pre-trained language model called ProphetNet for sequence-to-sequence learning with a novel self-supervised objective called future n-gram prediction.</p>
  <p><a href="https://github.com/microsoft/ProphetNet" rel="nofollow noreferrer">https://github.com/microsoft/ProphetNet</a></p>
  <p>Also, there are few variants on hugging face website as well
  <a href="https://huggingface.co/models?search=ProphetNet" rel="nofollow noreferrer">https://huggingface.co/models?search=ProphetNet</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-13T14:47:56.160Z
Answers.LastActivityDate: 2021-08-13T14:47:56.160Z
Answers.UserDisplayName: usct01
Body: <p>I know that several tokenization methods that are used for tranformer models like WordPiece for Bert and BPE for Roberta and others. What I was wondering if there is also a transformer which uses a method for tokenization similarly to the embeddings that are used in the fasttext library, so based on the summations of embeddings for the n-grams the words are made of.</p>
  <p>To me it seems weird that this way of creating word(piece) embeddings that can function as the input of a transformer isn't used in these new transformer architectures. Is there a reason why this is not tried yet? Or is this question just an result of my inability to find the right papers/repo's.</p>
  
Tags: <transformer><word-embedding><bert>
Title: Is there a pretrained (NLP) transformer that uses subword n-gram embeddings for tokenization like fasttext?
UserDisplayName: Michiel
ThreadText: # Is there a pretrained (NLP) transformer that uses subword n-gram embeddings for tokenization like fasttext?
  
  (Asked by: username_0 on 2020-11-03)
  
  I know that several tokenization methods that are used for tranformer models like WordPiece for Bert and BPE for Roberta and others. What I was wondering if there is also a transformer which uses a method for tokenization similarly to the embeddings that are used in the fasttext library, so based on the summations of embeddings for the n-grams the words are made of.
  
  To me it seems weird that this way of creating word(piece) embeddings that can function as the input of a transformer isn't used in these new transformer architectures. Is there a reason why this is not tried yet? Or is this question just an result of my inability to find the right papers/repo's.
  
  **Tags:** transformer, word-embedding, bert
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-08-13. Score: 2
  
  There is a pre-trained language model called ProphetNet for sequence-to-sequence learning with a novel self-supervised objective called future n-gram prediction.
  
  <https://github.com/microsoft/ProphetNet>
  
  Also, there are few variants on hugging face website as well
  <https://huggingface.co/models?search=ProphetNet>
  
  
  ---
  
  