Answers.Body: <p>Generally speaking, the power of BERT for applications like NER is that the authors (of whichever implementation you use) performed a large-scale pretraining effort to create the embeddings. You can then “fine-tune” those for your specific task using far less computation, but the rub is that you need to use the same tokenization scheme (I.e. the BERT Tokenizer) in order to have your input “fit” the existing embeddings. Intuitively, tokenization is mapping a word in your text to an index number. If the embedding was trained thinking that word number 42 is “cat” then things won’t work well if you tokenize differently and provide a 43 instead when “cat” pops up in your text.</p>
  <p>Unless you’re training on a sparse language that hasn’t been well-represented by one of the public embeddings, the above is almost certainly your wisest approach. If, however you really want to train the BERT architecture on new embeddings, then you can technically use any embedding scheme you like.</p>
  <p>The BERT Tokenizer uses subwords along with a few specific administrative tokens. If you were going to explore further, a byte pair encoder might be useful, especially if the language starts to beer away from eg English.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-11T06:00:51.043Z
Answers.LastActivityDate: 2021-08-11T06:00:51.043Z
Answers.UserDisplayName: G__
Answers.Body: <p>word2vec and BERT are completely different things. You don't need to use BERT Tokenizer for training word2vec model. But, if you want to go ahead. Here's a <a href="https://towardsdatascience.com/byte-pair-encoding-the-dark-horse-of-modern-nlp-eb36c7df4f10" rel="nofollow noreferrer">link</a> which can help you.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-13T11:15:15.037Z
Answers.Text: Sorry, but I didn't ask that. My question was if I want to use BertTokenizer for a particular NLP task, say NER then is it mandatory to use BertEmbedding to generate the word embedding or I can train my own embeddings and use it along with BertTokenizer for that particular task (NER).
Answers.UserDisplayName: thenocturnalguy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-13T11:28:38.407Z
Answers.Text: It is only mandatory if you are using BERT to create the embeddings. That's what I wrote in the answer also. You first tokenize then give it to BERT for generating embeddings.
Answers.UserDisplayName: Abhishek Verma
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-13T11:31:59.803Z
Answers.Text: Oh! thank you. Can you suggest to me any link or article for generating word embedding from BERT.
Answers.UserDisplayName: thenocturnalguy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-13T11:35:38.707Z
Answers.Text: https://mccormickml.com/2019/05/14/BERT-word-embeddings-tutorial/#1-loading-pre-trained-bert
Answers.UserDisplayName: Abhishek Verma
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-13T10:55:45.403Z
Answers.LastActivityDate: 2021-03-13T10:55:45.403Z
Answers.UserDisplayName: Abhishek Verma
Body: <p>I am new to BERT and NLP and I am a little confused with tokenization and word embedding.
  My doubt is if I use the BertTokenizer for tokenizing a sentence then do I have to compulsorily use BertEmbedding for generating its corresponding word vectors of the tokens or I can train my own word2vec model to generate my word embedding while using BertTokenizer?</p>
  <p>Pardon me if this question doesn't make any sense.</p>
  
Tags: <natural-language-processing><word-embedding><bert><word2vec>
Title: Should I need to use BERT embeddings while tokenizing using BERT tokenizer?
UserDisplayName: thenocturnalguy
ThreadText: # Should I need to use BERT embeddings while tokenizing using BERT tokenizer?
  
  (Asked by: username_0 on 2021-03-13)
  
  I am new to BERT and NLP and I am a little confused with tokenization and word embedding.
  My doubt is if I use the BertTokenizer for tokenizing a sentence then do I have to compulsorily use BertEmbedding for generating its corresponding word vectors of the tokens or I can train my own word2vec model to generate my word embedding while using BertTokenizer?
  
  Pardon me if this question doesn't make any sense.
  
  **Tags:** natural-language-processing, word-embedding, bert, word2vec
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2021-08-11. Score: 2
  
  Generally speaking, the power of BERT for applications like NER is that the authors (of whichever implementation you use) performed a large-scale pretraining effort to create the embeddings. You can then “fine-tune” those for your specific task using far less computation, but the rub is that you need to use the same tokenization scheme (I.e. the BERT Tokenizer) in order to have your input “fit” the existing embeddings. Intuitively, tokenization is mapping a word in your text to an index number. If the embedding was trained thinking that word number 42 is “cat” then things won’t work well if you tokenize differently and provide a 43 instead when “cat” pops up in your text.
  
  Unless you’re training on a sparse language that hasn’t been well-represented by one of the public embeddings, the above is almost certainly your wisest approach. If, however you really want to train the BERT architecture on new embeddings, then you can technically use any embedding scheme you like.
  
  The BERT Tokenizer uses subwords along with a few specific administrative tokens. If you were going to explore further, a byte pair encoder might be useful, especially if the language starts to beer away from eg English.
  
  
  ---
  
  ### Answer by username_2 on 2021-03-13. Score: 0
  
  word2vec and BERT are completely different things. You don't need to use BERT Tokenizer for training word2vec model. But, if you want to go ahead. Here's a [link](https://towardsdatascience.com/byte-pair-encoding-the-dark-horse-of-modern-nlp-eb36c7df4f10) which can help you.
  
  > **Comments:**
  > **username_0**: Sorry, but I didn't ask that. My question was if I want to use BertTokenizer for a particular NLP task, say NER then is it mandatory to use BertEmbedding to generate the word embedding or I can train my own embeddings and use it along with BertTokenizer for that particular task (NER).  
  > **username_2**: It is only mandatory if you are using BERT to create the embeddings. That's what I wrote in the answer also. You first tokenize then give it to BERT for generating embeddings.  
  > **username_0**: Oh! thank you. Can you suggest to me any link or article for generating word embedding from BERT.  
  > **username_2**: https://mccormickml.com/2019/05/14/BERT-word-embeddings-tutorial/#1-loading-pre-trained-bert  
  
  ---
  
  