Answers.Body: <p>Word including Hash tag need to be tokanised. It has a special meaning and given a context. The word by itself has a different meaning.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-18T13:03:27.260Z
Answers.Text: So have they been encoded during training by models such as GPT or BERT?
Answers.UserDisplayName: user1234
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-16T16:03:53.620Z
Answers.LastActivityDate: 2021-02-16T16:03:53.620Z
Answers.UserDisplayName: raghu
Body: <p>I am exploring a potential NLP project. I was wondering what generally is done with the hashtags words (e.g. <code>#hello</code>). Are those words ignored? is the <code>#</code> removed and the word tokenised? Is it tokenised with the <code>#</code>?</p>
  
Tags: <natural-language-processing><recurrent-neural-networks><word-embedding>
Title: NLP: Are hashtags tokenised?
UserDisplayName: user1234
ThreadText: # NLP: Are hashtags tokenised?
  
  (Asked by: username_0 on 2021-02-16)
  
  I am exploring a potential NLP project. I was wondering what generally is done with the hashtags words (e.g. `#hello`). Are those words ignored? is the `#` removed and the word tokenised? Is it tokenised with the `#`?
  
  **Tags:** natural-language-processing, recurrent-neural-networks, word-embedding
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-02-16. Score: 0 | Accepted answer
  
  Word including Hash tag need to be tokanised. It has a special meaning and given a context. The word by itself has a different meaning.
  
  > **Comments:**
  > **username_0**: So have they been encoded during training by models such as GPT or BERT?  
  
  ---
  
  