Answers.Body: <pre><code>def tokenize_text(text):
      return re.findall(r'\S+|\n|\s', text.replace('\n\n', '\n \n'))
  </code></pre>
  <p>This gives the tokenization of text, including space and new-line</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-12T19:58:01.617Z
Answers.LastActivityDate: 2023-02-12T19:58:01.617Z
Answers.UserDisplayName: Arjun Reddy
Body: <p>I want to tokenize a text and also want to consider space and new line as separate tokens:</p>
  <p>if there are multiple spaces, then tokenize them separately, for example, &quot;Hi, I am\n  Teddy&quot; this should tokenize as [&quot;Hi&quot;,&quot; &quot;, &quot;,&quot;,&quot; &quot;,&quot;I&quot;,&quot; &quot;,&quot;am&quot;,&quot;\n&quot;,&quot; &quot;,&quot; &quot;,&quot;Teddy&quot;]</p>
  <p><strong>Note: That After new-line (\n) in text there are 2 spaces</strong></p>
  
Tags: <natural-language-processing>
Title: I want to tokenize a text and also want to consider space and new line as separate tokens: for NLP Task
UserDisplayName: Arjun Reddy
ThreadText: # I want to tokenize a text and also want to consider space and new line as separate tokens: for NLP Task
  
  (Asked by: username_0 on 2023-02-12)
  
  I want to tokenize a text and also want to consider space and new line as separate tokens:
  
  if there are multiple spaces, then tokenize them separately, for example, "Hi, I am\n Teddy" this should tokenize as ["Hi"," ", ","," ","I"," ","am","\n"," "," ","Teddy"]
  
  **Note: That After new-line (\n) in text there are 2 spaces**
  
  **Tags:** natural-language-processing
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2023-02-12. Score: 1
  
  def tokenize_text(text):
      return re.findall(r'\S+|\n|\s', text.replace('\n\n', '\n \n'))
  
  This gives the tokenization of text, including space and new-line
  
  
  ---
  
  