<pre><code>def tokenize_text(text):
    return re.findall(r'\S+|\n|\s', text.replace('\n\n', '\n \n'))
</code></pre>
<p>This gives the tokenization of text, including space and new-line</p>

CC BY-SA 4.0
2023-02-12T19:58:01.617Z
2023-02-12T19:58:01.617Z
Arjun Reddy
<p>I want to tokenize a text and also want to consider space and new line as separate tokens:</p>
<p>if there are multiple spaces, then tokenize them separately, for example, &quot;Hi, I am\n  Teddy&quot; this should tokenize as [&quot;Hi&quot;,&quot; &quot;, &quot;,&quot;,&quot; &quot;,&quot;I&quot;,&quot; &quot;,&quot;am&quot;,&quot;\n&quot;,&quot; &quot;,&quot; &quot;,&quot;Teddy&quot;]</p>
<p><strong>Note: That After new-line (\n) in text there are 2 spaces</strong></p>

<natural-language-processing>
I want to tokenize a text and also want to consider space and new line as separate tokens: for NLP Task
Arjun Reddy
# I want to tokenize a text and also want to consider space and new line as separate tokens: for NLP Task

(Asked by: username_0 on 2023-02-12)

I want to tokenize a text and also want to consider space and new line as separate tokens:

if there are multiple spaces, then tokenize them separately, for example, "Hi, I am\n Teddy" this should tokenize as ["Hi"," ", ","," ","I"," ","am","\n"," "," ","Teddy"]

**Note: That After new-line (\n) in text there are 2 spaces**

**Tags:** natural-language-processing


---

## 1 answers

### Answer by username_0 on 2023-02-12. Score: 1

def tokenize_text(text):
    return re.findall(r'\S+|\n|\s', text.replace('\n\n', '\n \n'))

This gives the tokenization of text, including space and new-line


---

