Answers.Body: <p>As you have spotted, there is no particular reason to choose a specific character. However, the author of the article is operating under some constraints that I can <em>guess</em> have guided the choices:</p>
  <ul>
  <li><p>The eventual encoding will include mix of complete words with endings, word ending partials, middle of word partials and characters. For consistency, the encoding will need some way to track the difference.</p>
  </li>
  <li><p>The end of word marker is important adaptation of the more general BPE compression technique:</p>
  </li>
  </ul>
  <blockquote>
  <p>This symbol is important as it marks word boundaries, which prevents the algorithm from confusing the end of one word with the start of another.</p>
  </blockquote>
  <ul>
  <li><p>Given the need for a marker, it helps to have a value that doesn't appear in the rest of the data, within any string that needs to be represented. Here <code>_</code> is not necessarily a universal or even good choice in general. But it's fine for the data in the article.</p>
  </li>
  <li><p>It helps if readers can see the marker. That makes using a space a bit awkward, as a space can easily be overlooked. It also rules out more general choices of non-displaying character codes, which are what many systems will use in practice.</p>
  </li>
  </ul>
  <p>Using underscore to represent spaces or joins between words has precedent in a lot of software. It can be seen but doesn't visually interfere with reading the other characters as separate.</p>
  <p>In more complex datasets, where underscore (or whichever character) could appear within the strings, you have a choice of either finding a (perhaps not real) character code or byte sequence that is not in the data, or adding a special parsing rule for the end of tokens. Both of these are added complications that the article can save for another time.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-22T10:35:20.850Z
Answers.Text: Thanks for your response. Can you clear the core doubt as to why we even need to underscore? We could just not take any extra characters. Example: Instead of mentioning "low_" with underscore we could just write "low" and do the BPE algorithm.
Answers.UserDisplayName: Aamod Thakur
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-22T10:39:56.647Z
Answers.Text: @AamodThakur That is covered in my first bullet-point
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-23T06:20:53.117Z
Answers.Text: How adding an underscore at the end of the word will let us know whether the word is complete or partial. Example: "lowest_" we got some encoding like "low" and "est_" Here we know that "est_" has an underscore, but it is not a complete word.
  
  Why does encoding need to know the difference between complete words and partial words? I don't have any requirement for that; we have embeddings for all words.
Answers.UserDisplayName: Aamod Thakur
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-23T06:26:58.563Z
Answers.Text: @AamodThakur You are right, it is not "complete", but tracking the end of the word, which is still important. I will update the answer
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-23T10:00:20.210Z
Answers.Text: Thanks for your answer. Could you please let us know why we don't have a "space" token to mark the boundaries instead, just like in classical encoding techniques?
Answers.UserDisplayName: Aamod Thakur
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-23T12:01:09.513Z
Answers.Text: @AamodThakur I think a space could work fine, but that may depend on how reconstruction will work when using the tokens. For visibility of what the token represents, a space is a problem for a human viewer. It might be missed when reading. So the author may simply have chosen underscore so that they did not have to keep explaining that there was a space in certain strings.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-22T07:50:28.067Z
Answers.LastActivityDate: 2024-05-23T08:09:42.577Z
Answers.LastEditDate: 2024-05-23T08:09:42.577Z
Answers.UserDisplayName: Neil Slater
Body: <p>I have recently read an article for Byte-Pair Encoding (BPE), in which they have replaced space with an underscore(_), or we can say, adding underscore after each word/token.</p>
  <p>I did not understand the intuition for adding an underscore; we can just do it without adding an underscore.
  We can also have any other character that is not frequently used; why underscore?
  We can also add an underscore at the start of the word or both end and start, but why add to the end?</p>
  <p>Site: <a href="https://medium.com/@hsinhungw/understanding-byte-pair-encoding-fd196ebfe93f" rel="nofollow noreferrer">https://medium.com/@hsinhungw/understanding-byte-pair-encoding-fd196ebfe93f</a></p>
  
Tags: <natural-language-processing><tokenization>
Title: Why Tokenization Algorithm replace space with underscore "_"?
UserDisplayName: Aamod Thakur
ThreadText: # Why Tokenization Algorithm replace space with underscore "_"?
  
  (Asked by: username_0 on 2024-05-21)
  
  I have recently read an article for Byte-Pair Encoding (BPE), in which they have replaced space with an underscore(\_), or we can say, adding underscore after each word/token.
  
  I did not understand the intuition for adding an underscore; we can just do it without adding an underscore.
  We can also have any other character that is not frequently used; why underscore?
  We can also add an underscore at the start of the word or both end and start, but why add to the end?
  
  Site: <https://medium.com/@hsinhungw/understanding-byte-pair-encoding-fd196ebfe93f>
  
  **Tags:** natural-language-processing, tokenization
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-05-22. Score: 0 | Accepted answer
  
  As you have spotted, there is no particular reason to choose a specific character. However, the author of the article is operating under some constraints that I can *guess* have guided the choices:
  
  * The eventual encoding will include mix of complete words with endings, word ending partials, middle of word partials and characters. For consistency, the encoding will need some way to track the difference.
  * The end of word marker is important adaptation of the more general BPE compression technique:
  
  > This symbol is important as it marks word boundaries, which prevents the algorithm from confusing the end of one word with the start of another.
  
  * Given the need for a marker, it helps to have a value that doesn't appear in the rest of the data, within any string that needs to be represented. Here `_` is not necessarily a universal or even good choice in general. But it's fine for the data in the article.
  * It helps if readers can see the marker. That makes using a space a bit awkward, as a space can easily be overlooked. It also rules out more general choices of non-displaying character codes, which are what many systems will use in practice.
  
  Using underscore to represent spaces or joins between words has precedent in a lot of software. It can be seen but doesn't visually interfere with reading the other characters as separate.
  
  In more complex datasets, where underscore (or whichever character) could appear within the strings, you have a choice of either finding a (perhaps not real) character code or byte sequence that is not in the data, or adding a special parsing rule for the end of tokens. Both of these are added complications that the article can save for another time.
  
  > **Comments:**
  > **username_0**: Thanks for your response. Can you clear the core doubt as to why we even need to underscore? We could just not take any extra characters. Example: Instead of mentioning "low_" with underscore we could just write "low" and do the BPE algorithm.  
  > **username_1**: @AamodThakur That is covered in my first bullet-point  
  > **username_0**: How adding an underscore at the end of the word will let us know whether the word is complete or partial. Example: "lowest_" we got some encoding like "low" and "est_" Here we know that "est_" has an underscore, but it is not a complete word.  Why does encoding need to know the difference between complete words and partial words? I don't have any requirement for that; we have embeddings for all words.  
  > **username_1**: @AamodThakur You are right, it is not "complete", but tracking the end of the word, which is still important. I will update the answer  
  > **username_0**: Thanks for your answer. Could you please let us know why we don't have a "space" token to mark the boundaries instead, just like in classical encoding techniques?  
  > **username_1**: @AamodThakur I think a space could work fine, but that may depend on how reconstruction will work when using the tokens. For visibility of what the token represents, a space is a problem for a human viewer. It might be missed when reading. So the author may simply have chosen underscore so that they did not have to keep explaining that there was a space in certain strings.  
  
  ---
  
  