Answers.Body: <p>Indeed several studies provide evidence that the design of tokenizers significantly impacts transformer model performance such as the original BERT <a href="https://arxiv.org/abs/2012.15524" rel="nofollow noreferrer">paper</a> that introduced WordPiece tokenization algorithm and demonstrated how splitting words into meaningful subwords improved performance across a wide range of NLP tasks.</p>
  <blockquote>
  <p>Tokenization is a fundamental preprocessing step for almost all NLP tasks. In this paper, we propose efficient algorithms for the WordPiece tokenization used in BERT, from single-word tokenization to general text (e.g., sentence) tokenization.... As a midpoint between words and characters, subword units retain linguistic meaning (like morphemes), while alleviating out-of-vocabulary situations even with a relatively
  small-size vocabulary.</p>
  </blockquote>
  <p>Therefore without subword tokenization, an extremely powerful transformer cannot deal with out-of-vocabulary (OOV) word well such as 'rel8tivity'. On the other hand, if it's too granular the resulting artificially increased sequence length could likely make training much more computationally expensive and potentially losing semantic coherence. The graph shape you concerned might more or less look like a bell curve with some optimally performed tokenization scheme in between such as WordPiece (BERT) and BPE (GPT).</p>
  <p>Finally to claim that <em>tokenizer algorithm (which relied on rule-based approach instead of blackbox) is the most important than tuning the hyperparameter of the blackbox (transformer)</em> is not objective, since tokenization schemes belong to the traditional NLP and computational linguistics arsenals and without transformers they cannot produce same level of  current LLM's performance, if with other traditional language models such as n-grams.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-07T23:39:02.883Z
Answers.LastActivityDate: 2024-12-07T23:39:02.883Z
Answers.UserDisplayName: cinch
Body: <p>Any there reference that saying having many sub word or token or part of sentence or term are generally good for transformer model performance?</p>
  <p>If so, then the tokenizer algorithm (which relied on rule-based approach instead of blackbox) is the most importance than tuning the hyperparameter of the blackbox (transformer).</p>
  <p>What is the graphic looks like if it's plotted into visualization of the relationship between number of tokens (x-axis) and the model performance (y-axis)?</p>
  
Tags: <natural-language-processing><transformer><word-embedding><performance>
Title: Is having many tokens generally good for NLP performance?
UserDisplayName: Muhammad Ikhwan Perwira
ThreadText: # Is having many tokens generally good for NLP performance?
  
  (Asked by: username_0 on 2024-12-07)
  
  Any there reference that saying having many sub word or token or part of sentence or term are generally good for transformer model performance?
  
  If so, then the tokenizer algorithm (which relied on rule-based approach instead of blackbox) is the most importance than tuning the hyperparameter of the blackbox (transformer).
  
  What is the graphic looks like if it's plotted into visualization of the relationship between number of tokens (x-axis) and the model performance (y-axis)?
  
  **Tags:** natural-language-processing, transformer, word-embedding, performance
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-12-07. Score: 1 | Accepted answer
  
  Indeed several studies provide evidence that the design of tokenizers significantly impacts transformer model performance such as the original BERT [paper](https://arxiv.org/abs/2012.15524) that introduced WordPiece tokenization algorithm and demonstrated how splitting words into meaningful subwords improved performance across a wide range of NLP tasks.
  
  > Tokenization is a fundamental preprocessing step for almost all NLP tasks. In this paper, we propose efficient algorithms for the WordPiece tokenization used in BERT, from single-word tokenization to general text (e.g., sentence) tokenization.... As a midpoint between words and characters, subword units retain linguistic meaning (like morphemes), while alleviating out-of-vocabulary situations even with a relatively
  > small-size vocabulary.
  
  Therefore without subword tokenization, an extremely powerful transformer cannot deal with out-of-vocabulary (OOV) word well such as 'rel8tivity'. On the other hand, if it's too granular the resulting artificially increased sequence length could likely make training much more computationally expensive and potentially losing semantic coherence. The graph shape you concerned might more or less look like a bell curve with some optimally performed tokenization scheme in between such as WordPiece (BERT) and BPE (GPT).
  
  Finally to claim that *tokenizer algorithm (which relied on rule-based approach instead of blackbox) is the most important than tuning the hyperparameter of the blackbox (transformer)* is not objective, since tokenization schemes belong to the traditional NLP and computational linguistics arsenals and without transformers they cannot produce same level of current LLM's performance, if with other traditional language models such as n-grams.
  
  
  ---
  
  