Answers.Body: <blockquote>
  <p>For LLM, how was decided how big and how many unique tokens exist for the English language?</p>
  </blockquote>
  <p>Empirically. E.g., see some info from <a href="https://arxiv.org/pdf/2310.08754" rel="nofollow noreferrer">Tokenizer Choice For LLM Training: Negligible or Crucial?</a>:</p>
  <blockquote>
  <p>Analyzing the impact of the vocabulary size revealed that in the monolingual English setting, the
  smaller/medium-sized, i.e., a vocabulary size of
  33k/50k performs better (Table 5) whereas in the
  multilingual setting, in all cases except for German,
  larger vocabulary sizes result in better downstream
  performance. Taking into account the results presented in Table 3 showing that in the monolingual
  English setting, the best-performing tokenizer on
  average across all tasks had a vocabulary size of
  33k and that the best-performing multilingual tokenizer had a vocabulary size of 100k additionally
  supports the observation that for the monolingual
  English setting a small vocabulary size is beneficial
  and for the multilingual setting a large vocabulary
  size is required.</p>
  <p>Our findings in Fig. 3 show that increasing the vocabulary
  size from 50k to larger vocabulary sizes increases
  the computational costs in all cases.
  <a href="https://i.sstatic.net/3dkm4jlD.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/3dkm4jlD.png" alt="enter image description here" /></a></p>
  <p><a href="https://i.sstatic.net/oBWUR7A4.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/oBWUR7A4.png" alt="enter image description here" /></a></p>
  <p><a href="https://i.sstatic.net/iOiqi6j8.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/iOiqi6j8.png" alt="enter image description here" /></a></p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-07T05:21:27.277Z
Answers.LastActivityDate: 2024-10-07T05:21:27.277Z
Answers.UserDisplayName: Franck Dernoncourt
Body: <p>For LLM, how was decided how big and how many unique tokens exist for the english language?</p>
  
Tags: <large-language-models><tokenization>
Title: For LLM how is the token dictionary built?
UserDisplayName: Jack Miles
ThreadText: # For LLM how is the token dictionary built?
  
  (Asked by: username_0 on 2024-09-27)
  
  For LLM, how was decided how big and how many unique tokens exist for the english language?
  
  **Tags:** large-language-models, tokenization
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-10-07. Score: 3
  
  > For LLM, how was decided how big and how many unique tokens exist for the English language?
  
  Empirically. E.g., see some info from [Tokenizer Choice For LLM Training: Negligible or Crucial?](https://arxiv.org/pdf/2310.08754):
  
  > Analyzing the impact of the vocabulary size revealed that in the monolingual English setting, the
  > smaller/medium-sized, i.e., a vocabulary size of
  > 33k/50k performs better (Table 5) whereas in the
  > multilingual setting, in all cases except for German,
  > larger vocabulary sizes result in better downstream
  > performance. Taking into account the results presented in Table 3 showing that in the monolingual
  > English setting, the best-performing tokenizer on
  > average across all tasks had a vocabulary size of
  > 33k and that the best-performing multilingual tokenizer had a vocabulary size of 100k additionally
  > supports the observation that for the monolingual
  > English setting a small vocabulary size is beneficial
  > and for the multilingual setting a large vocabulary
  > size is required.
  >
  > Our findings in Fig. 3 show that increasing the vocabulary
  > size from 50k to larger vocabulary sizes increases
  > the computational costs in all cases.
  > [![enter image description here](https://i.sstatic.net/3dkm4jlD.png)](https://i.sstatic.net/3dkm4jlD.png)
  >
  > [![enter image description here](https://i.sstatic.net/oBWUR7A4.png)](https://i.sstatic.net/oBWUR7A4.png)
  >
  > [![enter image description here](https://i.sstatic.net/iOiqi6j8.png)](https://i.sstatic.net/iOiqi6j8.png)
  
  
  ---
  
  