Answers.Body: <p>The stage of setting the available token encodings when training LLMs is very early on. It is before, and separate to the token prediction training (the core training process for the models).</p>
  <p>It is not possible to change the tokenisation after training by any mechanism. You would need to restart the full training process from scratch, which is very time-consuming and expensive.</p>
  <p>There is no special detection or pre-processing of requests based on the type of question and passing to different &quot;modules&quot;. Whilst this is theoretically possible, the effort required would be high, and I doubt anyone has gone to lengths required to impress you with ChatGPT being good at a task that can be easily and 100% reliably done in a single line of regular code without any AI.</p>
  <p>So the answer to your question is that your random sequences are being tokenised using ChatGPT's existing tokens. It does not create any new placeholder tokens, and there is no pre-parsing to route this kind of request. Instead, it really does perform string concatenation - relatively accurately and reliably - on long strings of tokens, when requested using natural language, using the core model.</p>
  <blockquote>
  <p>In my project I have done good amount of training and it only manages to move around only little segments of the sequence</p>
  </blockquote>
  <p>For a person writing a hobby transformer project, a &quot;good amount of training&quot; is at a very different kind of scale than large companies publishing LLMs that can perform the concatenation feat amongst many other capabilities.</p>
  <p>For GPT-3 (a relatively old model, but still capable of performing the open-ended natural language concatenation task), OpenAI spent over four million dollars, and trained on 45TB of data, taking over 9 days to train on large numbers of V100 instances.</p>
  <p>If all you want to do is train a transformer model to concatenate inputs, you should be able to do it much more cheaply. If that is your goal, you could try sharing more details of your approach and results in a separate question.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-26T15:29:49.943Z
Answers.LastActivityDate: 2024-02-26T20:49:03.227Z
Answers.LastEditDate: 2024-02-26T20:49:03.227Z
Answers.UserDisplayName: Neil Slater
Body: <p>I have asked ChatGPT the following:</p>
  <blockquote>
  <p>Can you concatenate jfef9230rj2mreg90r23ewfrn02eqwdk and
  32ir20r3i2ofg90r32kee?</p>
  </blockquote>
  <p>And without any error the model produces:</p>
  <blockquote>
  <p>jfef9230rj2mreg90r23ewfrn02eqwdk32ir20r3i2ofg90r32kee</p>
  </blockquote>
  <p>My question:</p>
  <p>Does ChatGPT rawly tokenizes random sequences, or do they do some preprocessing and replace these sequences by new Tokens like:</p>
  <blockquote>
  <p>Can you concatenate SEQUENCE1 and SEQUENCE2?</p>
  </blockquote>
  <p>where SEQUENCE1 ... SEQUENCE_N is in the vocabulary, and before the Cross-Entropy on the prediction (which is hopefully SEQUENCE1||SEQUENCE2) they resubstitute back the sequences.</p>
  <p>I think the copying is quite straightforward, shouldn't a Transformer be able to learn how to copypaste/move around long sequences of bytes without any preprocessing techniques?</p>
  <p>In my project I have done good amount of training and it only manages to move around only little segments of the sequence, and I was thinking if this is even the right approach to rawly Tokenize the sequences, or if I just had too little training.</p>
  
Tags: <transformer><chatgpt>
Title: How can Transformers handle random sequences?
UserDisplayName: killertoge
ThreadText: # How can Transformers handle random sequences?
  
  (Asked by: username_0 on 2024-02-26)
  
  I have asked ChatGPT the following:
  
  > Can you concatenate jfef9230rj2mreg90r23ewfrn02eqwdk and
  > 32ir20r3i2ofg90r32kee?
  
  And without any error the model produces:
  
  > jfef9230rj2mreg90r23ewfrn02eqwdk32ir20r3i2ofg90r32kee
  
  My question:
  
  Does ChatGPT rawly tokenizes random sequences, or do they do some preprocessing and replace these sequences by new Tokens like:
  
  > Can you concatenate SEQUENCE1 and SEQUENCE2?
  
  where SEQUENCE1 ... SEQUENCE\_N is in the vocabulary, and before the Cross-Entropy on the prediction (which is hopefully SEQUENCE1||SEQUENCE2) they resubstitute back the sequences.
  
  I think the copying is quite straightforward, shouldn't a Transformer be able to learn how to copypaste/move around long sequences of bytes without any preprocessing techniques?
  
  In my project I have done good amount of training and it only manages to move around only little segments of the sequence, and I was thinking if this is even the right approach to rawly Tokenize the sequences, or if I just had too little training.
  
  **Tags:** transformer, chatgpt
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-02-26. Score: 7 | Accepted answer
  
  The stage of setting the available token encodings when training LLMs is very early on. It is before, and separate to the token prediction training (the core training process for the models).
  
  It is not possible to change the tokenisation after training by any mechanism. You would need to restart the full training process from scratch, which is very time-consuming and expensive.
  
  There is no special detection or pre-processing of requests based on the type of question and passing to different "modules". Whilst this is theoretically possible, the effort required would be high, and I doubt anyone has gone to lengths required to impress you with ChatGPT being good at a task that can be easily and 100% reliably done in a single line of regular code without any AI.
  
  So the answer to your question is that your random sequences are being tokenised using ChatGPT's existing tokens. It does not create any new placeholder tokens, and there is no pre-parsing to route this kind of request. Instead, it really does perform string concatenation - relatively accurately and reliably - on long strings of tokens, when requested using natural language, using the core model.
  
  > In my project I have done good amount of training and it only manages to move around only little segments of the sequence
  
  For a person writing a hobby transformer project, a "good amount of training" is at a very different kind of scale than large companies publishing LLMs that can perform the concatenation feat amongst many other capabilities.
  
  For GPT-3 (a relatively old model, but still capable of performing the open-ended natural language concatenation task), OpenAI spent over four million dollars, and trained on 45TB of data, taking over 9 days to train on large numbers of V100 instances.
  
  If all you want to do is train a transformer model to concatenate inputs, you should be able to do it much more cheaply. If that is your goal, you could try sharing more details of your approach and results in a separate question.
  
  
  ---
  
  