Answers.Body: <p>Obviously the decoder-only transformers like GPT or CTRL models are more naturally suited for generative autocorrection tasks using BPE like subword tokenization methods along with their vast training corpus containing the correct titles of most songs and movies, or at least most of them are correct, as further explained <a href="https://ai.stackexchange.com/questions/47210/does-training-llm-on-very-high-quality-data-might-lead-to-bad-performance-on-mis/47213#47213">here</a>.</p>
  <p>Encoder transformers like RoBERTa can also be used for typo detection and correction, alibeit in different ways. One possible way is to first prepare a RAG like vector database with embeddings of correct titles of songs and movies, etc, then use the pretrained encoder's output embedding of your input title with typos to do a similarity search across the vector database to find the closest match. If your vector database is large and sufficient, it should perform quite well for most popular songs or movies.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-15T19:24:44.703Z
Answers.LastActivityDate: 2024-11-15T19:24:44.703Z
Answers.UserDisplayName: cinch
Body: <p>Can typos be detected only using &quot;decoder only&quot; transformers, or in &quot;encoder&quot; transformers as well (that lack the generative part)? This would assume typos in titles of songs, movies, video games, etc. that are autocorrected in a way similar in how search engines correct typos in these names.</p>
  
Tags: <transformer>
Title: How transformers can be used to fix typos?
UserDisplayName: Hephaestus
ThreadText: # How transformers can be used to fix typos?
  
  (Asked by: username_0 on 2024-11-15)
  
  Can typos be detected only using "decoder only" transformers, or in "encoder" transformers as well (that lack the generative part)? This would assume typos in titles of songs, movies, video games, etc. that are autocorrected in a way similar in how search engines correct typos in these names.
  
  **Tags:** transformer
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-11-15. Score: 0 | Accepted answer
  
  Obviously the decoder-only transformers like GPT or CTRL models are more naturally suited for generative autocorrection tasks using BPE like subword tokenization methods along with their vast training corpus containing the correct titles of most songs and movies, or at least most of them are correct, as further explained [here](https://ai.stackexchange.com/questions/47210/does-training-llm-on-very-high-quality-data-might-lead-to-bad-performance-on-mis/47213#47213).
  
  Encoder transformers like RoBERTa can also be used for typo detection and correction, alibeit in different ways. One possible way is to first prepare a RAG like vector database with embeddings of correct titles of songs and movies, etc, then use the pretrained encoder's output embedding of your input title with typos to do a similarity search across the vector database to find the closest match. If your vector database is large and sufficient, it should perform quite well for most popular songs or movies.
  
  
  ---
  
  