title: T-FREE: Tokenizer-Free Generative LLMs via Sparse Representations for
    Memory-Efficient Embeddings
summary: Tokenizers are crucial for encoding information in Large Language Models, but
  their development has recently stagnated, and they contain inherent weaknesses.
  Major limitations include computational overhead, ineffective vocabulary use,
  and unnecessarily large embedding and head layers. Additionally, their
  performance is biased towards a reference corpus, leading to reduced
  effectiveness for underrepresented languages.
    To remedy these issues, we propose T-FREE, which directly embeds words
  through sparse activation patterns over character triplets, and does not
  require a reference corpus. T-FREE inherently exploits morphological
  similarities and allows for strong compression of embedding layers. In our
  exhaustive experimental evaluation, we achieve competitive downstream
  performance with a parameter reduction of more than 85% on these layers.
  Further, T-FREE shows significant improvements in cross-lingual transfer
  learning.
paper_authors._id: 667e76ecf0754a8a1d55471d
paper_authors.name: Björn Deiseroth
paper_authors._id: 667e76ecf0754a8a1d55471e
paper_authors.name: Manuel Brack
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-30T20:57:29.924Z
paper_authors._id: 62fa1d95e8c9c532aa75331c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62fa1d95e8c9c532aa75331c/WFfk_n8gOj845pSkfdazA.jpeg
paper_authors.fullname: Manuel Brack
paper_authors.type: user
paper_authors.user: mbrack
paper_authors._id: 667e76ecf0754a8a1d55471f
paper_authors.name: Patrick Schramowski
paper_authors._id: 667e76ecf0754a8a1d554720
paper_authors.name: Kristian Kersting
paper_authors._id: 667e76ecf0754a8a1d554721
paper_authors.name: Samuel Weinbach
paper_title: T-FREE: Tokenizer-Free Generative LLMs via Sparse Representations for
    Memory-Efficient Embeddings
paper_summary: Tokenizers are crucial for encoding information in Large Language Models, but
  their development has recently stagnated, and they contain inherent weaknesses.
  Major limitations include computational overhead, ineffective vocabulary use,
  and unnecessarily large embedding and head layers. Additionally, their
  performance is biased towards a reference corpus, leading to reduced
  effectiveness for underrepresented languages.
    To remedy these issues, we propose T-FREE, which directly embeds words
  through sparse activation patterns over character triplets, and does not
  require a reference corpus. T-FREE inherently exploits morphological
  similarities and allows for strong compression of embedding layers. In our
  exhaustive experimental evaluation, we achieve competitive downstream
  performance with a parameter reduction of more than 85% on these layers.
  Further, T-FREE shows significant improvements in cross-lingual transfer
  learning.
paper_ai_keywords: Tokenizers Large Language Models computational overhead ineffective vocabulary use embedding layers head layers performance reference corpus T-FREE sparse activation patterns character triplets morphological similarities cross-lingual transfer learning parameter reduction downstream performance