title: Cramming 1568 Tokens into a Single Vector and Back Again: Exploring the
    Limits of Embedding Space Capacity
summary: A range of recent works addresses the problem of compression of sequence of
  tokens into a shorter sequence of real-valued vectors to be used as inputs
  instead of token embeddings or key-value cache. These approaches allow to
  reduce the amount of compute in existing language models. Despite relying on
  powerful models as encoders, the maximum attainable lossless compression ratio
  is typically not higher than x10. This fact is highly intriguing because, in
  theory, the maximum information capacity of large real-valued vectors is far
  beyond the presented rates even for 16-bit precision and a modest vector size.
  In this work, we explore the limits of compression by replacing the encoder
  with a per-sample optimization procedure. We show that vectors with compression
  ratios up to x1500 exist, which highlights two orders of magnitude gap between
  existing and practically attainable solutions. Furthermore, we empirically show
  that the compression limits are determined not by the length of the input but
  by the amount of uncertainty to be reduced, namely, the cross-entropy loss on
  this sequence without any conditioning. The obtained limits highlight the
  substantial gap between the theoretical capacity of input embeddings and their
  practical utilization, suggesting significant room for optimization in model
  design.
paper_authors._id: 67b5a7896f72266cb765e744
paper_authors.name: Yuri Kuratov
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-06-06T17:43:24.656Z
paper_authors._id: 618b9540682ec1c38327e586
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/618b9540682ec1c38327e586/v_ZBkfh8O9Zh6C2YQpuBX.jpeg
paper_authors.fullname: Yury Kuratov
paper_authors.type: user
paper_authors.user: yurakuratov
paper_authors._id: 67b5a7896f72266cb765e745
paper_authors.name: Mikhail Arkhipov
paper_authors._id: 67b5a7896f72266cb765e746
paper_authors.name: Aydar Bulatov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-21T14:43:06.400Z
paper_authors._id: 64c8b321cb2f1bf0e7c0f54b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64c8b321cb2f1bf0e7c0f54b/JflXxMVnG9I0IB5YNyhXF.jpeg
paper_authors.fullname: Aydar Bulatov
paper_authors.type: user
paper_authors.user: booydar
paper_authors._id: 67b5a7896f72266cb765e747
paper_authors.name: Mikhail Burtsev
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-19T09:56:59.080Z
paper_authors._id: 639c6e978a34ed9a404c6a7b
paper_authors.avatarUrl: /avatars/c98ca8c9f9ed8509c2f1bb6aa994fd57.svg
paper_authors.fullname: MIKHAIL BURTSEV
paper_authors.type: user
paper_authors.user: mbur
paper_title: Cramming 1568 Tokens into a Single Vector and Back Again: Exploring the
    Limits of Embedding Space Capacity
paper_summary: A range of recent works addresses the problem of compression of sequence of
  tokens into a shorter sequence of real-valued vectors to be used as inputs
  instead of token embeddings or key-value cache. These approaches allow to
  reduce the amount of compute in existing language models. Despite relying on
  powerful models as encoders, the maximum attainable lossless compression ratio
  is typically not higher than x10. This fact is highly intriguing because, in
  theory, the maximum information capacity of large real-valued vectors is far
  beyond the presented rates even for 16-bit precision and a modest vector size.
  In this work, we explore the limits of compression by replacing the encoder
  with a per-sample optimization procedure. We show that vectors with compression
  ratios up to x1500 exist, which highlights two orders of magnitude gap between
  existing and practically attainable solutions. Furthermore, we empirically show
  that the compression limits are determined not by the length of the input but
  by the amount of uncertainty to be reduced, namely, the cross-entropy loss on
  this sequence without any conditioning. The obtained limits highlight the
  substantial gap between the theoretical capacity of input embeddings and their
  practical utilization, suggesting significant room for optimization in model
  design.
paper_ai_summary: Using per-sample optimization, compression ratios of up to x1500 are achieved for sequence-to-vector compression in language models, highlighting a large gap between theoretical and practical limits.
paper_ai_keywords: sequence compression token embeddings per-sample optimization compression ratio cross-entropy loss model design