More Effective LLM Compressed Tokens with Uniformly Spread Position Identifiers and Compression Loss
<|reference_start|>More Effective LLM Compressed Tokens with Uniformly Spread Position Identifiers and Compression Loss: Compressing Transformer inputs into compressd tokens allows running LLMs with improved speed and cost efficiency. Based on the compression method ICAE, we carefully examine the position identifier choices for compressed tokens and also propose a new compression loss. We demonstrate empirically that our proposed methods achieve significantly higher compression ratios (15x compared to 4x for ICAE), while being able to attain comparable reconstruction performance.<|reference_end|>
@article{zhao2024more,
    title={More Effective LLM Compressed Tokens with Uniformly Spread Position
  Identifiers and Compression Loss},
    author={Runsong Zhao, Pengcheng Huang, Xinyu Liu, Chunyang Xiao, Tong Xiao,
  Jingbo Zhu},
    journal={arXiv preprint arXiv:2409.14364},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.14364},
    primaryClass={cs.CL}
}