Increasing transformer token length with a Maximum Entropy Principle Method
<|reference_start|>Increasing transformer token length with a Maximum Entropy Principle Method: Transformers suffer from the computational overhead of their quadratic dependence on the length of sequences processed. We present three methods, all adding an intermediate step between training and inference/generation, which extend the autoregressive length of transformers. All rely on a Maximum Entropy Principle (MEP) whereby entropy is maximized in the presence of suitable constraints, accounted for by use of Lagrange Multipliers. These constraint methods extend the autoregressive character from T to 2T tokens in a linear-with-T fashion. There is overhead associated with this added step, but they should still be faster than the standard methods.<|reference_end|>
@article{cukier2024increasing,
    title={Increasing transformer token length with a Maximum Entropy Principle
  Method},
    author={R. I. Cukier},
    journal={arXiv preprint arXiv:2408.10277},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.10277},
    primaryClass={cs.LG}
}