From a Lossless (~15:1) Compression Algorithm for Llama2 7B Weights to Variable Precision, Variable Range, Compressed Numeric Data Types for CNNs and LLMs
<|reference_start|>From a Lossless (~15:1) Compression Algorithm for Llama2 7B Weights to Variable Precision, Variable Range, Compressed Numeric Data Types for CNNs and LLMs: This paper starts with a simple lossless ~1.5:1 compression algorithm for the weights of the Large Language Model (LLM) Llama2 7B [1] that can be implemented in ~200 LUTs in AMD FPGAs, processing over 800 million bfloat16 numbers per second. This framework is then extended to variable precision, variable range, compressed numerical data types that are a user defined super set of both floats and posits [2]. The paper then discusses a simple hardware implementation of such format based on ANS (Asymmetrical Numeral Systems) [3] that acts as a bridge between this flexible data format and a computational engine while, at the same time, achieving bandwidth reduction. An example of a token factory using weight compression and sharing is also given.<|reference_end|>
@article{liguori2024from,
    title={From a Lossless (~1.5:1) Compression Algorithm for Llama2 7B Weights to
  Variable Precision, Variable Range, Compressed Numeric Data Types for CNNs
  and LLMs},
    author={Vincenzo Liguori},
    journal={arXiv preprint arXiv:2404.10896},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.10896},
    primaryClass={cs.CV cs.AI cs.AR}
}