Language Models as Zero-shot Lossless Gradient Compressors: Towards General Neural Parameter Prior Models
<|reference_start|>Language Models as Zero-shot Lossless Gradient Compressors: Towards General Neural Parameter Prior Models: Despite the widespread use of statistical prior models in various fields, such models for neural network gradients have long been overlooked. The inherent challenge stems from their high-dimensional structures and complex interdependencies, which complicate effective modeling. In this work, we demonstrate the potential of large language models (LLMs) to act as gradient priors in a zero-shot setting. We examine the property by considering lossless gradient compression -- a critical application in distributed learning -- that depends heavily on precise probability modeling. To achieve this, we introduce LM-GC, a novel method that integrates LLMs with arithmetic coding. Our technique converts plain gradients into text-like formats, enhancing token efficiency by up to 38 times compared to their plain representations. We ensure that this data conversion maintains a close alignment with the structure of plain gradients and the symbols commonly recognized by LLMs. Our experiments indicate that LM-GC surpasses existing state-of-the-art lossless compression methods, improving compression rates by 10\% up to 17.2\% across various datasets and architectures. Additionally, our approach shows promising compatibility with lossy compression techniques such as quantization and sparsification. These findings highlight the significant potential of LLMs as a model for effectively handling gradients. We will release the source code upon publication.<|reference_end|>
@article{wang2024language,
    title={Language Models as Zero-shot Lossless Gradient Compressors: Towards
  General Neural Parameter Prior Models},
    author={Hui-Po Wang, Mario Fritz},
    journal={arXiv preprint arXiv:2409.17836},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.17836},
    primaryClass={cs.LG cs.AI}
}