LLMZip: Lossless Text Compression using Large Language Models
We provide new estimates of an asymptotic upper bound on the entropy of
English using the large language model LLaMA-7B as a predictor for the next
token given a window of past tokens. This estimate is significantly smaller
than currently available estimates in cover1978convergent,
lutati2023focus. A natural byproduct is an algorithm for lossless
compression of English text which combines the prediction from the large
language model with a lossless compression scheme. Preliminary results from
limited experiments suggest that our scheme outperforms state-of-the-art text
compression schemes such as BSC, ZPAQ, and paq8h.
648146dd856901b0edbd9824
Chandra Shekhara Kaushik Valmeekam
claimed_verified
2023-06-09T15:32:25.325Z
639533db00e782e0c0c95ab3
/avatars/b7d4e2dbf6261d3442cd9646102c0f5d.svg
Kaushik Valmeekam
user
vcskaushik9
648146dd856901b0edbd9825
Krishna Narayanan
648146dd856901b0edbd9826
Dileep Kalathil
648146dd856901b0edbd9827
Jean-Francois Chamberland
648146dd856901b0edbd9828
Srinivas Shakkottai
LLMZip: Lossless Text Compression using Large Language Models
We provide new estimates of an asymptotic upper bound on the entropy of
English using the large language model LLaMA-7B as a predictor for the next
token given a window of past tokens. This estimate is significantly smaller
than currently available estimates in cover1978convergent,
lutati2023focus. A natural byproduct is an algorithm for lossless
compression of English text which combines the prediction from the large
language model with a lossless compression scheme. Preliminary results from
limited experiments suggest that our scheme outperforms state-of-the-art text
compression schemes such as BSC, ZPAQ, and paq8h.
A new estimate of English entropy using LLaMA-7B for next-token prediction leads to a compression algorithm that outperforms current text compression methods.
large language model LLaMA-7B next-token prediction entropy estimation lossless compression BSC ZPAQ paq8h