NeuZip: Memory-Efficient Training and Inference with Dynamic Compression
  of Neural Networks
The performance of neural networks improves when more parameters are used.
However, the model sizes are constrained by the available on-device memory
during training and inference. Although applying techniques like quantization
can alleviate the constraint, they suffer from performance degradation. In this
work, we introduce NeuZip, a new weight compression scheme based on the entropy
of floating-point numbers in neural networks. With NeuZip, we are able to
achieve memory-efficient training and inference without sacrificing
performance. Notably, we significantly reduce the memory footprint of training
a Llama-3 8B model from 31GB to less than 16GB, while keeping the training
dynamics fully unchanged. In inference, our method can reduce memory usage by
more than half while maintaining near-lossless performance. Our code is
publicly available.
672549dbb8abe375fe499cf2
Yongchang Hao
extracted_confirmed
2024-11-01T21:38:33.040Z
62d5fc0a34e211202759496f
/avatars/5ed4d2531e33129f142adf0277b094a6.svg
Yongchang Hao
user
yongchanghao
672549dbb8abe375fe499cf3
Yanshuai Cao
672549dbb8abe375fe499cf4
Lili Mou
NeuZip: Memory-Efficient Training and Inference with Dynamic Compression
  of Neural Networks
The performance of neural networks improves when more parameters are used.
However, the model sizes are constrained by the available on-device memory
during training and inference. Although applying techniques like quantization
can alleviate the constraint, they suffer from performance degradation. In this
work, we introduce NeuZip, a new weight compression scheme based on the entropy
of floating-point numbers in neural networks. With NeuZip, we are able to
achieve memory-efficient training and inference without sacrificing
performance. Notably, we significantly reduce the memory footprint of training
a Llama-3 8B model from 31GB to less than 16GB, while keeping the training
dynamics fully unchanged. In inference, our method can reduce memory usage by
more than half while maintaining near-lossless performance. Our code is
publicly available.
NeuZip compresses neural network weights using entropy for memory-efficient training and inference without performance loss.
NeuZip entropy weight compression Llama-3 memory footprint training dynamics near-lossless performance