title: NeuZip: Memory-Efficient Training and Inference with Dynamic Compression
    of Neural Networks
summary: The performance of neural networks improves when more parameters are used.
  However, the model sizes are constrained by the available on-device memory
  during training and inference. Although applying techniques like quantization
  can alleviate the constraint, they suffer from performance degradation. In this
  work, we introduce NeuZip, a new weight compression scheme based on the entropy
  of floating-point numbers in neural networks. With NeuZip, we are able to
  achieve memory-efficient training and inference without sacrificing
  performance. Notably, we significantly reduce the memory footprint of training
  a Llama-3 8B model from 31GB to less than 16GB, while keeping the training
  dynamics fully unchanged. In inference, our method can reduce memory usage by
  more than half while maintaining near-lossless performance. Our code is
  publicly available.
paper_authors._id: 672549dbb8abe375fe499cf2
paper_authors.name: Yongchang Hao
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-11-01T21:38:33.040Z
paper_authors._id: 62d5fc0a34e211202759496f
paper_authors.avatarUrl: /avatars/5ed4d2531e33129f142adf0277b094a6.svg
paper_authors.fullname: Yongchang Hao
paper_authors.type: user
paper_authors.user: yongchanghao
paper_authors._id: 672549dbb8abe375fe499cf3
paper_authors.name: Yanshuai Cao
paper_authors._id: 672549dbb8abe375fe499cf4
paper_authors.name: Lili Mou
paper_title: NeuZip: Memory-Efficient Training and Inference with Dynamic Compression
    of Neural Networks
paper_summary: The performance of neural networks improves when more parameters are used.
  However, the model sizes are constrained by the available on-device memory
  during training and inference. Although applying techniques like quantization
  can alleviate the constraint, they suffer from performance degradation. In this
  work, we introduce NeuZip, a new weight compression scheme based on the entropy
  of floating-point numbers in neural networks. With NeuZip, we are able to
  achieve memory-efficient training and inference without sacrificing
  performance. Notably, we significantly reduce the memory footprint of training
  a Llama-3 8B model from 31GB to less than 16GB, while keeping the training
  dynamics fully unchanged. In inference, our method can reduce memory usage by
  more than half while maintaining near-lossless performance. Our code is
  publicly available.
paper_ai_summary: NeuZip compresses neural network weights using entropy for memory-efficient training and inference without performance loss.
paper_ai_keywords: NeuZip entropy weight compression Llama-3 memory footprint training dynamics near-lossless performance