Neural Image Compression with Text-guided Encoding for both Pixel-level and Perceptual Fidelity
<|reference_start|>Neural Image Compression with Text-guided Encoding for both Pixel-level and Perceptual Fidelity: Recent advances in text-guided image compression have shown great potential to enhance the perceptual quality of reconstructed images. These methods, however, tend to have significantly degraded pixel-wise fidelity, limiting their practicality. To fill this gap, we develop a new text-guided image compression algorithm that achieves both high perceptual and pixel-wise fidelity. In particular, we propose a compression framework that leverages text information mainly by text-adaptive encoding and training with joint image-text loss. By doing so, we avoid decoding based on text-guided generative models -- known for high generative diversity -- and effectively utilize the semantic information of text at a global level. Experimental results on various datasets show that our method can achieve high pixel-level and perceptual quality, with either human- or machine-generated captions. In particular, our method outperforms all baselines in terms of LPIPS, with some room for even more improvements when we use more carefully generated captions.<|reference_end|>
@article{lee2024neural,
    title={Neural Image Compression with Text-guided Encoding for both Pixel-level
  and Perceptual Fidelity},
    author={Hagyeong Lee, Minkyu Kim, Jun-Hyuk Kim, Seungeon Kim, Dokwan Oh, Jaeho
  Lee},
    journal={arXiv preprint arXiv:2403.02944},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.02944},
    primaryClass={cs.CV cs.LG}
}