MISC: Ultra-low Bitrate Image Semantic Compression Driven by Large Multimodal Model
<|reference_start|>MISC: Ultra-low Bitrate Image Semantic Compression Driven by Large Multimodal Model: With the evolution of storage and communication protocols, ultra-low bitrate image compression has become a highly demanding topic. However, existing compression algorithms must sacrifice either consistency with the ground truth or perceptual quality at ultra-low bitrate. In recent years, the rapid development of the Large Multimodal Model (LMM) has made it possible to balance these two goals. To solve this problem, this paper proposes a method called Multimodal Image Semantic Compression (MISC), which consists of an LMM encoder for extracting the semantic information of the image, a map encoder to locate the region corresponding to the semantic, an image encoder generates an extremely compressed bitstream, and a decoder reconstructs the image based on the above information. Experimental results show that our proposed MISC is suitable for compressing both traditional Natural Sense Images (NSIs) and emerging AI-Generated Images (AIGIs) content. It can achieve optimal consistency and perception results while saving 50% bitrate, which has strong potential applications in the next generation of storage and communication. The code will be released on https://github.com/lcysyzxdxc/MISC.<|reference_end|>
@article{li2024misc:,
    title={MISC: Ultra-low Bitrate Image Semantic Compression Driven by Large
  Multimodal Model},
    author={Chunyi Li, Guo Lu, Donghui Feng, Haoning Wu, Zicheng Zhang, Xiaohong
  Liu, Guangtao Zhai, Weisi Lin, Wenjun Zhang},
    journal={arXiv preprint arXiv:2402.16749},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.16749},
    primaryClass={cs.CV cs.AI eess.IV}
}