MRIR: Integrating Multimodal Insights for Diffusion-based Realistic Image Restoration
<|reference_start|>MRIR: Integrating Multimodal Insights for Diffusion-based Realistic Image Restoration: Realistic image restoration is a crucial task in computer vision, and the use of diffusion-based models for image restoration has garnered significant attention due to their ability to produce realistic results. However, the quality of the generated images is still a significant challenge due to the severity of image degradation and the uncontrollability of the diffusion model. In this work, we delve into the potential of utilizing pre-trained stable diffusion for image restoration and propose MRIR, a diffusion-based restoration method with multimodal insights. Specifically, we explore the problem from two perspectives: textual level and visual level. For the textual level, we harness the power of the pre-trained multimodal large language model to infer meaningful semantic information from low-quality images. Furthermore, we employ the CLIP image encoder with a designed Refine Layer to capture image details as a supplement. For the visual level, we mainly focus on the pixel level control. Thus, we utilize a Pixel-level Processor and ControlNet to control spatial structures. Finally, we integrate the aforementioned control information into the denoising U-Net using multi-level attention mechanisms and realize controllable image restoration with multimodal insights. The qualitative and quantitative results demonstrate our method's superiority over other state-of-the-art methods on both synthetic and real-world datasets.<|reference_end|>
@article{zhang2024mrir:,
    title={MRIR: Integrating Multimodal Insights for Diffusion-based Realistic
  Image Restoration},
    author={Yuhong Zhang, Hengsheng Zhang, Xinning Chai, Rong Xie, Li Song, Wenjun
  Zhang},
    journal={arXiv preprint arXiv:2407.03635},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.03635},
    primaryClass={cs.CV}
}