JarvisIR: Elevating Autonomous Driving Perception with Intelligent Image Restoration

Model Description

JarvisIR is a novel system that leverages a Vision-Language Model (VLM) to intelligently restore images for autonomous driving perception in adverse weather. It acts as a central controller, dynamically coordinating multiple expert restoration models to tackle complex degradations such as rain, fog, low-light, and snow.

Key Features

VLM Controller: The first framework to employ a Vision-Language Model for orchestrating image restoration workflows.
Multi-Expert Coordination: Dynamically schedules specialized restoration models for tasks like denoising, super-resolution, and deraining.
Adaptive Restoration: Effectively handles a wide range of adverse weather conditions, including night/low-light, rain, fog, and snow.
Advanced Training Strategy: Utilizes a two-stage process of Supervised Fine-Tuning (SFT) followed by alignment with Mixed-Rank Reward-based Human Feedback (MRRHF).

Model Architecture

The system comprises three core components:

VLM Controller: A LLaVA-v1.5-7B model serves as the core for task planning and expert model selection.
Expert Models: A suite of specialized networks, each tailored for a specific restoration task (e.g., deraining, defogging).
Reward Models: A set of Image Quality Assessment (IQA) models that provide feedback for quality assessment and alignment during training.

Training Data

JarvisIR was trained on a large-scale, comprehensive dataset:

CleanBench-Synthetic: A dataset of 150,000 synthetically degraded images with corresponding annotations.
CleanBench-Real: A collection of 80,000 real-world images captured in adverse weather, used for alignment training.
Comprehensive Coverage: The data covers four primary weather scenarios (night, rain, fog, snow) with various combinations of degradations.

Performance

Achieves a 50% average improvement in perception metrics on the CleanBench-Real dataset compared to state-of-the-art all-in-one methods.
Demonstrates superior performance across all tested weather conditions.
Exhibits enhanced robustness and generalization capabilities in real-world driving scenarios.

Intended Use

Primary Use Cases:

Enhancing perception systems in autonomous vehicles.
Building robust, multi-weather image restoration pipelines.
Advancing research into the applications of Vision-Language Models in image processing.

Model Checkpoints

This repository provides the following model weights:

pertained: The complete model after both Supervised Fine-Tuning and MRRHF alignment stages.
agent-tools/: The weights for each individual expert restoration model.

Citation

If you find JarvisIR useful in your research, please cite our paper:

@inproceedings{lin2025jarvisir,
  title={Jarvisir: Elevating autonomous driving perception with intelligent image restoration},
  author={Lin, Yunlong and Lin, Zixu and Chen, Haoyu and Pan, Panwang and Li, Chenxin and Chen, Sixiang and Wen, Kairun and Jin, Yeying and Li, Wenbo and Ding, Xinghao},
  booktitle={Proceedings of the Computer Vision and Pattern Recognition Conference},
  pages={22369--22380},
  year={2025}
}

Related Resources

Project Page: https://cvpr2025-jarvisir.github.io/
Code Repository: https://github.com/LYL1015/JarvisIR
Paper: https://arxiv.org/pdf/2504.04158

Acknowledgments

This work contributes to the advancement of intelligent image restoration by integrating Vision-Language Models with expert system coordination.

LYL1015
/

JarvisIR