Deep Optics for Video Snapshot Compressive Imaging
<|reference_start|>Deep Optics for Video Snapshot Compressive Imaging: Video snapshot compressive imaging (SCI) aims to capture a sequence of video frames with only a single shot of a 2D detector, whose backbones rest in optical modulation patterns (also known as masks) and a computational reconstruction algorithm. Advanced deep learning algorithms and mature hardware are putting video SCI into practical applications. Yet, there are two clouds in the sunshine of SCI: i) low dynamic range as a victim of high temporal multiplexing, and ii) existing deep learning algorithms' degradation on real system. To address these challenges, this paper presents a deep optics framework to jointly optimize masks and a reconstruction network. Specifically, we first propose a new type of structural mask to realize motion-aware and full-dynamic-range measurement. Considering the motion awareness property in measurement domain, we develop an efficient network for video SCI reconstruction using Transformer to capture long-term temporal dependencies, dubbed Res2former. Moreover, sensor response is introduced into the forward model of video SCI to guarantee end-to-end model training close to real system. Finally, we implement the learned structural masks on a digital micro-mirror device. Experimental results on synthetic and real data validate the effectiveness of the proposed framework. We believe this is a milestone for real-world video SCI. The source code and data are available at https://github.com/pwangcs/DeepOpticsSCI.<|reference_end|>
@article{wang2024deep,
    title={Deep Optics for Video Snapshot Compressive Imaging},
    author={Ping Wang, Lishun Wang, and Xin Yuan},
    journal={arXiv preprint arXiv:2404.05274},
    year={2024},
    doi={10.1109/ICCV51070.2023.00977},
    archivePrefix={arXiv},
    eprint={2404.05274},
    primaryClass={cs.CV}
}