Multi-modal Crowd Counting via a Broker Modality
<|reference_start|>Multi-modal Crowd Counting via a Broker Modality: Multi-modal crowd counting involves estimating crowd density from both visual and thermal/depth images. This task is challenging due to the significant gap between these distinct modalities. In this paper, we propose a novel approach by introducing an auxiliary broker modality and on this basis frame the task as a triple-modal learning problem. We devise a fusion-based method to generate this broker modality, leveraging a non-diffusion, lightweight counterpart of modern denoising diffusion-based fusion models. Additionally, we identify and address the ghosting effect caused by direct cross-modal image fusion in multi-modal crowd counting. Through extensive experimental evaluations on popular multi-modal crowd-counting datasets, we demonstrate the effectiveness of our method, which introduces only 4 million additional parameters, yet achieves promising results. The code is available at https://github.com/HenryCilence/Broker-Modality-Crowd-Counting.<|reference_end|>
@article{meng2024multi-modal,
    title={Multi-modal Crowd Counting via a Broker Modality},
    author={Haoliang Meng and Xiaopeng Hong and Chenhao Wang and Miao Shang and
  Wangmeng Zuo},
    journal={arXiv preprint arXiv:2407.07518},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.07518},
    primaryClass={cs.CV}
}