All in One Framework for Multimodal Re-identification in the Wild
<|reference_start|>All in One Framework for Multimodal Re-identification in the Wild: In Re-identification (ReID), recent advancements yield noteworthy progress in both unimodal and cross-modal retrieval tasks. However, the challenge persists in developing a unified framework that could effectively handle varying multimodal data, including RGB, infrared, sketches, and textual information. Additionally, the emergence of large-scale models shows promising performance in various vision tasks but the foundation model in ReID is still blank. In response to these challenges, a novel multimodal learning paradigm for ReID is introduced, referred to as All-in-One (AIO), which harnesses a frozen pre-trained big model as an encoder, enabling effective multimodal retrieval without additional fine-tuning. The diverse multimodal data in AIO are seamlessly tokenized into a unified space, allowing the modality-shared frozen encoder to extract identity-consistent features comprehensively across all modalities. Furthermore, a meticulously crafted ensemble of cross-modality heads is designed to guide the learning trajectory. AIO is the \textbf{first} framework to perform all-in-one ReID, encompassing four commonly used modalities. Experiments on cross-modal and multimodal ReID reveal that AIO not only adeptly handles various modal data but also excels in challenging contexts, showcasing exceptional performance in zero-shot and domain generalization scenarios.<|reference_end|>
@article{li2024all,
    title={All in One Framework for Multimodal Re-identification in the Wild},
    author={He Li, Mang Ye, Ming Zhang, Bo Du},
    journal={arXiv preprint arXiv:2405.04741},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.04741},
    primaryClass={cs.CV}
}