MINIMA: Modality Invariant Image Matching
Image matching for both cross-view and cross-modality plays a critical role
in multimodal perception. In practice, the modality gap caused by different
imaging systems/styles poses great challenges to the matching task. Existing
works try to extract invariant features for specific modalities and train on
limited datasets, showing poor generalization. In this paper, we present
MINIMA, a unified image matching framework for multiple cross-modal cases.
Without pursuing fancy modules, our MINIMA aims to enhance universal
performance from the perspective of data scaling up. For such purpose, we
propose a simple yet effective data engine that can freely produce a large
dataset containing multiple modalities, rich scenarios, and accurate matching
labels. Specifically, we scale up the modalities from cheap but rich RGB-only
matching data, by means of generative models. Under this setting, the matching
labels and rich diversity of the RGB dataset are well inherited by the
generated multimodal data. Benefiting from this, we construct MD-syn, a new
comprehensive dataset that fills the data gap for general multimodal image
matching. With MD-syn, we can directly train any advanced matching pipeline on
randomly selected modality pairs to obtain cross-modal ability. Extensive
experiments on in-domain and zero-shot matching tasks, including 19
cross-modal cases, demonstrate that our MINIMA can significantly outperform the
baselines and even surpass modality-specific methods. The dataset and code are
available at https://github.com/LSXI7/MINIMA .
67893aed91260c834a4d7cd9
Xingyu Jiang
67893aed91260c834a4d7cda
Jiangwei Ren
claimed_verified
2025-01-17T17:06:00.673Z
668e05b7410a13fa3dabb2c3
/avatars/c197ce4957404b2e3223d0ac3031581c.svg
Jiangwei Ren
user
lsxi77777
67893aed91260c834a4d7cdb
Zizhuo Li
67893aed91260c834a4d7cdc
Xin Zhou
claimed_verified
2025-03-14T08:59:31.315Z
6717c5c36bc2876059ed23ab
/avatars/52c68fb315760df5ef9323cd8ada5a3c.svg
Xin Zhou
user
LMD0311
67893aed91260c834a4d7cdd
Dingkang Liang
admin_assigned
2025-01-16T22:13:25.294Z
67467b5979406f42a14517e9
https://cdn-avatars.huggingface.co/v1/production/uploads/67467b5979406f42a14517e9/wgnUxTd8vWOo0Cr2gaoyG.jpeg
Dingkang Liang
user
dkliang
67893aed91260c834a4d7cde
Xiang Bai
MINIMA: Modality Invariant Image Matching
Image matching for both cross-view and cross-modality plays a critical role
in multimodal perception. In practice, the modality gap caused by different
imaging systems/styles poses great challenges to the matching task. Existing
works try to extract invariant features for specific modalities and train on
limited datasets, showing poor generalization. In this paper, we present
MINIMA, a unified image matching framework for multiple cross-modal cases.
Without pursuing fancy modules, our MINIMA aims to enhance universal
performance from the perspective of data scaling up. For such purpose, we
propose a simple yet effective data engine that can freely produce a large
dataset containing multiple modalities, rich scenarios, and accurate matching
labels. Specifically, we scale up the modalities from cheap but rich RGB-only
matching data, by means of generative models. Under this setting, the matching
labels and rich diversity of the RGB dataset are well inherited by the
generated multimodal data. Benefiting from this, we construct MD-syn, a new
comprehensive dataset that fills the data gap for general multimodal image
matching. With MD-syn, we can directly train any advanced matching pipeline on
randomly selected modality pairs to obtain cross-modal ability. Extensive
experiments on in-domain and zero-shot matching tasks, including 19
cross-modal cases, demonstrate that our MINIMA can significantly outperform the
baselines and even surpass modality-specific methods. The dataset and code are
available at https://github.com/LSXI7/MINIMA .
A unified framework MINIMA uses generative models to create a large, multimodal dataset (MD-syn) for enhancing cross-view and cross-modality image matching performance.
modality gap invariant features generative models data engine multimodal dataset cross-modal image matching in-domain matching zero-shot matching