Hard Negative Contrastive Learning for Fine-Grained Geometric
  Understanding in Large Multimodal Models
Benefiting from contrastively trained visual encoders on large-scale natural
scene images, Large Multimodal Models (LMMs) have achieved remarkable
performance across various visual perception tasks. However, the inherent
limitations of contrastive learning upon summarized descriptions fundamentally
restrict the capabilities of models in meticulous reasoning, particularly in
crucial scenarios of geometric problem-solving. To enhance geometric
understanding, we propose a novel hard negative contrastive learning framework
for the vision encoder, which combines image-based contrastive learning using
generation-based hard negatives created by perturbing diagram generation code,
and text-based contrastive learning using rule-based negatives derived from
modified geometric descriptions and retrieval-based negatives selected based on
caption similarity. We train CLIP using our strong negative learning method,
namely MMCLIP (Multimodal Math CLIP), and subsequently train an LMM for
geometric problem-solving. Experiments show that our trained model, MMGeoLM,
significantly outperforms other open-source models on three geometric reasoning
benchmarks. Even with a size of 7B, it can rival powerful closed-source models
like GPT-4o. We further study the impact of different negative sample
construction methods and the number of negative samples on the geometric
reasoning performance of LMM, yielding fruitful conclusions. The code and
dataset are available at https://github.com/THU-KEG/MMGeoLM.
6835385ebd4d4208167d15ac
Kai Sun
6835385ebd4d4208167d15ad
Yushi Bai
admin_assigned
2025-05-27T13:06:45.273Z
64ed568ccf6118a9379a61b8
/avatars/6d040cbcb4a9b624cbe64c9d01cd5c88.svg
Yushi Bai
user
bys0318
6835385ebd4d4208167d15ae
Zhen Yang
6835385ebd4d4208167d15af
Jiajie Zhang
admin_assigned
2025-05-27T13:06:22.005Z
66cdd285c51a915bd5f2d017
/avatars/14e5794307e4672b1b51d26b31227e0f.svg
Jiajie Zhang
user
NeoZ123
6835385ebd4d4208167d15b0
Ji Qi
6835385ebd4d4208167d15b1
Lei Hou
6835385ebd4d4208167d15b2
Juanzi Li
admin_assigned
2025-05-27T13:06:29.209Z
65df8cbc2705d9672f55d1aa
/avatars/63e46f15bb76bd9d4508fd0f54f39829.svg
Juanzi Li
user
juanli
Hard Negative Contrastive Learning for Fine-Grained Geometric
  Understanding in Large Multimodal Models
Benefiting from contrastively trained visual encoders on large-scale natural
scene images, Large Multimodal Models (LMMs) have achieved remarkable
performance across various visual perception tasks. However, the inherent
limitations of contrastive learning upon summarized descriptions fundamentally
restrict the capabilities of models in meticulous reasoning, particularly in
crucial scenarios of geometric problem-solving. To enhance geometric
understanding, we propose a novel hard negative contrastive learning framework
for the vision encoder, which combines image-based contrastive learning using
generation-based hard negatives created by perturbing diagram generation code,
and text-based contrastive learning using rule-based negatives derived from
modified geometric descriptions and retrieval-based negatives selected based on
caption similarity. We train CLIP using our strong negative learning method,
namely MMCLIP (Multimodal Math CLIP), and subsequently train an LMM for
geometric problem-solving. Experiments show that our trained model, MMGeoLM,
significantly outperforms other open-source models on three geometric reasoning
benchmarks. Even with a size of 7B, it can rival powerful closed-source models
like GPT-4o. We further study the impact of different negative sample
construction methods and the number of negative samples on the geometric
reasoning performance of LMM, yielding fruitful conclusions. The code and
dataset are available at https://github.com/THU-KEG/MMGeoLM.
A novel hard negative contrastive learning framework improves geometric reasoning in Large Multimodal Models, significantly enhancing their performance compared to existing models.
contrastively trained visual encoders Large Multimodal Models geometric problem-solving hard negative contrastive learning generation-based hard negatives rule-based negatives retrieval-based negatives CLIP MMCLIP multimodal math clip MMGeoLM geometric reasoning benchmarks