title: Hard Negative Contrastive Learning for Fine-Grained Geometric
    Understanding in Large Multimodal Models
summary: Benefiting from contrastively trained visual encoders on large-scale natural
  scene images, Large Multimodal Models (LMMs) have achieved remarkable
  performance across various visual perception tasks. However, the inherent
  limitations of contrastive learning upon summarized descriptions fundamentally
  restrict the capabilities of models in meticulous reasoning, particularly in
  crucial scenarios of geometric problem-solving. To enhance geometric
  understanding, we propose a novel hard negative contrastive learning framework
  for the vision encoder, which combines image-based contrastive learning using
  generation-based hard negatives created by perturbing diagram generation code,
  and text-based contrastive learning using rule-based negatives derived from
  modified geometric descriptions and retrieval-based negatives selected based on
  caption similarity. We train CLIP using our strong negative learning method,
  namely MMCLIP (Multimodal Math CLIP), and subsequently train an LMM for
  geometric problem-solving. Experiments show that our trained model, MMGeoLM,
  significantly outperforms other open-source models on three geometric reasoning
  benchmarks. Even with a size of 7B, it can rival powerful closed-source models
  like GPT-4o. We further study the impact of different negative sample
  construction methods and the number of negative samples on the geometric
  reasoning performance of LMM, yielding fruitful conclusions. The code and
  dataset are available at https://github.com/THU-KEG/MMGeoLM.
paper_authors._id: 6835385ebd4d4208167d15ac
paper_authors.name: Kai Sun
paper_authors._id: 6835385ebd4d4208167d15ad
paper_authors.name: Yushi Bai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-27T13:06:45.273Z
paper_authors._id: 64ed568ccf6118a9379a61b8
paper_authors.avatarUrl: /avatars/6d040cbcb4a9b624cbe64c9d01cd5c88.svg
paper_authors.fullname: Yushi Bai
paper_authors.type: user
paper_authors.user: bys0318
paper_authors._id: 6835385ebd4d4208167d15ae
paper_authors.name: Zhen Yang
paper_authors._id: 6835385ebd4d4208167d15af
paper_authors.name: Jiajie Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-27T13:06:22.005Z
paper_authors._id: 66cdd285c51a915bd5f2d017
paper_authors.avatarUrl: /avatars/14e5794307e4672b1b51d26b31227e0f.svg
paper_authors.fullname: Jiajie Zhang
paper_authors.type: user
paper_authors.user: NeoZ123
paper_authors._id: 6835385ebd4d4208167d15b0
paper_authors.name: Ji Qi
paper_authors._id: 6835385ebd4d4208167d15b1
paper_authors.name: Lei Hou
paper_authors._id: 6835385ebd4d4208167d15b2
paper_authors.name: Juanzi Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-27T13:06:29.209Z
paper_authors._id: 65df8cbc2705d9672f55d1aa
paper_authors.avatarUrl: /avatars/63e46f15bb76bd9d4508fd0f54f39829.svg
paper_authors.fullname: Juanzi Li
paper_authors.type: user
paper_authors.user: juanli
paper_title: Hard Negative Contrastive Learning for Fine-Grained Geometric
    Understanding in Large Multimodal Models
paper_summary: Benefiting from contrastively trained visual encoders on large-scale natural
  scene images, Large Multimodal Models (LMMs) have achieved remarkable
  performance across various visual perception tasks. However, the inherent
  limitations of contrastive learning upon summarized descriptions fundamentally
  restrict the capabilities of models in meticulous reasoning, particularly in
  crucial scenarios of geometric problem-solving. To enhance geometric
  understanding, we propose a novel hard negative contrastive learning framework
  for the vision encoder, which combines image-based contrastive learning using
  generation-based hard negatives created by perturbing diagram generation code,
  and text-based contrastive learning using rule-based negatives derived from
  modified geometric descriptions and retrieval-based negatives selected based on
  caption similarity. We train CLIP using our strong negative learning method,
  namely MMCLIP (Multimodal Math CLIP), and subsequently train an LMM for
  geometric problem-solving. Experiments show that our trained model, MMGeoLM,
  significantly outperforms other open-source models on three geometric reasoning
  benchmarks. Even with a size of 7B, it can rival powerful closed-source models
  like GPT-4o. We further study the impact of different negative sample
  construction methods and the number of negative samples on the geometric
  reasoning performance of LMM, yielding fruitful conclusions. The code and
  dataset are available at https://github.com/THU-KEG/MMGeoLM.
paper_ai_summary: A novel hard negative contrastive learning framework improves geometric reasoning in Large Multimodal Models, significantly enhancing their performance compared to existing models.
paper_ai_keywords: contrastively trained visual encoders Large Multimodal Models geometric problem-solving hard negative contrastive learning generation-based hard negatives rule-based negatives retrieval-based negatives CLIP MMCLIP multimodal math clip MMGeoLM geometric reasoning benchmarks