title: Euclid: Supercharging Multimodal LLMs with Synthetic High-Fidelity
    Visual Descriptions
summary: Multimodal large language models (MLLMs) have made rapid progress in recent
  years, yet continue to struggle with low-level visual perception (LLVP) --
  particularly the ability to accurately describe the geometric details of an
  image. This capability is crucial for applications in areas such as robotics,
  medical image analysis, and manufacturing. In this paper, we first introduce
  Geoperception, a benchmark designed to evaluate an MLLM's ability to accurately
  transcribe 2D geometric information from an image. Using this benchmark, we
  demonstrate the limitations of leading MLLMs, and then conduct a comprehensive
  empirical study to explore strategies for improving their performance on
  geometric tasks. Our findings highlight the benefits of certain model
  architectures, training techniques, and data strategies, including the use of
  high-fidelity synthetic data and multi-stage training with a data curriculum.
  Notably, we find that a data curriculum enables models to learn challenging
  geometry understanding tasks which they fail to learn from scratch. Leveraging
  these insights, we develop Euclid, a family of models specifically optimized
  for strong low-level geometric perception. Although purely trained on synthetic
  multimodal data, Euclid shows strong generalization ability to novel geometry
  shapes. For instance, Euclid outperforms the best closed-source model,
  Gemini-1.5-Pro, by up to 58.56% on certain Geoperception benchmark tasks and
  10.65% on average across all tasks.
paper_authors._id: 675bac3048fca25a2189e618
paper_authors.name: Jiarui Zhang
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-12-13T03:42:40.590Z
paper_authors._id: 635b99d47a1656011516bff9
paper_authors.avatarUrl: /avatars/7243c4171ff127ba90631f105881d9d7.svg
paper_authors.fullname: jiarui zhang
paper_authors.type: user
paper_authors.user: jrzhang
paper_authors._id: 675bac3048fca25a2189e619
paper_authors.name: Ollie Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-13T15:22:16.732Z
paper_authors._id: 66197a8afeb55cbe39e50ae8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/pcSS1TsCoHxRcAEkcMNm0.png
paper_authors.fullname: Ollie Liu
paper_authors.type: user
paper_authors.user: oliu-io
paper_authors._id: 675bac3048fca25a2189e61a
paper_authors.name: Tianyu Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-16T09:42:16.890Z
paper_authors._id: 64abc4aa6cadc7aca585dddf
paper_authors.avatarUrl: /avatars/736afea979cd0021c7a37f68731524ea.svg
paper_authors.fullname: Tianyu Yu
paper_authors.type: user
paper_authors.user: Yirany
paper_authors._id: 675bac3048fca25a2189e61b
paper_authors.name: Jinyi Hu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-16T09:42:18.725Z
paper_authors._id: 637a06580a77f602dc4ac922
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/637a06580a77f602dc4ac922/JR2VHWhG1tj3TKKNzD8GD.jpeg
paper_authors.fullname: Jinyi Hu
paper_authors.type: user
paper_authors.user: JamesHujy
paper_authors._id: 675bac3048fca25a2189e61c
paper_authors.name: Willie Neiswanger
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-07T08:42:56.038Z
paper_authors._id: 644bf65522d211df6444a7f4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/644bf65522d211df6444a7f4/k_ddZdQDg2fzhwjI1EXyx.jpeg
paper_authors.fullname: Willie Neiswanger
paper_authors.type: user
paper_authors.user: willieneis
paper_title: Euclid: Supercharging Multimodal LLMs with Synthetic High-Fidelity
    Visual Descriptions
paper_summary: Multimodal large language models (MLLMs) have made rapid progress in recent
  years, yet continue to struggle with low-level visual perception (LLVP) --
  particularly the ability to accurately describe the geometric details of an
  image. This capability is crucial for applications in areas such as robotics,
  medical image analysis, and manufacturing. In this paper, we first introduce
  Geoperception, a benchmark designed to evaluate an MLLM's ability to accurately
  transcribe 2D geometric information from an image. Using this benchmark, we
  demonstrate the limitations of leading MLLMs, and then conduct a comprehensive
  empirical study to explore strategies for improving their performance on
  geometric tasks. Our findings highlight the benefits of certain model
  architectures, training techniques, and data strategies, including the use of
  high-fidelity synthetic data and multi-stage training with a data curriculum.
  Notably, we find that a data curriculum enables models to learn challenging
  geometry understanding tasks which they fail to learn from scratch. Leveraging
  these insights, we develop Euclid, a family of models specifically optimized
  for strong low-level geometric perception. Although purely trained on synthetic
  multimodal data, Euclid shows strong generalization ability to novel geometry
  shapes. For instance, Euclid outperforms the best closed-source model,
  Gemini-1.5-Pro, by up to 58.56% on certain Geoperception benchmark tasks and
  10.65% on average across all tasks.
paper_ai_keywords: Geoperception multimodal large language models (MLLMs) low-level visual perception (LLVP) geometric details robotics medical image analysis manufacturing 2D geometric information empirical study high-fidelity synthetic data data curriculum Euclid strong low-level geometric perception generalization ability Gemin-1.5-Pro