Euclid: Supercharging Multimodal LLMs with Synthetic High-Fidelity
  Visual Descriptions
Multimodal large language models (MLLMs) have made rapid progress in recent
years, yet continue to struggle with low-level visual perception (LLVP) --
particularly the ability to accurately describe the geometric details of an
image. This capability is crucial for applications in areas such as robotics,
medical image analysis, and manufacturing. In this paper, we first introduce
Geoperception, a benchmark designed to evaluate an MLLM's ability to accurately
transcribe 2D geometric information from an image. Using this benchmark, we
demonstrate the limitations of leading MLLMs, and then conduct a comprehensive
empirical study to explore strategies for improving their performance on
geometric tasks. Our findings highlight the benefits of certain model
architectures, training techniques, and data strategies, including the use of
high-fidelity synthetic data and multi-stage training with a data curriculum.
Notably, we find that a data curriculum enables models to learn challenging
geometry understanding tasks which they fail to learn from scratch. Leveraging
these insights, we develop Euclid, a family of models specifically optimized
for strong low-level geometric perception. Although purely trained on synthetic
multimodal data, Euclid shows strong generalization ability to novel geometry
shapes. For instance, Euclid outperforms the best closed-source model,
Gemini-1.5-Pro, by up to 58.56% on certain Geoperception benchmark tasks and
10.65% on average across all tasks.
675bac3048fca25a2189e618
Jiarui Zhang
extracted_confirmed
2024-12-13T03:42:40.590Z
635b99d47a1656011516bff9
/avatars/7243c4171ff127ba90631f105881d9d7.svg
jiarui zhang
user
jrzhang
675bac3048fca25a2189e619
Ollie Liu
claimed_verified
2024-12-13T15:22:16.732Z
66197a8afeb55cbe39e50ae8
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/pcSS1TsCoHxRcAEkcMNm0.png
Ollie Liu
user
oliu-io
675bac3048fca25a2189e61a
Tianyu Yu
claimed_verified
2024-12-16T09:42:16.890Z
64abc4aa6cadc7aca585dddf
/avatars/736afea979cd0021c7a37f68731524ea.svg
Tianyu Yu
user
Yirany
675bac3048fca25a2189e61b
Jinyi Hu
claimed_verified
2024-12-16T09:42:18.725Z
637a06580a77f602dc4ac922
https://cdn-avatars.huggingface.co/v1/production/uploads/637a06580a77f602dc4ac922/JR2VHWhG1tj3TKKNzD8GD.jpeg
Jinyi Hu
user
JamesHujy
675bac3048fca25a2189e61c
Willie Neiswanger
claimed_verified
2025-01-07T08:42:56.038Z
644bf65522d211df6444a7f4
https://cdn-avatars.huggingface.co/v1/production/uploads/644bf65522d211df6444a7f4/k_ddZdQDg2fzhwjI1EXyx.jpeg
Willie Neiswanger
user
willieneis
Euclid: Supercharging Multimodal LLMs with Synthetic High-Fidelity
  Visual Descriptions
Multimodal large language models (MLLMs) have made rapid progress in recent
years, yet continue to struggle with low-level visual perception (LLVP) --
particularly the ability to accurately describe the geometric details of an
image. This capability is crucial for applications in areas such as robotics,
medical image analysis, and manufacturing. In this paper, we first introduce
Geoperception, a benchmark designed to evaluate an MLLM's ability to accurately
transcribe 2D geometric information from an image. Using this benchmark, we
demonstrate the limitations of leading MLLMs, and then conduct a comprehensive
empirical study to explore strategies for improving their performance on
geometric tasks. Our findings highlight the benefits of certain model
architectures, training techniques, and data strategies, including the use of
high-fidelity synthetic data and multi-stage training with a data curriculum.
Notably, we find that a data curriculum enables models to learn challenging
geometry understanding tasks which they fail to learn from scratch. Leveraging
these insights, we develop Euclid, a family of models specifically optimized
for strong low-level geometric perception. Although purely trained on synthetic
multimodal data, Euclid shows strong generalization ability to novel geometry
shapes. For instance, Euclid outperforms the best closed-source model,
Gemini-1.5-Pro, by up to 58.56% on certain Geoperception benchmark tasks and
10.65% on average across all tasks.
A benchmark called Geoperception is introduced to evaluate MLLMs' geometric description accuracy, leading to the development of Euclid, a model optimized for low-level geometric perception using synthetic data and a data curriculum.
multimodal large language models low-level visual perception Geoperception benchmark model architectures training techniques data strategies high-fidelity synthetic data multi-stage training data curriculum geometry understanding tasks Euclid