Large Language Models Can Understanding Depth from Monocular Images
<|reference_start|>Large Language Models Can Understanding Depth from Monocular Images: Monocular depth estimation is a critical function in computer vision applications. This paper shows that large language models (LLMs) can effectively interpret depth with minimal supervision, using efficient resource utilization and a consistent neural network architecture. We introduce LLM-MDE, a multimodal framework that deciphers depth through language comprehension. Specifically, LLM-MDE employs two main strategies to enhance the pretrained LLM's capability for depth estimation: cross-modal reprogramming and an adaptive prompt estimation module. These strategies align vision representations with text prototypes and automatically generate prompts based on monocular images, respectively. Comprehensive experiments on real-world MDE datasets confirm the effectiveness and superiority of LLM-MDE, which excels in few-/zero-shot tasks while minimizing resource use. The source code is available.<|reference_end|>
@article{xia2024large,
    title={Large Language Models Can Understanding Depth from Monocular Images},
    author={Zhongyi Xia and Tianzhao Wu},
    journal={arXiv preprint arXiv:2409.01133},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.01133},
    primaryClass={cs.CV cs.AI}
}