Evaluating Zero-Shot GPT-4V Performance on 3D Visual Question Answering Benchmarks
<|reference_start|>Evaluating Zero-Shot GPT-4V Performance on 3D Visual Question Answering Benchmarks: As interest in "reformulating" the 3D Visual Question Answering (VQA) problem in the context of foundation models grows, it is imperative to assess how these new paradigms influence existing closed-vocabulary datasets. In this case study, we evaluate the zero-shot performance of foundational models (GPT-4 Vision and GPT-4) on well-established 3D VQA benchmarks, namely 3D-VQA and ScanQA. We provide an investigation to contextualize the performance of GPT-based agents relative to traditional modeling approaches. We find that GPT-based agents without any fine-tuning perform on par with the closed vocabulary approaches. Our findings corroborate recent results that "blind" models establish a surprisingly strong baseline in closed-vocabulary settings. We demonstrate that agents benefit significantly from scene-specific vocabulary via in-context textual grounding. By presenting a preliminary comparison with previous baselines, we hope to inform the community's ongoing efforts to refine multi-modal 3D benchmarks.<|reference_end|>
@article{singh2024evaluating,
    title={Evaluating Zero-Shot GPT-4V Performance on 3D Visual Question Answering
  Benchmarks},
    author={Simranjit Singh, Georgios Pavlakos, Dimitrios Stamoulis},
    journal={arXiv preprint arXiv:2405.18831},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.18831},
    primaryClass={cs.CV cs.LG}
}