FineCops-Ref: A new Dataset and Task for Fine-Grained Compositional Referring Expression Comprehension
<|reference_start|>FineCops-Ref: A new Dataset and Task for Fine-Grained Compositional Referring Expression Comprehension: Referring Expression Comprehension (REC) is a crucial cross-modal task that objectively evaluates the capabilities of language understanding, image comprehension, and language-to-image grounding. Consequently, it serves as an ideal testing ground for Multi-modal Large Language Models (MLLMs). In pursuit of this goal, we have established a new REC dataset characterized by two key features: Firstly, it is designed with controllable varying levels of difficulty, necessitating multi-level fine-grained reasoning across object categories, attributes, and multi-hop relationships. Secondly, it includes negative text and images created through fine-grained editing and generation based on existing data, thereby testing the model's ability to correctly reject scenarios where the target object is not visible in the image--an essential aspect often overlooked in existing datasets and approaches. Utilizing this high-quality dataset, we conducted comprehensive evaluations of both state-of-the-art specialist models and MLLMs. Our findings indicate that there remains a significant gap in achieving satisfactory grounding performance. We anticipate that our dataset will inspire new approaches to enhance visual reasoning and develop more advanced cross-modal interaction strategies, ultimately unlocking the full potential of MLLMs. Our code and the datasets are available at https://github.com/liujunzhuo/FineCops-Ref.<|reference_end|>
@article{liu2024finecops-ref:,
    title={FineCops-Ref: A new Dataset and Task for Fine-Grained Compositional
  Referring Expression Comprehension},
    author={Junzhuo Liu, Xuzheng Yang, Weiwei Li, Peng Wang},
    journal={arXiv preprint arXiv:2409.14750},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.14750},
    primaryClass={cs.CV cs.CL}
}