TaskCLIP: Extend Large Vision-Language Model for Task Oriented Object Detection
<|reference_start|>TaskCLIP: Extend Large Vision-Language Model for Task Oriented Object Detection: Task-oriented object detection aims to find objects suitable for accomplishing specific tasks. As a challenging task, it requires simultaneous visual data processing and reasoning under ambiguous semantics. Recent solutions are mainly all-in-one models. However, the object detection backbones are pre-trained without text supervision. Thus, to incorporate task requirements, their intricate models undergo extensive learning on a highly imbalanced and scarce dataset, resulting in capped performance, laborious training, and poor generalizability. In contrast, we propose TaskCLIP, a more natural two-stage design composed of general object detection and task-guided object selection. Particularly for the latter, we resort to the recently successful large Vision-Language Models (VLMs) as our backbone, which provides rich semantic knowledge and a uniform embedding space for images and texts. Nevertheless, the naive application of VLMs leads to sub-optimal quality, due to the misalignment between embeddings of object images and their visual attributes, which are mainly adjective phrases. To this end, we design a transformer-based aligner after the pre-trained VLMs to re-calibrate both embeddings. Finally, we employ a trainable score function to post-process the VLM matching results for object selection. Experimental results demonstrate that our TaskCLIP outperforms the state-of-the-art DETR-based model TOIST by 3.5% and only requires a single NVIDIA RTX 4090 for both training and inference.<|reference_end|>
@article{chen2024taskclip:,
    title={TaskCLIP: Extend Large Vision-Language Model for Task Oriented Object
  Detection},
    author={Hanning Chen, Wenjun Huang, Yang Ni, Sanggeon Yun, Yezi Liu, Fei Wen,
  Alvaro Velasquez, Hugo Latapie, Mohsen Imani},
    journal={arXiv preprint arXiv:2403.08108},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.08108},
    primaryClass={cs.CV}
}