Groupwise Query Specialization and Quality-Aware Multi-Assignment for Transformer-based Visual Relationship Detection
<|reference_start|>Groupwise Query Specialization and Quality-Aware Multi-Assignment for Transformer-based Visual Relationship Detection: Visual Relationship Detection (VRD) has seen significant advancements with Transformer-based architectures recently. However, we identify two key limitations in a conventional label assignment for training Transformer-based VRD models, which is a process of mapping a ground-truth (GT) to a prediction. Under the conventional assignment, an unspecialized query is trained since a query is expected to detect every relation, which makes it difficult for a query to specialize in specific relations. Furthermore, a query is also insufficiently trained since a GT is assigned only to a single prediction, therefore near-correct or even correct predictions are suppressed by being assigned no relation as a GT. To address these issues, we propose Groupwise Query Specialization and Quality-Aware Multi-Assignment (SpeaQ). Groupwise Query Specialization trains a specialized query by dividing queries and relations into disjoint groups and directing a query in a specific query group solely toward relations in the corresponding relation group. Quality-Aware Multi-Assignment further facilitates the training by assigning a GT to multiple predictions that are significantly close to a GT in terms of a subject, an object, and the relation in between. Experimental results and analyses show that SpeaQ effectively trains specialized queries, which better utilize the capacity of a model, resulting in consistent performance gains with zero additional inference cost across multiple VRD models and benchmarks. Code is available at https://github.com/mlvlab/SpeaQ.<|reference_end|>
@article{kim2024groupwise,
    title={Groupwise Query Specialization and Quality-Aware Multi-Assignment for
  Transformer-based Visual Relationship Detection},
    author={Jongha Kim, Jihwan Park, Jinyoung Park, Jinyoung Kim, Sehyung Kim,
  Hyunwoo J. Kim},
    journal={arXiv preprint arXiv:2403.17709},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.17709},
    primaryClass={cs.CV}
}