DSGG: Dense Relation Transformer for an End-to-end Scene Graph Generation
<|reference_start|>DSGG: Dense Relation Transformer for an End-to-end Scene Graph Generation: Scene graph generation aims to capture detailed spatial and semantic relationships between objects in an image, which is challenging due to incomplete labelling, long-tailed relationship categories, and relational semantic overlap. Existing Transformer-based methods either employ distinct queries for objects and predicates or utilize holistic queries for relation triplets and hence often suffer from limited capacity in learning low-frequency relationships. In this paper, we present a new Transformer-based method, called DSGG, that views scene graph detection as a direct graph prediction problem based on a unique set of graph-aware queries. In particular, each graph-aware query encodes a compact representation of both the node and all of its relations in the graph, acquired through the utilization of a relaxed sub-graph matching during the training process. Moreover, to address the problem of relational semantic overlap, we utilize a strategy for relation distillation, aiming to efficiently learn multiple instances of semantic relationships. Extensive experiments on the VG and the PSG datasets show that our model achieves state-of-the-art results, showing a significant improvement of 3.5\% and 6.7\% in mR@50 and mR@100 for the scene-graph generation task and achieves an even more substantial improvement of 8.5\% and 10.3\% in mR@50 and mR@100 for the panoptic scene graph generation task. Code is available at \url{https://github.com/zeeshanhayder/DSGG}.<|reference_end|>
@article{hayder2024dsgg:,
    title={DSGG: Dense Relation Transformer for an End-to-end Scene Graph
  Generation},
    author={Zeeshan Hayder and Xuming He},
    journal={arXiv preprint arXiv:2403.14886},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.14886},
    primaryClass={cs.CV}
}