BCTR: Bidirectional Conditioning Transformer for Scene Graph Generation
<|reference_start|>BCTR: Bidirectional Conditioning Transformer for Scene Graph Generation: Scene Graph Generation (SGG) remains a challenging task due to its compositional property. Previous approaches improve prediction efficiency by learning in an end-to-end manner. However, these methods exhibit limited performance as they assume unidirectional conditioning between entities and predicates, leading to insufficient information interaction. To address this limitation, we propose a novel bidirectional conditioning factorization for SGG, introducing efficient interaction between entities and predicates. Specifically, we develop an end-to-end scene graph generation model, Bidirectional Conditioning Transformer (BCTR), to implement our factorization. BCTR consists of two key modules. First, the Bidirectional Conditioning Generator (BCG) facilitates multi-stage interactive feature augmentation between entities and predicates, enabling mutual benefits between the two predictions. Second, Random Feature Alignment (RFA) regularizes the feature space by distilling multi-modal knowledge from pre-trained models, enhancing BCTR's ability on tailed categories without relying on statistical priors. We conduct a series of experiments on Visual Genome and Open Image V6, demonstrating that BCTR achieves state-of-the-art performance on both benchmarks. The code will be available upon acceptance of the paper.<|reference_end|>
@article{hao2024bctr:,
    title={BCTR: Bidirectional Conditioning Transformer for Scene Graph Generation},
    author={Peng Hao, Xiaobing Wang, Yingying Jiang, Hanchao Jia, Xiaoshuai Hao},
    journal={arXiv preprint arXiv:2407.18715},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.18715},
    primaryClass={cs.CV}
}