HiKER-SGG: Hierarchical Knowledge Enhanced Robust Scene Graph Generation
<|reference_start|>HiKER-SGG: Hierarchical Knowledge Enhanced Robust Scene Graph Generation: Being able to understand visual scenes is a precursor for many downstream tasks, including autonomous driving, robotics, and other vision-based approaches. A common approach enabling the ability to reason over visual data is Scene Graph Generation (SGG); however, many existing approaches assume undisturbed vision, i.e., the absence of real-world corruptions such as fog, snow, smoke, as well as non-uniform perturbations like sun glare or water drops. In this work, we propose a novel SGG benchmark containing procedurally generated weather corruptions and other transformations over the Visual Genome dataset. Further, we introduce a corresponding approach, Hierarchical Knowledge Enhanced Robust Scene Graph Generation (HiKER-SGG), providing a strong baseline for scene graph generation under such challenging setting. At its core, HiKER-SGG utilizes a hierarchical knowledge graph in order to refine its predictions from coarse initial estimates to detailed predictions. In our extensive experiments, we show that HiKER-SGG does not only demonstrate superior performance on corrupted images in a zero-shot manner, but also outperforms current state-of-the-art methods on uncorrupted SGG tasks. Code is available at https://github.com/zhangce01/HiKER-SGG.<|reference_end|>
@article{zhang2024hiker-sgg:,
    title={HiKER-SGG: Hierarchical Knowledge Enhanced Robust Scene Graph Generation},
    author={Ce Zhang, Simon Stepputtis, Joseph Campbell, Katia Sycara, Yaqi Xie},
    journal={arXiv preprint arXiv:2403.12033},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.12033},
    primaryClass={cs.CV}
}