  Visual relationship detection aims to detect the interactions between objects
in an image; however, this task suffers from combinatorial explosion due to the
variety of objects and interactions. Since the interactions associated with the
same object are dependent, we explore the dependency of interactions to reduce
the search space. We explicitly model objects and interactions by an
interaction graph and then propose a message-passing-style algorithm to
propagate the contextual information. We thus call the proposed method neural
message passing (NMP). We further integrate language priors and spatial cues to
rule out unrealistic interactions and capture spatial interactions.
Experimental results on two benchmark datasets demonstrate the superiority of
our proposed method. Our code is available at https://github.com/PhyllisH/NMP.

Computer Vision and Pattern Recognition