  We propose a new model for detecting visual relationships, such as "person
riding motorcycle" or "bottle on table". This task is an important step towards
comprehensive structured image understanding, going beyond detecting individual
objects. Our main novelty is a Box Attention mechanism that allows to model
pairwise interactions between objects using standard object detection
pipelines. The resulting model is conceptually clean, expressive and relies on
well-justified training and prediction procedures. Moreover, unlike previously
proposed approaches, our model does not introduce any additional complex
components or hyperparameters on top of those already required by the
underlying detection model. We conduct an experimental evaluation on three
challenging datasets, V-COCO, Visual Relationships and Open Images,
demonstrating strong quantitative and qualitative results.

Computer Vision and Pattern Recognition