A method for using a multi-hop reasoning framework to perform multi-step compositional long-term reasoning is presented. The method includes extracting (1001) feature maps and frame-level representations from a video stream by using a convolutional neural network (CNN), performing (1003) object representation learning and detection, linking (1005) objects through time via tracking to generate object tracks and image feature tracks, feeding (1007) the object tracks and the image feature tracks to a multi-hop transformer that hops over frames in the video stream while concurrently attending to one or more of the objects in the video stream until the multi-hop transformer arrives at a correct answer, and employing (1009) video representation learning and recognition from the objects and image context to locate a target object within the video stream.
MULTI-HOP TRANSFORMER FOR SPATIO-TEMPORAL REASONING AND LOCALIZATION