A method of automatic image captioning which involves mixing results of an image encoder and a language decoder to emit a sequence of caption words for an input image, with the mixing governed by a gate probability mass determined from a visual sentinel vector of the language decoder and a current hidden state vector of the language decoder. The method also involves determining the results of the image encoder by processing the image through the image encoder to produce image feature vectors for regions of the image and computing a global image feature vector from the image feature vectors. The method further involves determining the results of the language decoder by processing words through the language decoder, including beginning at an initial timestep with a start-of-caption token and the global image feature vector, continuing in successive timesteps using a most recently emitted caption word and the global image feature vector as input to the language decoder, and at each timestep, generating a visual sentinel vector that combines the most recently emitted caption word, the global image feature vector, a previous hidden state vector of the language decoder, and memory contents of the language decoder. The method also involves, at each timestep, using at least a current hidden state vector of the language decoder to determine unnormalized attention values for the image feature vectors and an unnormalized gate value for the visual sentinel vector, and concatenating the unnormalized attention values and the unnonnalized gate value and exponentially normalizing the concatenated attention and gate values to produce a vector of attention probability masses and the gate probability mass. The method also involves applying the attention probability masses to the image feature vectors to accumulate in an image context vector a weighted sum of the image feature vectors and determining an adaptive context vector as a mix of the image context vector and the visual sentinel vector according to the gate probability mass. The method also involves submitting the adaptive context vector and the current hidden state of the language decoder to a feed-forward neural network and causing the feed-forward neural network to emit a next caption word, and repeating the processing of words through the language decoder, the using, the concatenating, the applying, the determining, and the submitting until the next caption word emitted is an end-of-caption token.
SPATIAL ATTENTION MODEL FOR IMAGE CAPTIONING