abstract—video captioning in essential is a complex natural
process, which is affected by various uncertainties stemming
from video content, subjective judgment, etc. in this paper we
build on the recent progress in using encoder-decoder framework
for video captioning and address what we find to be a critical
deficiency of the existing methods, that most of the decoders
propagate deterministic hidden states. such complex uncertainty
cannot be modeled efficiently by the deterministic models. in
this paper, we propose a generative approach, referred to as
multi-modal stochastic rnns networks (ms-rnn), which models
the uncertainty observed in the data using latent stochastic
variables. therefore, ms-rnn can improve the performance of
video captioning, and generate multiple sentences to describe a
video considering different random factors. specifically, a multimodal lstm (m-lstm) is first proposed to interact with both
visual and textual features to capture a high-level representation.
then, a backward stochastic lstm (s-lstm) is proposed to
support uncertainty propagation by introducing latent variables.
experimental results on the challenging datasets msvd and
msr-vtt show that our proposed ms-rnn approach outperforms the state-of-the-art video captioning benchmarks.
index terms—video captioning, rnn, uncertainty.