Two-Stream Convolutional Networks for Action Recognition in Videos
We investigate architectures of discriminatively trained deep Convolutional Networks (ConvNets) for action recognition in video. The challenge is to capture the complementary information on appearance from still frames and motion between frames. We also aim to generalise the best performing hand-crafted features within a data-driven learning framework. Our contribution is three-fold. First, we propose a two-stream ConvNet architecture which incorporates spatial and temporal networks. Second, we demonstrate that a ConvNet trained on multi-frame dense optical flow is able to achieve very good performance in spite of limited training data. Finally, we show that multitask learning, applied to two different action classification datasets, can be used to increase the amount of training data and improve the performance on both. Our architecture is trained and evaluated on the standard video actions benchmarks of UCF-101 and HMDB-51, where it is competitive with the state of the art. It also exceeds by a large margin previous attempts to use deep nets for video classification.
Summary:This paper proposes a model for solving discriminative tasks with video inputs.  Themodel consists of two convolutional nets. The input to one net is an appearanceframe. The input to the second net is a stack of densely computed optical flowfeatures. Each pathway is trained separately to classify its input.  Theprediction for a video is obtained by taking a (weighted) average of thepredictions made by each net.The model is evaluated on two datasets.  The results are impressive and match(or come close to) the state-of-the-art methods which use intensivelyhand-crafted features.Strengths:- This model is a simple application of convolutional nets that gets good results.- Previous deep learning models have not tried to use optical flow (or otherhand-crafted features) for vision tasks, preferring to learn all features frompixels directly. This paper shows that at least for action recognition, opticalflow fields are a useful input representation. This is an important contribution.Weaknesses:- The model does not take into account the overall sequence ofactions over the entire video but only models fixed length (L=10) adjacent frames.Quality:The experiments are well-designed. It would be more insightful if the authors also include some analysis of the error modes, for example, are their some classes or group of classes that the model is unable to classify well ? Is it possible to characterize the kind of videos on which the model does not work well ?Clarity:The paper is clearly written. The model is well explained.Originality:The application of a convolutional net to optical flow features is novel.Significance:This approach could have a significant impact on the research in using videos and motion for various vision problems.