Spatiotemporal Residual Networks for Video Action Recognition
Two-stream Convolutional Networks (ConvNets) have shown strong performance for human action recognition in videos. Recently, Residual Networks (ResNets) have arisen as a new technique to train extremely deep architectures. In this paper, we introduce spatiotemporal ResNets as a combination of these two approaches. Our novel architecture generalizes ResNets for the spatiotemporal domain by introducing residual connections in two ways. First, we inject residual connections between the appearance and motion pathways of a two-stream architecture to allow spatiotemporal interaction between the two streams. Second, we transform pretrained image ConvNets into spatiotemporal networks by equipping them with learnable convolutional filters that are initialized as temporal residual connections and operate on adjacent feature maps in time. This approach slowly increases the spatiotemporal receptive field as the depth of the model increases and naturally integrates image ConvNet design principles. The whole model is trained end-to-end to allow hierarchical learning of complex spatiotemporal features. We evaluate our novel spatiotemporal ResNet using two widely used action recognition benchmarks where it exceeds the previous state-of-the-art.
This paper proposed a deep CNN architecture for video action recognition by extending residual networks. The proposed architecture combines both spatial stream and temporal stream into a single network, and extends to model longer period of temporal information. This work achieves state-of-the-art performance on two public action recognition datasets. 
This paper presents a framework that improves two stream networks for video action recognition by extending residual network to combine information from two streams into one single network. It significantly improves over previous state-of-the-art on two popular video action recognition benchmark. The downside of this paper is the limited novelty. There are previous work tried to combine two streams into a single network [1,2], and the temporal convolution is not new either [3]. Although the way to combine two streams is slightly different from previous work, the proposed approach is still pretty straightforward.  Since this paper mainly consists of incremental ideas on previous work, more details could be given on how those ideas help in solving the tasks to gain more insights. For example, separate ablation studies would be useful to understand how the techniques in section 3.1 and 3.2 help.  The temporal convolution described in section 3.2 is just replacing spatial convolution with temporal convolution, which is not really related to residual connections. The term “residual” could be dropped in the title of section 3.2 and line 166.  Some details in the paper are unclear: - Are the HMDB51 networks initialized from UCF101 networks as in previous work? - The proposed network sees 705 frames (line 193). How do you handle videos with less than 705 frames? - Line 233: “Batch-normalization uses a smaller batch size of 128/32.” Does that mean the batch size is 128/32 = 4 as discussed above, or the batch size is 128 or 32? This is a bit confusing. - Is the loss of ST-ResNet defined as the sum of loss of both streams?  Typo: Line 255: UCF0101 -> UCF101  Overall, this paper is clearly written, and produces very good results on video action recognition datasets.    [1] C. Feichtenhofer, A. Pinz, A. Zisserman. Convolutional Two-Stream Network Fusion for Video Action Recognition. https://arxiv.org/abs/1604.06573  [2] E. Park, X. Han, T. L. Berg, A. C. Berg. Combining Multiple Sources of Knowledge in Deep CNNs for Action Recognition. WACV 2016 [3] L. Pigou, A. van den Oord, S. Dieleman, M. Van Herreweghe, J. Dambre, Beyond Temporal Pooling: Recurrence and Temporal Convolutions for Gesture Recognition in Video. http://arxiv.org/abs/1506.01911