Deep Alternative Neural Network: Exploring Contexts as Early as Possible for Action Recognition
Contexts are crucial for action recognition in video. Current methods often mine contexts after extracting hierarchical local features and focus on their high-order encodings. This paper instead explores contexts as early as possible and leverages their evolutions for action recognition. In particular, we introduce a novel architecture called deep alternative neural network (DANN) stacking alternative layers. Each alternative layer consists of a volumetric convolutional layer followed by a recurrent layer. The former acts as local feature learner while the latter is used to collect contexts. Compared with feed-forward neural networks, DANN learns contexts of local features from the very beginning. This setting helps to preserve hierarchical context evolutions which we show are essential to recognize similar actions. Besides, we present an adaptive method to determine the temporal size for network input based on optical flow energy, and develop a volumetric pyramid pooling layer to deal with input clips of arbitrary sizes. We demonstrate the advantages of DANN on two benchmarks HMDB51 and UCF101 and report competitive or superior results to the state-of-the-art.
The authors argue that the utilizing context early in an deep neural  network architecture for action recognition in videos is beneficial. To this  end, they propose a novel architecture called deep alternative neural network (DANN)  which can utilize context as early as possible. DANN uses volumetric convolution  layer in conjunction with a recurrent layer to incorporate context into the  architecture. They empirically  demonstrate the benefits of their proposed model on standard datasets, and  compare their performance against other state of the art methods. In addition to  exploring the contexts early in the architecture, their proposed model is also  capable of handling arbitrary sized inputs through the usage of a volumetric  pyramid pooling layer. They also provide a mechanism  for adaptive fragmentation of a given video that can be beneficial for improved  performance.
Detailed review:  The problem of action recognition is well established in the literature. The  authors provide an interesting deep neural network based approach for  leveraging the context early on in the network architecture. They empirically show  the advantage of exploring context as early as possible in the deep  architecture. They also demonstrate  that exploring contexts at different levels of the architecture is also beneficial.  The capability of handling arbitrary sized input is also useful.   Novelty :  Their usage of context early in the network seems novel. The capability  to use arbitrary length input is also helpful.   Clarity :  The explanations seem sound.   Technical Quality:  The authors seem to provide the necessary technical details.  They also provide implementation details about  the actual experiments performed by them.   However, there are some questions/doubts/comments which were brought up by  the manuscript.  1) In the lower equation which is part of the Equation set 2, what is u_{(i-1)j}^{xyz} on the RHS. Is it the output of the previous layer (which can theoretically  be a recurrent layer unrolled T times )?  2) In Figure 3, is the right side dotted box the expansion of output->output  cycle in the left side dotted box, where the cycle repeats T times ?  3) An example that explains the author's comments on line 97-98 would have been  useful.  4) Some clarification regarding line 110 would be useful.  5) Explanation regarding the claim presented in line 144-145 would be nice.  What causes 3 times speedup when 2 GPUs are used instead of 1. that using 2 GPUs can provide 3 times speedup as compare to 1 GPU ?  6) At what layers are the dropouts used ?  7) How many random video clips are selected per video during data augmentation ?  8) What is the benefit of using a sliding window approach during testing , while  using random clips during training? If the aim is to get video level score,  then can the testing also be done using fixed sized random clips.