Attention in Convolutional LSTM for Gesture Recognition
Convolutional long short-term memory (LSTM) networks have been widely used for action/gesture recognition, and different attention mechanisms have also been embedded into the LSTM or the convolutional LSTM (ConvLSTM) networks. Based on the previous gesture recognition architectures which combine the threedimensional convolution neural network (3DCNN) and ConvLSTM, this paper explores the effects of attention mechanism in ConvLSTM. Several variants of ConvLSTM are evaluated: (a) Removing the convolutional structures of the three gates in ConvLSTM, (b) Applying the attention mechanism on the input of ConvLSTM, (c) Reconstructing the input and (d) output gates respectively with the modified channel-wise attention mechanism. The evaluation results demonstrate that the spatial convolutions in the three gates scarcely contribute to the spatiotemporal feature fusion, and the attention mechanisms embedded into the input and output gates cannot improve the feature fusion. In other words, ConvLSTM mainly contributes to the temporal fusion along with the recurrent steps to learn the long-term spatiotemporal features, when taking as input the spatial or spatiotemporal features. On this basis, a new variant of LSTM is derived, in which the convolutional structures are only embedded into the input-to-state transition of LSTM. The code of the LSTM variants is publicly available 2 .
SUMMARY:  This paper proposes a pipeline combining Res3D-3DCNN, convLSTM, MobileNet-CNN hybrid architecture for performing gesture recognition.  In particular, it explores the integration of pooling and neural attention mechanisms in ConvLSTM cells.  Four convLSTM variants are compared, which place pooling and attention mechanisms at different locations inside the cell. The pooling leads to a somewhat novel LSTM/convLSTM hybrid architecture.  Finally, an attention-inspired gating mechanism is proposed, with some differences to the formulation in [5].  The proposed architecture is evaluated on the Jester and ISOGD data sets and achieves state-of-the-art on the RGB-only setting of IsoGD.  QUALITY:  In the LSTM gates, fully connected weight matrices and matrix multiplication are used. Are there good reasons for doing this? In the original LSTM paper [3], a two vector Hadamard product is proposed (section II.A on the forward pass) instead of matrix multiplication. It will be of interest to compare the matrix-matrix or matrix-vector Hadamard-products in the gates. Following [3] it could also be interesting to explore gating based on output values instead of the cell state. The proposed max-pooling approach should allow this given that same padding was used in the convolutions, so that the spatial resolution inside and outside of the cell should remain the same.  Published attention mechanisms have thus far been based on the notion of probability distributions, e.g. in [r6], [r7] or even the cited [5] on which the current attention mechanism is based.  However, the proposed "attention" mechanism in equation (22) breaks the probabilistic interpretation by using a max instead of a sum operation.  The explanation given in 158-159 is not very clear, why using a max (and not a sum) will distribute the scores between 0-1.   Experimentation wise, all proposed modifications have been tested.  Additionally a comparison of the attention weights used in variants (c) and possibly a classical soft attention formulation as described in equation (18) would have been interesting in order to enable readers to asses the newly proposed mechanism from eq. (22). Overall the results are state of the art and suffice to support that the authors successfully reduced the number of convLSTM parameters though it is hard to asses the modified attention mechanism proposed by the authors.  In particular, looking at the supplementary material results, it looks as if (since the subjects are all stationary and the only movement in the scene is relevant to the hand gesture) that a simple flow detection would suffice.  To prove that the attention mechanism is actually working, one would need a baseline comparing against optical flow / movement detection.  Also is there some border effects in the attention?  There seem to be very strong response in the bottom edge of the scene for almost all samples?  Some comparisons to other methods are missing, e.g. the experimental results on the Jester-Data set are not compared to  state-of-the-art; they are better than [r4] but worse than [r2].  [r2] reports an accuracy of 96.33% on the Jester V1 data set (using flow and RGB) while [r1] reports an accuracy of 80.96% on the IsoGD-Data set.  The paper provides a comparison only to [8], and only to the RGB results, which are significantly worse (51.31% / 58.65% for RGBD).  Finally, some statistical analysis would make the results more convincing.           CLARITY: The paper is relatively easy to read and follow, though some important details are omitted, e.g.  What are the performance metrics being used for evaluation? What variant of gradient descent is used?  ORIGINALITY:  Attention has been shown to be beneficial in vision problems.  The pooling approach is somewhat interesting.   Additional related works on gesture recognition are given below in [r1, r2].  Other related works to convLSTM is [r5] which presents a convGRU; an interesting aside is [r3], which found that recurrent dropout may be applied on past state values during gate and candidate value computation. This means that for an RNN to function, past states must not be available in their original values at the gates.  This may be why GlobalAveragePooling can be regarded as an interesting choice in the convLSTM case when computing gate values.    SIGNIFICANCE:   The main contribution of the paper is the experimentation; the paper tests four variants of LSTMs to test the effects of attention in LSTMs.  It is found that spatial convolutions in the LSTM gates do not contribute to spatiotemporal feature fusion; furthermore, attention mechanisms embedded into the input and output gates also do not improve the feature fusion.  As such, the authors make a recommendation for a new cell architecture which reduces the number of required weights, leading to smaller memory footprints and less training time. Such a cell may also be interesting for those working in action recognition and video understanding.  The work adds to the body of literature for anyone working on integrating CNNs and recurrent architures.  EXTRA REFERENCES:  [r1] Gesture Recognition: Focus on the Hands by Pradyumna Narayana and J. Ross Beveridge in CVPR 2018 [r2] Motion Fused Frames: Data Level Fusion Strategy for Hand Gesture Recognition by Okan Köpüklü and Neslihan Köse and Gerhard Rigoll in CVPR Workshops 2018 [r3] Recurrent Dropout without Memory Loss by Stanislau Semeniuta, Aliaksei Severyn, Erhardt Barth in Proceedings of COLING 2016 [r4] Temporal Relational Reasoning in Videos by Bolei Zhou and Alex Andonian and Antonio Torralba  https://arxiv.org/pdf/1711.08496.pdf [r5] Deep Learning for Precipitation Nowcasting: A Benchmark and A New Model by Xingjian Shi et al. in (NIPS 2017) [r6] Listen, Attend and Spell by Chan, William and Jaitly, Navdeep and Le, Quoc V. and Vinyals, Oriol in 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) [r7] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.  Neural Machine Translation by Jointly Learning to Align and Translate. In ICLR 2015.