We introduce a new convolutional layer named the Temporal Gaussian Mixture (TGM) layer and present how it can be used to efficiently capture longer-term temporal information in continuous activity videos. The TGM layer is a temporal convolutional layer governed by a much smaller set of parameters (e.g., location/variance of Gaussians) that are fully differentiable. We present our fully convolutional video models with multiple TGM layers for activity detection. The experiments on multiple datasets including Charades and MultiTHUMOS confirm the effectiveness of TGM layers, outperforming the state-of-the-arts.
Summary:
This paper proposes a temporal convolution layer called Temporal Gaussian Mixture (TGM) for video classification tasks. Usually, video classification models can be decomposed into a feature extraction step, in which frames are processed separately or in local groups, and a classification step where labels are assigned considering the full video features/longer range dependencies. Typical choices for the classification step are Recurrent Neural Networks (RNNs) and variations, pooling layers and 1D temporal convolutions. TGMs are 1D temporal convolutions with weights defined by taking samples from a mixture of Gaussians. This allows to define large convolutional kernels with a reduced number of parameters -the mean and std of the gaussians- at the cost of having reduced expressiveness. The authors experiment replacing 1D temporal convolutions with TGMs in standard video classifications models and datasets and report state-of-the-art results.
Strengths:
[+] The idea of defining weights as samples from a Mixture of Gaussians is interesting. It allows to define convolutional kernels that scale with the number of channels (equivalent to the number of Gaussians) instead of the number of channels and receptive field.
Weaknesses:
[-] The explanation of the TGM layers is very confusing in its current state. 
Certain aspects of TGMs that would help understand them are not clearly stated in the text. For example, it is not clear that 1) TGMs are actually restricted 1D convolutions as the kernel is expressed through a mixture of gaussians, 2) In a TGM the number of mixtures corresponds to the number of output channels in a standard 1D convolution and 3) despite TGMs are 1D convolutions, they are applied differently by keeping the number of frame features (output channels of a CNN) constant through them. Section 3 does not explain any of these points clearly and induces confusion.
[-] The comparisons in the experimental section are unfair for the baselines and do not justify the advantages of TGMs. 
TGMs are 1D convolutions but they are applied to a different axis - 1D convolutions take as input channels the frame features. Instead, TGMs add a dummy axis, keep the number of frame features constant through them, and finally use a 1x1 convolution to map the resulting tensor to the desired number of classes. There is no reason why TGMs can’t be used as standard 1D convolutions taking as the first input channels the frame features - in other words, consider as many input mixture components as frame features for the first TGM. This is a crucial difference because the number of parameters of TGMs and the baselines is greatly affected by it.
The implicit argument in the paper is that TGMs perform better than standard 1D convolutions because they have less parameters. However, in the experiments they compare to 1D convolutions that have smaller temporal ranges – TGMs have a temporal kernel of 30 timesteps or 10 per layer (in a stack of 3 TGMs), whereas the 1D convolutions either have 5 or 10 timestep kernels according to section 4. Thus, it is impossible to know whether TGMs work better because 1) in these experiments they have longer temporal range – we would need a comparison to 1D convolutions applied in the same way and with the same temporal range to clarify it, 2) because they are applied differently – we would need a comparison when TGMs are used the same way as 1D convolutions, 3) because the reduced number of parameters leads to less overfitting – we would need to see training metrics and see the generalization gap when compared to equivalent 1D convolutions or 4) because the reduced number of parameters eases the optimization process. To sum up, it's true that TGMs would have a reduced number of parameters compared to the equivalent 1D convolutions with the same temporal range, but how does that translate to better results and, in this case, why is the comparison made with non-equivalent 1D convolutions? What would happen if the authors compared to 1D convolutions used in the same way as TGMs?
[-] To claim SOTA, there are comparisons missing to recently published methods.
In particular, [1] and [2] are well-known models that report metrics for the Charades dataset also used in this paper.
Rating:
At this moment I believe the TGM layer, while a good idea with potential, is not sufficiently well explained in the paper and is not properly compared to other baselines. Thus, I encourage the authors to perform the following changes:
-Rewrite section 3, better comparing to 1D convolutions and justifying why TGMs are not used in the same manner but instead using a dummy axis and a linear/1x1 conv layer at the end and what are the repercussions of it.
-Show a fair comparison to 1D convolutions as explained above.
-If claiming SOTA results, compare to [1] or [2] which use different methods to the one proposed in these paper but outperform the baselines used in the experiments.
[1] Wang, Xiaolong, et al. "Non-local neural networks." CVPR 2018
[2] Zhou, Bolei et al. "Temporal Relational Reasoning in Videos" ECCV 2018
----------------------
Update: In light of the authors' rebuttal I have updated my rating from 5 to 6.