The success of deep learning research has catapulted deep models into production systems that our society is becoming increasingly dependent on, especially in the image and video domains. However, recent work has shown that these largely uninterpretable models exhibit glaring security vulnerabilities in the presence of an adversary. In this work, we develop a powerful untargeted adversarial attack for action recognition systems in both white-box and black-box settings. Action recognition models differ from image-classification models in that their inputs contain a temporal dimension, which we explicitly target in the attack. Drawing inspiration from image classifier attacks, we create new attacks which achieve state-of-the-art success rates on a two-stream classifier trained on the UCF-101 dataset. We find that our attacks can significantly degrade a model's performance with sparsely and imperceptibly perturbed examples. We also demonstrate the transferability of our attacks to black-box action recognition systems.
Summary:
This paper presents a framework for generating adversarial perturbations for videos. Specifically, the paper proceeds by using a standard image-based adversarial noise generation setup (such as the FGSM scheme), and applies it to the motion stream of a two-stream action recognition pipeline; this motion stream typically using optical flow images. As such flow generation is usually done offline and thus is not differentiable, the paper resorts to the recent FlowNet 2.0 scheme that uses an end-to-end learnable deep flow generation model. Three variants of the scheme are provided, (i) that perturbs all frames in a flow stack, (ii) that perturbs only a sparse set of frames as decided by the importance of a frame to action classification, and (iii) a variant of (ii) that recalculates the gradients for all frames if the ones selected in (ii) were not adversarial. Experiments are provided on UCF101 dataset and show promise. Analysis is presented on the transferability of  the learned noise to flow images generated via external means.
Strengths:
1) The different variants of the scheme and sparse selection of the frames to be perturbed are interesting. 
2) The paper makes some interesting observations, namely that (i) only a single frame perturbation might be sufficient to make the video adversarial, and (ii) perturbations computed via FlowNet models are not transferable to those with flow computed via external software -- which is often the practice.
Weaknesses:
1) I think the main weakness of this paper is the lack of any surprise/significant novelty in the presented approach. The main idea follows the common trend in adversarial noise generation for image classification problems, except that the inputs are a stack of frames instead of  a single one; however, such a setting do not seem to bring along any non-trivial challenges. In the second contribution of this paper -- on the sparse selection of frames to attack, there is a lack of clarity in how one would do the iterative attacks at test time, given such sparse frame selection is done via computing the frame level saliency values via the classification loss, which depends on ground truth class labels, which are unavailable at test time. 
2) There are previous works that have attempted video level adversarial perturbation generation, which the paper do not cite or contrast to; such as a few below. Further, the literature survey fails to provide any compelling motivation as to why video perturbation generation is any difficult than image based noise generation -- it does not appear so from the subsequent text that this problem deserves any special treatment in the considered context.
[a] Learning Discriminative Video Representations Using Adversarial Perturbations, Wang and Cherian, ECCV 2018
[b] Sparse Adversarial Perturbations for Videos, Wei et al., arxiv, 2018
3) It is unclear why the paper chose to consider flow produced by a FlowNet model as their inputs for the attack? Why not consider the flow images directly? Of course, the optical flow algorithm may not be differentiable, but that is perhaps besides the point; the focus should be in perturbing flow, in whatever way it is generated. To that end, given that flow (on static camera images) can be sparse, it would be interesting to see how would a perturbation be generated that needs to operate on local regions (where motion happens). In my opinion, using a FlowNet model for flow generation trivializes the proposed algorithm.
4) It could have been interesting if the paper also provided some qualitative results of the optical flow images generated by FlowNet after adding perturbations to the input frames. Are these flow images also quasi-impercitable? 
Overall, I think the paper has some observations that may be slightly interesting; however, it lacks novelty and the analysis or presentation are unconvincing.