Learning to represent videos is a very challenging task both algorithmically and computationally. Standard video CNN architectures have been designed by directly extending architectures devised for image understanding to include the time dimension, using modules such as 3D convolutions, or by using two-stream design to capture both appearance and motion in videos. We interpret a video CNN as a collection of multi-stream convolutional blocks connected to each other, and propose the approach of automatically finding neural architectures with better connectivity and spatio-temporal interactions for video understanding. This is done by evolving a population of overly-connected architectures guided by connection weight learning. Architectures combining representations that abstract different input types (i.e., RGB and optical flow) at multiple temporal resolutions are searched for, allowing different types or sources of information to interact with each other. Our method, referred to as AssembleNet, outperforms prior approaches on public video datasets, in some cases by a great margin.
Summary: 
This paper aims to adapt the standard neural architecture search scheme to search a two-input convolutional neural network for video representations. To this end, the paper formulates a direct acyclic graph with two input nodes (for RGB image and optical flow), where each node represents some pre-composed layers and edge represents the data flow with a trainable weight. The searching policy is a modified evolutionary algorithm, which is guided by the trainable weights on the edge, and a set of graph limitations are in-place to avoid over-complicated graphs. The best-selected model outperforms previous baselines and achieves a new state-of-the-art on two video datasets.
Overall, this paper presents a concrete application of neural architecture search for video CNN with interesting results. Edge-weight guided evolutionary algorithm also demonstrates a small improvement in the ablation study. My concern, as detailed later, is if the comparison only with previously human-designed models is necessary. Nevertheless, this paper presents an interesting application of NAS and discover a feasible way to conduct an evolutionary algorithm within a reasonable cost (less than 100 sampled architectures). 
Strength:
+ Writing in good shape, easy to follow and understand.
+ Motivation is clear and timely, reformulate neural architecture search for video representation is novel.
+ Clear experimental settings and reasonable convincing results.
Weakness:
- Lack of comparison with previous neural architecture search algorithms
Although the results yield that the proposed new search space is meaningful, considering each model has a similar dimension comparing to ResNet-50, it is still unknown if only comparing to human-designed model is a truly fair baseline. In my perspective, since this paper is built on top of NAS strategy with minor adaptation, could the author add one comparison experiment that, the proposed new search space is superior to those previous NAS spaces? For example, one could based on the earlier two-stream ResNet-50 with RGB+F modality, switch the backbone model into a NAS-based one and search with the same evolutionary algorithm (removing the edge weights adaptation). Otherwise, the improvement shown in the paper is not that surprising.