Semi-Supervised Learning for Optical Flow with Generative Adversarial Networks
Convolutional neural networks (CNNs) have recently been applied to the optical flow estimation problem. As training the CNNs requires sufficiently large amounts of labeled data, existing approaches resort to synthetic, unrealistic datasets. On the other hand, unsupervised methods are capable of leveraging real-world videos for training where the ground truth flow fields are not available. These methods, however, rely on the fundamental assumptions of brightness constancy and spatial smoothness priors that do not hold near motion boundaries. In this paper, we propose to exploit unlabeled videos for semi-supervised learning of optical flow with a Generative Adversarial Network. Our key insight is that the adversarial loss can capture the structural patterns of flow warp errors without making explicit assumptions. Extensive experiments on benchmark datasets demonstrate that the proposed semi-supervised algorithm performs favorably against purely supervised and baseline semi-supervised learning schemes.
Summary:
The paper presents a semi-supervised approach to learning optical flow using a generative adversarial network (GAN) on flow warp errors.  Rather than using a handcrafted loss (e.g., deviation of brightness constancy + deviation from smoothness) the paper explores the use of a GAN applied to flow warp errors.
Strengths:
+ novel semi-supervised approach to learning; some concerns on the novelty in the light of [21]
+ generally written well
Weaknesses:
- some key evaluations missing
Comments:
Supervised (e.g., [8]) and unsupervised (e.g., [39]) approaches to optical flow prediction have previously been investigated, the type of semi-supervised supervision proposed here appears novel.  The main contribution is in the introduction of an adversarial loss for training rather than the particulars of the flow prediction architecture.  As discussed in Sec. 2, [21] also proposes an adversarial scheme.  A further comparative discussion is suggested.  Is the main difference in the application domain?  This needs to be clarified.  Also the paper claims that "it is not making explicit assumptions on the brightness constancy".  While the proposed approach does not use an Lp or robust loss, as considered in previous work, the concept of taking flow warp error images as input does in fact rely on a form of brightness constancy assumption.  The deviation from prior work is not the assumption of brightness constancy but the actual loss.
In reference to [39], it is unclear what is meant by the approach "requires significantly more sophisticated data augmentation techniques" since the augmentation follows [8] and is similar to that used in the current paper.
The main weakness of the paper is in the evaluation.  One of the main stated contributions (line 64) is that the proposed approach outperforms purely supervised approaches.  Judging from Table 3, this may be a gross misstatement.  For example, on Sintel-Clean FlowNet2 achieves 3.96 aEPE vs 6.28 by the proposed approach.
Why weren't results reported on the KITTI test sets using the test server?  Are the reported KITTI results including occluded regions (in other words all points) or limited to non-occluded regions?
The limitations of the presented training procedure (Sec. 4.4) should have been presented much earlier.  It would have been interesting to train the network with data all from the same domain, e.g., driving scenes.  For instance, rather than train on FlyingChairs and KITTI raw, it would have been interesting to train on KITTI raw (no ground truth) and Virtual KITTI (with ground truth) and evaluate on KITTI TEST.
Reported fine-tuned results should be presented for all approaches.
Minor point:
- line 22 "Due to the lack of the large-scale ground truth flow" --> "Due to the lack of large-scale ground truth flow"
- line 32: "With the limited quantity and unrealistic of ground truth" 
Overall rating:
The training approach itself appears to be novel (a more detailed comparative discussion with respect to [21] is suggested) and will be of interest to those working on flow and depth prediction.  Based on the issues cited with the evaluation I cannot recommend acceptance at this point.
Rebuttal:
The reviewer acknowledges that the authors have addressed some of my concerns/suggestions regarding their evaluation. The performance is still below that of the state-of-the-art. Finetuning may help but has not been evaluated.  Is the performance deficit, as suggested by the authors, due to the capacity of their chosen network or is it due to their proposed GAN approach? Would the actual training of a higher capacity network with the proposed adversarial loss be problematic? These are topics to be explored.