SPIGAN: PRIVILEGED ADVERSARIAL LEARNING FROM SIMULATION
Deep Learning for Computer Vision depends mainly on the source of supervision. Photo-realistic simulators can generate large-scale automatically labeled synthetic data, but introduce a domain gap negatively impacting performance. We propose a new unsupervised domain adaptation algorithm, called SPIGAN, relying on Simulator Privileged Information (PI) and Generative Adversarial Networks (GAN). We use internal data from the simulator as PI during the training of a target task network. We experimentally evaluate our approach on semantic segmentation. We train the networks on real-world Cityscapes and Vistas datasets, using only unlabeled real-world images and synthetic labeled data with z-buffer (depth) PI from the SYNTHIA dataset. Our method improves over no adaptation and state-of-theart unsupervised domain adaptation techniques.
This papers presents an unsupervised domain adaptation algorithm for semantic segmentation. A generative adversarial network is envisaged to carry out synthetic-to-real image translation. In doing so, depth information extracted from a simulator is used as privileged information (PI) to boost the transfer on the target domain, regularizing the model and ensuring a better generalization. 
*Quality*
The paper addresses a relevant problem, which is the adaptation of methods from simulated data to real ones. The authors devise a convincing method which takes advantage of state-of-the art generative adversarial architectures and privileged information. 
*Clarity*
The paper is sufficiently well written. In general, the main idea and proposed method are clear and easy to follow. The only problem is that some background concepts (such as privileged information or unsupervised domain adaptation) are given for granted, compromising the readability for someone not familiar with those topics. 
On a more technical side, for reproducibility purposes, the following aspects have to be clarified:
1.	Details about the validation set used for grid search. Is the validation set extracted from the target domain? (In principled labels from the target domain should not be used during learning).
2.	Number of iterations before convergence: is the training of the network numerically stable? Are there issues in convergence of some of the sub-modules? Which one is leading the learning?
3.	Comments about the relative magnitudes of losses. This will maybe give some intuitions about the values used for the hyper-parameters (e.g., the L_PI is only weighted by 0.1).
*Originality*
The way authors take advantage of depth information extracted from a simulator as privileged information is novel in the sense that, with respect to the original student-teacher paradigm of the paper by Vapnik & Vashist, here the idea of privileged information is interpreted as a regularizer to boost the training stage. 
*Significance*
The application of semantic segmentation in urban scenes for navigation tasks is relevant. The scored results are on pair with/ superior to state-of-the-art in unsupervised domain adaptation. 
However, the ablation study could be more extensive in order to understand the contribution of the several components, besides the PI network. In fact, it would be interesting to analyze the contribution of the perceptual loss (and others). Also, one could include the target-only result (as done in original LSD paper) to provide an upper bound on the best accuracy that is achievable.
*Pros*
1. The applicative setting of semantic segmentation in urban scenes for navigation is relevant. 
2. Using privileged information from simulators seems novel and well presented in this paper.
3. Strong experimental results achieved in challenging benchmarks.
*Cons*
1. The regularization effect of the PI network could be supported by a more extensive ablation study of the model, for example by ablating the several losses used (in particular, the perceptual loss).
2. A quite relevant amount of hyper-parameters need to be cross-validated. Is the method robust against different parameters’ configuration?
3. Missing citations [1, 2]: there are works in the literature that can hallucinate a missing modality during testing. Although such works approach a different problem, authors should cite them.
[1] Judy Hoffman, Saurabh Gupta, Trevor Darrell - Learning with Side Information through Modality Hallucination – CVPR 2016
[2] Nuno Garcia, Pietro Morerio, Vittorio Murino - Modality Distillation with Multiple Stream Networks for Action Recognition – ECCV 2018
*Final Evaluation*
The authors face the challenging synthetic-to-real adaptation setup, with an interesting usage of z-buffer from a simulator as privileged information. Overall, the work is fine, apart from the following points.
1.	In addition to a few missing citations [1, 2], an ablation study on the perceptual loss is necessary to dissect the impact of each component of the pipeline. 
2.	The clarity of the paper can be improved by adding some background material on unsupervised domain adaptation and learning with privileged information (PI), as to better highlight the technical novelty of using PI within a L1 regularizer. 
3.	The training stage of all submodules could have better investigated, for instance, by providing some convergence plots of the loss functions across iterations.
4.	How to do grid search for parameters in a domain adaptation setting is always a delicate aspect and authors seem elusive on that respect. 
5.	Again about hyper-parameters. Due to their high number, some sensitivity analysis should have provided.
As it is, the paper’s strengths slightly outperform the weaknesses, leading to an overall borderline-accept. If authors implement the suggested modification, a full acceptance will be feasible.
[COMMENTS AFTER AUTHORS' RESPONSE]
After the rebuttal provided by authors, all raised questions and criticisms have been fully solved. Therefore, I recommend for a full acceptance.