Towards Robust Detection of Adversarial Examples
Although the recent progress is substantial, deep learning methods can be vulnerable to the maliciously generated adversarial examples. In this paper, we present a novel training procedure and a thresholding test strategy, towards robust detection of adversarial examples. In training, we propose to minimize the reverse crossentropy (RCE), which encourages a deep network to learn latent representations that better distinguish adversarial examples from normal ones. In testing, we propose to use a thresholding strategy as the detector to filter out adversarial examples for reliable predictions. Our method is simple to implement using standard algorithms, with little extra training cost compared to the common cross-entropy minimization. We apply our method to defend various attacking methods on the widely used MNIST and CIFAR-10 datasets, and achieve significant improvements on robust predictions under all the threat models in the adversarial setting.

Update: Thank you for addressing my comments. Please make sure to update your paper according to your responses (in particular about adding additional experimental results in the appendix and updating the current set of results).  In this paper, the authors propose a new loss (i.e., reverse cross-entropy) whose goal is to force non-maximal probability predictions to be uniform (in a classification task). In other words, the goal of the classification network is to maximize the entropy of all non-maximal softmax outputs. The authors show that the intermediate representations learned by the network are more meaningful when it comes to compare real and adversarial examples. Experiments are done on MNIST and CIFAR-10.  The paper is well written and clearly explained. The proposed loss is analyzed both in practice and in theory. I am a bit concerned with novelty as the "reverse cross-entropy" loss is not strictly new (as pointed out by the authors themselves in Section 3.2). However, the added analysis and explanation w.r.t. the use-case of detecting adversarial examples is valuable to the community. The authors provide enough evidence that this loss is useful and, as such, I am leaning towards acceptance. The experimental section would benefit from a few added experiments.  1) In general, the problem of detecting adversarial examples is important. However, the proposed solutions are not yet compelling. The authors explain that "in semi-autonomous systems, the detection of adversarial examples would allow disabling autonomous operation and requesting human intervention", but the detection method is often easier to attack - and keeping the system from being used can also have serious consequences. The authors should elaborate on this problem and also include experiments that only aim to disable the classifiers. In general the AUC of the detector remains poor and I am assuming that a lot of non-adversarial examples are still detected as adversarial.  Details: a) There has been a lot of exciting and new work on verification and training provably robust networks (see Wong & Kolter arXiv:1805.12514, arXiv:1711.00851 and Krishnamurthy et al. arXiv:1805.10265, arXiv:1803.06567). They are definitely orthogonal approaches to preventing misclassification of adversarial perturbations, but I feel they should find a space in your introduction.  b) As previously stated, it would useful to incorporate results about the minimal detectable perturbation radius.  c) Fig 2(a) should include Carlini & Wagner, I am unsure why a separate plot is needed. Similarly, the analysis done in Fig. 2(b) could be extended to include the other attacks.


Bayesian Model-Agnostic Meta-Learning
Due to the inherent model uncertainty, learning to infer Bayesian posterior from a few-shot dataset is an important step towards robust meta-learning. In this paper, we propose a novel Bayesian model-agnostic meta-learning method. The proposed method combines efficient gradient-based meta-learning with nonparametric variational inference in a principled probabilistic framework. Unlike previous methods, during fast adaptation, the method is capable of learning complex uncertainty structure beyond a simple Gaussian approximation, and during meta-update, a novel Bayesian mechanism prevents meta-level overfitting. Remaining a gradientbased method, it is also the first Bayesian model-agnostic meta-learning method applicable to various tasks including reinforcement learning. Experiment results show the accuracy and robustness of the proposed method in sinusoidal regression, image classification, active learning, and reinforcement learning.

The paper introduces a Bayesian variant of the model-agnostic meta-learning (MAML) algorithm. MAML works by trying to find an "initial" parameter point from which one can adapt well with a few gradient update steps to different tasks. The first extension (Bayesian fast adaptation) generalizes this into an ensemble, where a set of particles is updated using Stein variational gradient descent to approximate task training posteriors and the validation loss that is used for the meta-update. BMAML extends this further such that the "validation" loss is changed to a loss between the (approximate) posteriors of the task-training and task-training-plus-validation distributions. MAML and BMAML (together with ensemble version of MAML) are compared in regression, classification, and reinforcement learning experiments.  Paper is mainly excellently written and the method is very interesting extension of MAML that seems to clearly improve the performance in the presented experiments. Yet, I have some confusions about the interpretation of the method.  Main comments: (1) The proposed method seems very much tied to a computational algorithm (SVGD) rather than be a model or procedure description in a Bayesian sense (where one doesn't usually need to specify specific computational algorithms, but models and/or utilities). Also, the used number of SVGD steps seems so small that it seems questionable if the distributions have converged such that they could be thought of as even approximately the posteriors. And if they would be ran to convergence, the initial particles would not have an effect anymore? This makes the meaning of, for example, the train-task posterior and it's conditioning on Theta_0 very vague. (Unless I misunderstood some parts of the method.) (2) The interpretation of the initial particles Theta_0 seems somewhat vague in the manuscript (see also comment 1). Can they be interpreted as representing an implicit prior (or is the implicit prior a combination of these and the SVGD instance, as Grant et al.'s interpretation seems to be based on a quick glance, but which point of view is not given in this paper)? (3) The BMAML meta-objective is trying to find a set of initial particles, such that the approx. posterior with task-train data is similar to the approx. posterior of the task-train-plus-validation data. Again from a Bayesian perspective, this sounds a bit weird that one tries to match posteriors where the only difference is that one has more data than the other (and the somewhat vague conditioning on Theta_0). On the other hand, a practical interpretation of the this is that one is trying to fit the initial particles such that adding the validation data to the training data doesn't change the parameter distribution much. (4) Wouldn't another way of trying to avoid the overfitting of Bayesian fast adaption be to use a proper scoring rule or some such loss function that would evaluate the full posterior predictive distribution and require honest uncertainties rather than a point prediction in Equation 5? (5) Why are there no comparisons to Grant et al. paper? Would it be too computationally costly for the experiments here? Also, why isn't the Bayesian fast adaptation method included in the results? It is mentioned that initial experiments found that it might overfit, but it would still be interesting to compare against to understand where the main improvement of BMAML comes from. (6) One mentioned motivation is that Gaussian approx. as in Grant et al. would not be appropriate in many cases (e.g., BNNs). Yet, the number of particles used is rather small and one would not expect it to be able to represent any complicated posteriors very well. (7) Will code be made available?  In summary, it seems that my main confusions arise from the interpretation of Theta_0 and the interpretation of the distributions, e.g., p(theta_tau | D^trn_tau, Theta_0), as posteriors while actually the conditioning on Theta_0 seems to require that the inference procedure is not run to convergence, so that they would actually represent the posteriors. Hopefully, the authors could clarify the interpretation of the method (or point out my misunderstandings). Apart from this, I think the method seems to be an interesting extension of MAML and would be a relevant development of this line of work for NIPS.  Update after author feedback: I appreciate the authors' careful response and hope that they revise the manuscript in line with it. I'm still a bit conflicted about the approach from a Bayesian point of view, as it explicitly relies on not running the inference procedure (SVGD) for too long (that is, to actual convergence), since then the effect of the initial set of parameters Theta_0, and hence the meta-learning, would be lost (at least in theory, although possibly not in practice fully). And I'm wondering whether the approach could be formulated in some more principled manner with regard to this issue (e.g., the more direct way of using a proper scoring rule; I'm not sure if I agree that this would be a less "principled Bayesian" way of formulating the problem). On the other hand, from a practical point of view, I think the approach is very interesting and, overall, I think the work is certainly worth publishing. And I guess actually the original MAML has much of this same flavour of as it relies on a limited number of gradient updates (but, of course, it's not and not advertised as a Bayesian method).


Playing hard exploration games by watching YouTube
Deep reinforcement learning methods traditionally struggle with tasks where environment rewards are particularly sparse. One successful method of guiding exploration in these domains is to imitate trajectories provided by a human demonstrator. However, these demonstrations are typically collected under artificial conditions, i.e. with access to the agent's exact environment setup and the demonstrator's action and reward trajectories. Here we propose a two-stage method that overcomes these limitations by relying on noisy, unaligned footage without access to such data. First, we learn to map unaligned videos from multiple sources to a common representation using self-supervised objectives constructed over both time and modality (i.e. vision and sound). Second, we embed a single YouTube video in this representation to construct a reward function that encourages an agent to imitate human gameplay. This method of one-shot imitation allows our agent to convincingly exceed human-level performance on the infamously hard exploration games MONTEZUMA'S REVENGE, PITFALL! and PRIVATE EYE for the first time, even if the agent is not presented with any environment rewards.

Summary: This paper studies learning from demonstration/imitation learning. While usually supervision has to be intentionally provided by a human, the authors instead use YouTube videos as a form of supervision. They first align videos to a shared representation, using two concept that do not require any labeling: i) predicting the temporal distance between frames in the same video, and ii) predicting the temporal distance between a video and audio frame of the same video. Subsequently, they use the embedding on a novel video to generate checkpoints, which serve a intermediate rewards for an RL agent. Experiments show state-of-the-art performance amongst learning from demonstration approaches.   Strength: - The paper is clearly written, with logical steps between the sections, and good motivations. - Related work is well covered. - The results are impressive, and exceed the state-of-the-art on the studied games.   Weakness/Questions: - Introduction: You mention intrinsic motivation, but you miss another branch of exploration work, based on uncertain value functions. See for example [1,2].  -In the beginning of Section 4 I did not understand why you could suddenly sample checkpoints, while in Sec. 3 you only focused on learning a frame-wise embedding. The end of Sec 5. does explain how this is implemented (with a held-out video), which could be mentioned a bit earlier for clarity. - How much did you select the YouTube demonstrations? Were they expert players, who were very good at solving these games? That might explain why you results are better than e.g. DqfD? - Atari games have a fixed viewpoint, and you additionally spatially align the demonstrations if the videos don’t. To what extend is spatial alignment crucial? Would this approach scale to first-person-view games, where alignment is probably much harder?   Conclusion:  I believe this paper studies an interesting problem. There is big potential for the use of (human) supervision to guide learning, but supervision is a time consuming process. This paper studies video data as a potential form of supervision, which would indeed provide access to a huge amount of demonstrations. The paper contributes a novel method for alignment, and a simple but effective way to incorporate this embedding in an RL agent, which shows good results. It would be good to mention how expert the demonstration videos were, and maybe discuss the potential to generalize to non-fixed viewpoints. Besides that, I think this paper makes a clear contribution.   [1] White, D. J. "Uncertain value functions." Management Science 19.1 (1972): 31-41. [2] Osband, Ian, et al. "Deep exploration via randomized value functions." arXiv preprint arXiv:1703.07608 (2017).  After rebuttal: It is important to mention the scores and acquisition procedure of the demonstration videos indeed. They were expert demonstrations, but as you mention yourself, a benefit of your approach is that (through the alignment) you can leverage a series of unlabeled expert demonstrations.


Pipe-SGD: A Decentralized Pipelined SGD Framework for Distributed Deep Net Training
Distributed training of deep nets is an important technique to address some of the present day computing challenges like memory consumption and computational demands. Classical distributed approaches, synchronous or asynchronous, are based on the parameter server architecture, i.e., worker nodes compute gradients which are communicated to the parameter server while updated parameters are returned. Recently, distributed training with AllReduce operations gained popularity as well. While many of those operations seem appealing, little is reported about wall-clock training time improvements. In this paper, we carefully analyze the AllReduce based setup, propose timing models which include network latency, bandwidth, cluster size and compute time, and demonstrate that a pipelined training with a width of two combines the best of both synchronous and asynchronous training. Specifically, for a setup consisting of a four-node GPU cluster we show wall-clock time training improvements of up to 5.4× compared to conventional approaches.

This paper proposed a pipelined training setup for neural nets. The pipeline structure allows computation and communication to be carried out concurrently within each worker.     Experiments show speed improvements over existing methods. Overall I think the method and the evaluations are convincing. However, I believe the NIPS conference is not the appropriate venue for this paper for the following reasons.    This paper deals with a high performance computing problem, namely, how to get more throughput (computation) in a distributed network. This is a low level computer systems problem that does not involve machine learning, statistics or neural science.    As much as it is designed for deep neural nets, the nature of the approach is not really about improving SGD itself or neural nets architecture or coming up with a new loss.     In summary, I do believe this work has the potential of changing the basic library for distributed neural nets training because of its improved performance. I just feel that this paper would fit better to other high performance computing conferences, like many of the cited related work.


Scalable Hyperparameter Transfer Learning
Bayesian optimization (BO) is a model-based approach for gradient-free black-box function optimization, such as hyperparameter optimization. Typically, BO relies on conventional Gaussian process (GP) regression, whose algorithmic complexity is cubic in the number of evaluations. As a result, GP-based BO cannot leverage large numbers of past function evaluations, for example, to warm-start related BO runs. We propose a multi-task adaptive Bayesian linear regression model for transfer learning in BO, whose complexity is linear in the function evaluations: one Bayesian linear regression model is associated to each black-box function optimization problem (or task), while transfer learning is achieved by coupling the models through a shared deep neural net. Experiments show that the neural net learns a representation suitable for warm-starting the black-box optimization problems and that BO runs can be accelerated when the target black-box function (e.g., validation loss) is learned together with other related signals (e.g., training loss). The proposed method was found to be at least one order of magnitude faster than competing methods recently published in the literature.

This paper proposes a novel Bayesian Optimization approach that is able to do transfer learning across tasks while remaining scalable.  Originality: This is very original work. Bayesian Optimization can work with any probabilistic regression algorithm, so the use of Bayesian linear regression to make it more scalable is well-known, as are its limitations (e.g. it doesn’t extrapolate well). The main novelty here lies in the extension to multi-task learning, which allows it to benefit from prior evaluations on previous tasks. When such evaluations are available, this can provide a significant advantage. The really clever part, however, is to use a neural net to learn a shared representation of the HPO space, and to learn this representation while performing Bayesian Optimization.  Sections 1 and 2 provide a nice overview of related work, which seems quite complete to me.  One very recent related paper that was not mentioned (maybe the authors were unaware of it) is the following arXiv paper: https://arxiv.org/abs/1802.02219 It does transfer learning by storing surrogate models of past tasks, and learns how to weight them in an ensemble during the Bayesian Optimization.  One additional paper that comes to mind (since it also does transfer learning to warm-start BayesOpt) is the following: Wistuba, M. and Schilling, N. and Schmidt-Thieme, L. Learning hyperparameter optimization initializations, DSAA 2015  Clarity: The paper is very well written. It does require solid knowledge of Gaussian Processes to be fully understandable, but otherwise the method and experiments are explained concisely but clearly.  Note: sometimes the explanations are a bit too concise, maybe because of page limits. For instance: L342: “When leveraging the auto-grad operators for the Cholesky” -> Cholesky decomposition.  What is not entirely clear from the paper is how the method learns similarity between tasks/black box functions. Previous work always aims to learn how similar other tasks are, so it can transfer from the right tasks. From my understanding, it seems that the neural net does something similar in that the shared representation captures black-box function similarities. However, I didn’t find this explained very clearly. The results on 30 different OpenML datasets do seem to indicate that it learns similarity. Can this method also tell how similar a new task is to previous tasks, or actually, how similar a its black box function is to previous black box functions?  Significance: This is a very significant contribution. Scalable transfer learning is somewhat of a holy grail in Bayesian optimization, so this paper will very likely have a significant impact, and within the right setting it will likely see significant adoption. Within the scope of the presented experiments, the method seems to work well.  Quality: This is technically a very strong paper. The experiments are nice, although also limited in several regards:  First, the current experiments are a bit limited in that they explore the hyperparameter spaces of single algorithms/functions. These spaces are typically still smallish (3-10 hyperparameters). It would be interesting to see it used on much larger hyperparameter spaces. Other related work, e.g. auto-sklearn with warm-starting, does operate on much larger hyperparameter spaces (e.g. entire pipelines containing a wide range of algorithms and all their hyperparameters).  Second, the evaluation on OpenML datasets compares only to random search and GPs with transfer. It does mention that DNGO and BOHAMIANN were too slow or too unstable. However, it is not clear why other work that uses warm-starting or transfer learning is not compared against, even though this should be relatively easy on OpenML datasets?   It would be good to at least comment on these limitations, or better, add the comparisons.  Update: I grew more worried after reading the author responses. The question on comparing with other scalable HPO techniques was not answered directly. Why not compare against other established and robust and scalable AutoML methods, such as AutoSKLearn/SMAC? You could run that against the same surrogate benchmark? I agree that the methods that are currently compared against are in a way more related, but they also seem not scalable enough. One could argue that the method is interesting enough in itself and that the community can run further comparisons, but sadly there is no code available and the authors did not respond that it will be available after publication. This worries me.


Joint Autoregressive and Hierarchical Priors for Learned Image Compression
Recent models for learned image compression are based on autoencoders that learn approximately invertible mappings from pixels to a quantized latent representation. The transforms are combined with an entropy model, which is a prior on the latent representation that can be used with standard arithmetic coding algorithms to generate a compressed bitstream. Recently, hierarchical entropy models were introduced as a way to exploit more structure in the latents than previous fully factorized priors, improving compression performance while maintaining end-to-end optimization. Inspired by the success of autoregressive priors in probabilistic generative models, we examine autoregressive, hierarchical, and combined priors as alternatives, weighing their costs and benefits in the context of image compression. While it is well known that autoregressive models can incur a significant computational penalty, we find that in terms of compression performance, autoregressive and hierarchical priors are complementary and can be combined to exploit the probabilistic structure in the latents better than all previous learned models. The combined model yields state-of-the-art rate-distortion performance and generates smaller files than existing methods: 15.8% rate reductions over the baseline hierarchical model and 59.8%, 35%, and 8.4% savings over JPEG, JPEG2000, and BPG, respectively. To the best of our knowledge, our model is the first learning-based method to outperform the top standard image codec (BPG) on both the PSNR and MS-SSIM distortion metrics.

Summary ======= This paper extends the autoencoder trained for compression of Balle et al. (2018) with a small autoregressive model. The autoencoder of Balle uses Gaussian scale mixtures (GSMs) for entropy encoding of coefficients, and encodes its latent variables as side information in the bit stream. Here, conditional Gaussian mixtures are used which additionally use neighboring coefficients as context. The authors find that this significantly improves compression performance.  Good ==== – Good performance (notably, state-of-the-art MS-SSIM results without optimizing directly on this metric) – Extensive supplementary materials, including rate-distortion curves for individual images – Well written  Bad === – Incremental, with no real conceptual contributions – Missing related work: There is a long history of conditional Gaussian mixture models for autoregressive modeling of images – including for entropy rate estimation – that is arguably more relevant than other generative models mentioned in the paper: Domke et al. (2008), Hosseini et al. (2010), Theis et al. (2012), Uria et al. (2013), Theis et al. (2015)


Differentially Private Bayesian Inference for Exponential Families
The study of private inference has been sparked by growing concern regarding the analysis of data when it stems from sensitive sources. We present the first method for private Bayesian inference in exponential families that properly accounts for noise introduced by the privacy mechanism. It is efficient because it works only with sufficient statistics and not individual data. Unlike other methods, it gives properly calibrated posterior beliefs in the non-asymptotic data regime.

After considering author responses & other reviews, I keep my score unchanged. Some additional comments to the author responses  On truncation bounds (rows 23-26): In my opinion, this definitely deserves to be mentioned in the paper. I agree that there are plenty of problems where a reasonable, say, 95%-bound is known accurately enough a priori. However, this is still an exception and  more generally, especially in the strict privacy regime you are currently using, I would guess the bounds estimated under DP could actually be quite a bit off from the true values. Given that the truncation bound is a key hyperparamater that could completely break the method, I encourage you to add some (even simple) testing on this to the paper.  Ops vs Private MCMC (44-47): I don't doubt that the calibration would be more or less off, the question in my mind is how much. But granted, I agree it's not generally feasible, or necessary in this case, to start comparing against (eps,delta)-methods as well.  Other comments (14-17,41-43): Thank you, I'm happy with these.   Differentially Private Bayesian Inference for Exponential Families  The paper considers Bayesian posterior estimation for exponential family models under differential privacy (DP). More concretely, the main problem is how to infer a well-calibrated posterior distribution from noisy sufficient statistics, where the noise guarantees privacy by standard arguments. As a solution, the authors propose 2 versions of a Gibbs sampler: one for sampling from a general exponential family model with bounded sufficient statistics, and a variant that works with unbounded sufficient statistics but assumes the data is very low-dimensional.  The main novelty in the approach is to include the DP noise properly into the modelling instead of simply "forgetting" it and treating the actually noisy posterior as a noiseless one, as is often done in existing literature. Intuitively, this should lead to better calibrated posteriors, which also is the case in the empirical tests shown in the paper. The idea is not totally unheard of (see e.g.  references in the paper), but the formulation is nice, and the solution is novel. The problem as such is certainly an important one that needs to be solved properly, and the paper is a good step in the right direction.  The paper is clearly written and the main ideas are easy to understand. The mathematical proofs provided in the paper are mostly already known or small modifications to existing proofs. There doesn't seem to be any major problems with the theory, and it seems well-suited to the problem.  Pros:     * Including the DP noise into the modelling is a very relevant idea, and there does not seem to be      much papers on the topic.     * The proposed method is novel, and seems to give well-calibrated posteriors with limited sample sizes     under strict privacy.  Cons:     * The proposed solution has an exponential dependency on the dimensionality of the (unbounded)      sufficient statistics (as noted by the authors).     * No theory or testing on how the assumed bounds [a,b] for s_c affect the results.      * Fairly simplistic empirical tests.  Specific questions/requests for the authors:  1) Should the bounds [a,b] for the central interval s_c be given as (fixed) input to Algorithm 2? It seems like they are fixed in the algorithm, but are not provided as input or elsewhere.  2) How much the assumed bounds for s_c affect the results? Since the bounds affect the sensitivity, and thus the DP noise variance in the Laplace mechanism, the results can be made arbitrarily bad by choosing very poor bounds. Even if the bounds can be estimated under privacy (as suggested) and hence are unlikely to be totally off, the estimate is going to be noisy. It seems important to know how much this affects the reported results. Currently, I can't seem to find the bounds used in the actual tests reported in the paper anywhere, nor can I find any hint of how they were chosen. Additionally, it would be nice to have at least some test which shows the performance degradation with sub-optimal bounds.  3) On lines 247-48 you say "OPS suffices as a simpler straw man comparison". Do you think Private MCMC sampling would do better than OPS, or why is OPS a straw man? And if so, do you think you should then also compare to the private MCMC in addition to OPS?  Some minor comments/questions:  i) On line 36 you define calibration as being able to efficiently estimate p(theta|y). I would think calibration refers to something closer to the idea presented in the evaluation section (starting from line 252). This could be clarified.  ii) On line 61 you say your "contribution is a simple release mechanism A" etc. Since the release mechanism is simply adding Laplace noise to the sufficient statistics, i.e., doing the standard stuff (see lines 111-12), I don't think this should count as a contribution, or did I misinterpret something?  iii) On line 226 you mention that s_c needs to be drawn afresh for updating sigma^2. What happens if you use the previously drawn value used in s=s_l+s_c+s_u?  iv) I hope you release the code for the paper (at least after the paper is published).


LF-Net: Learning Local Features from Images
We present a novel deep architecture and a training strategy to learn a local feature pipeline from scratch, using collections of images without the need for human supervision. To do so we exploit depth and relative camera pose cues to create a virtual target that the network should achieve on one image, provided the outputs of the network for the other image. While this process is inherently non-differentiable, we show that we can optimize the network in a two-branch setup by confining it to one branch, while preserving differentiability in the other. We train our method on both indoor and outdoor datasets, with depth data from 3D sensors for the former, and depth estimates from an off-the-shelf Structure-from-Motion solution for the latter. Our models outperform the state of the art on sparse feature matching on both datasets, while running at 60+ fps for QVGA images.

Authors introduce a novel method for learning local feature detector and descriptors in a single framework. Compared to previous works, this method uses pair of images (instead of pair of image patches) and employs data with estimated depth maps and camera parameters. Learning on full images is achieved with an image loss which bypasses the lack of differentiability by computing gradient for only one branch. Even though the method is novel, I have concerns regarding the clarity and correctness of the evaluation.  The main strength of this work is the fact that it uses whole images for training the local feature detector. Thanks to the trick of using a global feature representation, authors are able to train with multiple images at once which mitigates the issue of an SGD update fitting a single image. Another claim, that the paper is able to discover valid points from scratch is unclear as it uses SFM results which, up to certain extent depend on quality of SIFT detector and descriptors. This makes their claim weaker and comparable to [1] which as well does not use an external local feature detector to bootstrap the training (and authors forget to cite this work).  However, my main concern is the lack of detail in the evaluation. As shown in many previous works, for evaluation of local feature matching algorithms, the measurement region size is a crucial parameter for a descriptor performance. Thus it is important to make sure that similar measurement regions size is used for all evaluated baselines in order to compare them fairly. A common approach is to evaluate the descriptor separately on pre-extracted patches [2, 3] which makes sure that the descriptors have access to the same information. And not all datasets have "heavily saturated scores". In this regard, it is not clear whether this has been taken into consideration. Similarly, it is not clear whether a similar amount of detected points was considered for each algorithm, as this can influence both the qualitative and quantitative evaluation.  However, a curse of local features is that each manuscript uses a different evaluation, thus a direct comparison is difficult. Separate evaluation of the detector a descriptors can yield some insight into the source of the main improvements, as e.g. the geometric precision of the detector is important for SFM application etc. Additionally, a speed comparison to other algorithms is missing. Comparison of performance on existing datasets would also be useful (such as in [4]), as it would allow to directly compare the numbers for the investigated baselines.   The writing does not seem to be well organized, mainly the Method section. The Figure 1 is hard to follow and explanation of branch i and j, used in section 3.2 is described in section 3.3. Similarly, it is not clear how the scores are combined across scales (Equation 1, across what elements is the sofmtax applied?), or how is scale taken into consideration while generating the "clean score maps", or how many points are considered for g. Similarly, it is not clear how the "split" mode of training works, considering that the orientation estimation is not being trained (line 203 and 273). It is not clear whether the common feature map is pre-trained or trained from scratch and how the camera parameters are used in the CNN (probably issue of wording).  Even though the method is quite interesting, my decision is mainly based on the issues with the evaluation which does not allow to verify fair comparison to existing methods. Additionally, I feel this submission would be more apt for computer vision conferences.  [1] Learning covariant feature detectors. K Lenc, A Vedaldi - Workshop of European Conference on Computer Vision, 2016. [2] Learning local image descriptors. S Winder and M Brown, CVPR 2007. [3] HPatches: A benchmark and evaluation of handcrafted and learned local descriptors, Balntas et al., CVPR 2017 [4] LIFT: Learned Invariant Feature Transform, Yi et al, ECCV 2016.  -------------------- Rebuttal feedback  I feel that the authors have addressed my main concerns really well and I am willing to improve the "overall score". It seems that the additional results provided in the rebuttal give additional evidence that the method is superior to existing ones and thus I would recommend the authors to include those in the final version of the paper together with improving the clarity of the text.  I am still slightly skeptical about their definition of the support region, provided as an average size of the region in pixels, which is a bit unconventional but ultimately seems to make sense. Traditionally, one uses magnification factor, however this actually addresses the issue of a possible bias in the detected scales.


Almost Optimal Algorithms for Linear Stochastic Bandits with Heavy-Tailed Payoffs
In linear stochastic bandits, it is commonly assumed that payoffs are with subGaussian noises. In this paper, under a weaker assumption on noises, we study the problem of linear stochastic bandits with heavy-tailed payoffs (LinBET), where the distributions have finite moments of order 1 + ϵ, for some ϵ ∈ (0, 1]. We rigorously analyze the regret lower bound of LinBET as Ω(T 1 1+ϵ ), implying that finite moments of order 2 (i.e., finite variances) yield the bound of Ω( √ T ), with T being the total number of rounds to play bandits. The provided lower bound also indicates that the state-of-the-art algorithms for LinBET are far from optimal. By adopting median of means with a well-designed allocation of decisions and truncation based on historical information, we develop two novel bandit algorithms, where the regret upper bounds match the lower bound up to polylogarithmic factors. To the best of our knowledge, we are the first to solve LinBET optimally in the sense of the polynomial order on T . Our proposed algorithms are evaluated based on synthetic datasets, and outperform the state-of-the-art results.

This paper studies linear stochastic bandits with heavy-tailed payoffs. Heavy-tailed means that the distributions have finite moments of order 1+epsilon for some 0
Reviewer 2
EDIT: I read your rebuttal, which is excellent and very informative, thank you. This is great work, I hope you'll make it as reproducible as possible by posting your code online afterwards :)  This paper analysis the stochastic linear bandits under heavy tailed noise models.  As this is not a novel problem, the authors clearly link their work with existing algorithms and results and show how they are able to improve on these existing bounds. They prove a lower bound under (1+\epsilon)-moment assumption, analysis their algorithms both theoretically and empirically. I was able to check most of the technical material in the supplementary paper and I believe it is sound and some results are even of independent interest (confidence ellipsoids for truncated observation for instance). I recommend acceptance.  Comments and questions:  - Could be nice to give a sort of guideline: for which problem should I use which algorithm ? I find it not very clear in the experiments why you chose either of them for the problems you defined.   - Table 1: I don’t see quite well the added value of the last two columns… the 5th is clearly redundant and the last one is not very informative, is it ?  - In the experiments, rather than describing the Table 1, I would have liked to read the rationale of the choices you made. Why comparing Student’s and Pareto ? Why choosing those parameters and not bigger or smaller ones ? I mean, given that you already give the details in the table, the text could be used to justify those choices.  - Experiments again: is it clear that you recover empirically the T^{1/(1+\epsilon)} rate that you obtained theoretically ? That would have deserved a comment.
Reviewer 3
The paper studies linear stochastic bandit problems with heavy-tailed noise. For a noise distribution with finite moments of order 1+epsilon, authors show algorithms with regret bounds that scales as O(T^{1/(1+epsilon)} polylog(T)). One algorithm is based on median of means and the other is based on truncation of payoffs. A matching lower bound of order \Omega(T^{1/(1+epsilon)}) is also shown. In particular, these regret bounds show the optimal scaling of \sqrt{T} for the case of finite variance. Medina and Yang (2016) studied the same problem, but only showed regret bounds that scale as T^{3/4} for the case of epsilon=1 (finite variance). For epsilon=1, regret of MENU (median of means) is O(d^{3/2} \sqrt{T}) while regret of TOFU (truncation) is O(d\sqrt{T}). The TOFU algorithm has a higher computational complexity.   The paper is well-written and the theoretical results are very interesting.   1) In Theorem 2, a condition on the noise is assumed while in Theorem 3, a condition on the payoff is assumed. Why the difference?   2) The first algorithm (based on median of means technique) requires playing the same action multiple times. This needs to be stated explicitly in the introduction.   Minor comments:  * Line 167: Typo: D_t \in R^d * Lines 210-216: Explain more. How do these modifications influence the corresponding regret bounds? * Line 227: Add more details. * Line 249: Typo: MENU -> TOFU * Notation u_{i,t} in Line 9 of Algorithm 2 is not defined.


Non-delusional Q-learning and Value Iteration
We identify a fundamental source of error in Q-learning and other forms of dynamic programming with function approximation. Delusional bias arises when the approximation architecture limits the class of expressible greedy policies. Since standard Q-updates make globally uncoordinated action choices with respect to the expressible policy class, inconsistent or even conflicting Q-value estimates can result, leading to pathological behaviour such as over/under-estimation, instability and even divergence. To solve this problem, we introduce a new notion of policy consistency and define a local backup process that ensures global consistency through the use of information sets-sets that record constraints on policies consistent with backed-up Q-values. We prove that both the model-based and model-free algorithms using this backup remove delusional bias, yielding the first known algorithms that guarantee optimal results under general conditions. These algorithms furthermore only require polynomially many information sets (from a potentially exponential support). Finally, we suggest other practical heuristics for value-iteration and Q-learning that attempt to reduce delusional bias.

UPDATE: Thank you to the authors for a well thought out response.  I definitely encourage you to compile the list of counterexamples that are or are not the result of delusion.  I think it will be a very important list for the theory community.  Also, thanks for the discounting example, that helped.  Good point on the Boyan example and ADP divergence and the expanded discussion.  Summary:   The paper defines the concept of ``delusional bias’’, the fact that the Bellman operator (or the sample-based version in Q-learning) combined with a function approximator, can backup values from a policy that exist in the realizable space of the approximator, essentially assuming that two actions that it cannot simultaneously take can be chosen at the same time. The authors propose a new operator that uses a partitioning of the parameter space into information sets.  The operator essentially maintains a large set of potential combinations of the partitions and q values for each of the sets, eventually choosing the maximum after convergence.  This requires significant (but sometimes polynomially bounded) computation, although in many cases an intractable witness procedure is needed for complex function approximators.  Review:  This is a potentially ground-breaking paper.  I have comments below on ways to improve the presentation of the paper and some minor mathematical notes but overall, I think the authors did an excellent job on a long standing foundational problem in RL.  The paper cites some important foundational work in the introduction but does not explicitly state which existing negative function approximation results are explained by delusional bias.  Is Baird’s counterexample a result of delusional bias and therefore solved by the new algorithm?  It would be better to explicitly mention which of the classical results can (and cannot) be explained by this phenomenon.  I do not see the direct link between delusional bias and the “discounting paradox” described in the paper.  The example given mixes a discounted MDP with an undiscounted one, and really those two problems are fundamentally different.  Is there an example where both discounts are < 1?  If so, is the conjecture of the paper that small discount factors essentially truncate the backups and therefore somehow enforce a partitioning?  I don’t understand why that would be the case.  Clearly changing the discount factor changes the solution, but it is not clear why the paradox is really related to delusional bias.    I thought perhaps the most important line in the paper was on line 235 where the authors note that their result is actually consistent with Petrik’s finding on the NP-Hardness of finding the best function approximator.  The authors essentially point out that Petrik was considering all possible action choices, and therefore maybe asking the wrong question.  Perhaps minimizing Bellman error isn’t all that important if one is considering actions that can’t actually be taken.  Perhaps that can be spelled out better here.  To make room for some of the explanations above, I suggest cutting down much of the speculation in the section 5.  These are interesting conjectures, but do not have any real empirical evidence behind them. If a particular example of one of these three approaches could be demonstrated here it would certainly improve the paper.  In section 3.2, it would be good to point out explicitly that there are different consequences of delusion in q-learning versus ADP.  Specifically, q-learning can actually diverge while I think the claim here is that ADP converges, but to something non-optimal given the approximator constraints, right?  That point should be made explicit.  Line 182 – the parentheses around the full sentence can be dropped.  It’s a full idea / sentence.


Using Large Ensembles of Control Variates for Variational Inference
Variational inference is increasingly being addressed with stochastic optimization. In this setting, the gradient's variance plays a crucial role in the optimization procedure, since high variance gradients lead to poor convergence. A popular approach used to reduce gradient's variance involves the use of control variates. Despite the good results obtained, control variates developed for variational inference are typically looked at in isolation. In this paper we clarify the large number of control variates that are available by giving a systematic view of how they are derived. We also present a Bayesian risk minimization framework in which the quality of a procedure for combining control variates is quantified by its effect on optimization convergence rates, which leads to a very simple combination rule. Results show that combining a large number of control variates this way significantly improves the convergence of inference over using the typical gradient estimators or a reduced number of control variates.

Thank you for the thoughtful response.  I have read the other reviews and the rebuttal, and after discussing the work I am electing to keep my score the same.   I am somewhat unsatisfied by the author response; for papers where gradient estimator efficiency (in terms of variance) is in service of the optimization problem, comparing ELBO traces by iteration can be very misleading.  If the machinery you introduce to efficiently use an ensemble of control variates is not very costly, then it should be measured or shown in your experiments.   My comments below weren't about optimal tuning, they were more about exploring/understanding the sensitivity of their method on the parameters they introduce. As I understand it, the main contribution of this work is the development of a regularized estimator of what is essentially a linear regression coefficient using conjugacy in exponential families.  I think that's a very clever way to approach the problem of combining control variates.  But, from the experiments, I did not get a sense the sensitivity of their estimator to these prior parameters was explored. How were they set? How do they interact with the number of samples chosen to construct this estimator? If this were a model for data, we would expect some sensitivity analysis to be done to see how much prior assumptions are factoring into posterior inferences. Analogously, I would hope to see some sensitivity analysis of the proposed method.  And from the rebuttal, I get the sense that these sensitivities will be left unexplored.    -------------------------------- The authors present a framework for constructing control variates for stochastic optimization in variational settings.  Looking at sub-parts of the variational inference objective, they define broad classes of ways to construct a control variate for each piece.  They then define a way to combine multiple control variates into a single, low-variance estimator in a sample-efficient way using a regret-minimizing estimator.  They compare the performance of these estimators on a logistic regression model with three different data sets at different learning rates.    - How does wall clock time compare for each estimator?  One issue with iteration-based comparisons is that some control variates (or their ensemble) may require more likelihood or gradient calculations.  In table 1 -- how much time did each iteration take for each set of estimators?  Another way to compare might be the number of likelihood evaluations required per iteration.   - Figure 3 is missing a legend --- do the colors correspond to the estimators defined in the legend in Figure 2?   - How does the running average parameter (gamma) from section 4.2 affect the variance of the estimator throughout optimization?  This is an interesting pivot point of computational tradeoff --- fitting the optimal C with many samples at an iteration of optimization would yield the lowest variance estimator.  How does that degrade when you (i) use fewer samples and rely on the InvWishart prior and (ii) use a running average of previous iterates?  At a certain point it becomes not worth computing a "better C" when optimizing an objective --- where is that point?  And how would you go about systematically finding it?  *Quality*: I think this paper presents an interesting idea, and points to a way to use a bunch of existing estimators to get the most out of a stochastic optimization procedure.  I think some of the experiments could have been conducted in a more convincing way, keeping in mind the goal of reducing the overall compute time of the optimization algorithm.  I liked how the experiments section compared a wide variety of estimators, and I would like to see them probe their own contribution a little more --- how their scheme for fitting C is affected by prior assumptions, how much compute time it takes per iteration, and how the running average scheme affects optimization.  *Clarity*: The paper is very clearly written and easy to follow.  *Originality*: The paper compares a bunch of existing estimators within a framework that is somewhat novel.  The regularization and running average of the control variate parameter is an interesting extension and a nice way to combine information from multiple estimators.  However the original aspect of their paper was empirically probed the least.  *Significance*: It is difficult to assess the potential impact of this method given the experiments presented. Ultimately, the utility of these gradient estimators for optimization hinge on wall clock time.


The Spectrum of the Fisher Information Matrix of a Single-Hidden-Layer Neural Network
An important factor contributing to the success of deep learning has been the remarkable ability to optimize large neural networks using simple first-order optimization algorithms like stochastic gradient descent. While the efficiency of such methods depends crucially on the local curvature of the loss surface, very little is actually known about how this geometry depends on network architecture and hyperparameters. In this work, we extend a recently-developed framework for studying spectra of nonlinear random matrices to characterize an important measure of curvature, namely the eigenvalues of the Fisher information matrix. We focus on a single-hidden-layer neural network with Gaussian data and weights and provide an exact expression for the spectrum in the limit of infinite width. We find that linear networks suffer worse conditioning than nonlinear networks and that nonlinear networks are generically non-degenerate. We also predict and demonstrate empirically that by adjusting the nonlinearity, the spectrum can be tuned so as to improve the efficiency of first-order optimization methods.

This submission is a theoretical contribution on the spectrum of Fisher information matrix of deep neural networks. Based on random matrix theory, the authors studied such a spectrum in a very simplified setting: a one-hidden layer feed-forward network, where both the inputs and all neuron network weights are i.i.d. Gaussian distributed, all layers have the same width, the activation function has zero Gaussian mean and finite Gaussian moments, and the cost function is the squared loss. The main results are summarized in theorem 1: in the limit case (very wide layer), the Stieltjes transform of the spectral density is in closed form. The author further showed consistency of the theoretic prediction of the spectrum density with numerical simulations using several activation functions (including the linear case), which shows perfect agreement.  The theoretical results present a solid advancement on the thread of applying random matrix theory to deep learning. It is meaningful in understanding the Fisher information matrix of neural networks.  While I am convinced that this paper should be accepted by NIPS as a nice theoretical contribution, I am not familiar with random matrix theory as indicated by the confidence score.  The main comments I have is regarding the very constrained assumptions and the writing.  It is true that it is difficult to get elegant results as theorem 1 for deep learning without making simplified assumptions. However, in the end, the authors did not make any attempt on generalizing the results to more general settings, or providing relevant discussions, or making numerical simulations to show the (in)consistency of the spectrum when those assumptions fail. Therefore the results have limited values and may not have a large impact.  As a relevant question, is it possible to extend the results to the case, where dim(X) is fixed while the size of the hidden layer goes to infinity? If this case can be provided in section 3, the contribution will be much more systematical.  Did you make any experimental study on the case, where the constraints are slightly violated, while the prediction still roughly holds? If such results can be presented as figures in section 3/4, it will make the contribution more stereoscopic.  Regarding the writing, the overall technical and English quality is good. The introduction is a bit too long and deviating from the main contributions. The authors raised some big questions, such as "under which conditions first-order methods will work well". These questions are not really answered at the end. The authors are therefore suggested to streamline section 1 and "cut to" the topic. It needs less words to motivate that the spectrum of the Fisher/Hessian is important. Instead, use the saved space to provide more discussion in the end on generalizing the results. Please see the following minor comments for more details.  Minor comments:  Through out the paper there are some extra commas just before the equations. For example, L80 "can be written as , H=H^0+H^1".  Please use parentheses or brackets to denote the expectations as E_X(...). The current notation looks a bit weird.  In equation (1) there is an expectation with respect to the random variable Y. In the following equations it is gone. Please check carefully and be consistent.  Use a different symbol such as ":=" for definitions.  L101 limiting spectral density can be accompanied with an equation.  Please try to keep the paper self-contained and refer a bit less to the SM.  equation (29) can be moved to an earlier place to introduce f_opt  Figure(2,b): please make clear which curves are predictions and which curves are simulations  A relevant (unpublished) work is "Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach. Ryo Karakida, Shotaro Akaho, Shun-ichi Amari. 2018.". As this work is unpublished, it is only for the authors' interest to have a look.  -After rebuttal-  Thank you for your rebuttal. Please try include related discussions on generalizations (e.g. in the last section).


Robot Learning in Homes: Improving Generalization and Reducing Dataset Bias
Data-driven approaches to solving robotic tasks have gained a lot of traction in recent years. However, most existing policies are trained on large-scale datasets collected in curated lab settings. If we aim to deploy these models in unstructured visual environments like people's homes, they will be unable to cope with the mismatch in data distribution. In such light, we present the first systematic effort in collecting a large dataset for robotic grasping in homes. First, to scale and parallelize data collection, we built a low cost mobile manipulator assembled for under 3K USD. Second, data collected using low cost robots suffer from noisy labels due to imperfect execution and calibration errors. To handle this, we develop a framework which factors out the noise as a latent variable. Our model is trained on 28K grasps collected in several houses under an array of different environmental conditions. We evaluate our models by physically executing grasps on a collection of novel objects in multiple unseen homes. The models trained with our home dataset showed a marked improvement of 43.7% over a baseline model trained with data collected in lab. Our architecture which explicitly models the latent noise in the dataset also performed 10% better than one that did not factor out the noise. We hope this effort inspires the robotics community to look outside the lab and embrace learning based approaches to handle inaccurate cheap robots.

In this paper a new dataset for robot grasping task is proposed. Compared to grasping data collected in a lab environment, the authors propose to collect the data from real world environments (homes). To collect data in the wild, the authors propose to use cheap robots (measured by the $ cost) with low DoF. In order to compensate the noisy behavior of the less calibrated robots, the authors model the noise as a latent variable and jointly learn it with the grasping task. Results show that the combination of the aforementioned ideas result in a robot grasping model that can work well on both lab environments, and new real world environment.  Pros: 1. I really like the motivation of the paper as it is, to the best of my knowledge, the first one to emphasize the following two very important perspectives in robot learning: 1. how do we develop and use cheap robots with less calibrated mechanical details (sensors, actuators, hardware wear-out etc.) 2. how we can extend robots to real world, so we can learn much richer representations. The paper does a good job from this perspective and opens up potentially a new research direction. 2. Results are competitive and clearly suggest the efficacy and advantage in generalization when learning in the wild. 3. As the major technical contribution (though the specific tech has been developed and used in [18]), the noise modeling network shows promising results and worth further discussions.  Cons: As probably the first attempt, the paper is still very preliminary. For cheap robots, it is currently measured by the cost. It is okay but it might be even more helpful to set up a standard of system identification for cheap robots, i.e. what are the possible factors that make a robot cheap? It is okay to just model the noise as a simple latent variable in this vision based grasping task, however, considering harder control tasks, or with a RL approach, we might need to have a better idea on what could possibly go wrong. One interesting thing to try is to see how cheap robots perform after a long period of working time without human recalibration, and see if the robust learning algorithm can handle that to some extent. Of course this is beyond the scope of this paper.  For real world environments, what are the major difference between different homes? Is it just different background image for grasping (floor/carpet?). Do you deploy the robots on places of different physical properties as well, e.g. on a table, on the bed, or in a bathtub?  The thing really worries me is the experimental evaluations. I have the feeling that the gap between Lab vs Home can be easily reduced by simple data augmentation. It is mentioned in the paper that in the lab environments, people usually only care about variations of objects for grasping. And I think the major difference of doing this at real homes (despite the issue of cheap robots etc), is adding a new data augmentation dimension on the grasping background. One experiment I could think of is, under lab environment, using different carpet on the workstation as the "background" of the grasping task. I would imagine this will completely match the performance of home collected data.  From the algorithm perspective, since it is basically a vision task (no need to worry about, say rl policy adaptation) simple domain adaptation could help reduce the gap too. I might be careless but I'm wondering what's the justification of no finetuning experiment for the experimental evaluation (learning on lab data, and finetune on the real world data for a small number of epochs).  Overall, this is an interesting paper. We are far from solving the sim2real transfer problem, but it is a good to think about "lab2real" transfer problem and this paper is, though not perfect, a good initial attempt.


Tree-to-tree Neural Networks for Program Translation
Program translation is an important tool to migrate legacy code in one language into an ecosystem built in a different language. In this work, we are the first to employ deep neural networks toward tackling this problem. We observe that program translation is a modular procedure, in which a sub-tree of the source tree is translated into the corresponding target sub-tree at each step. To capture this intuition, we design a tree-to-tree neural network to translate a source tree into a target one. Meanwhile, we develop an attention mechanism for the tree-to-tree model, so that when the decoder expands one non-terminal in the target tree, the attention mechanism locates the corresponding sub-tree in the source tree to guide the expansion of the decoder. We evaluate the program translation capability of our tree-to-tree model against several state-of-the-art approaches. Compared against other neural translation models, we observe that our approach is consistently better than the baselines with a margin of up to 15 points. Further, our approach can improve the previous state-of-the-art program translation approaches by a margin of 20 points on the translation of real-world projects.

Summary  This paper proposes a neural network architecture for tree to tree transduction, with application to program translation. Binarized trees are encoded with a TreeLSTM, and a tree-structured decoder generates binary trees breadth-first, using an attention mechanism over input tree nodes, and attention feeding within the tree decoder. On synthetic and real-world program translation tasks, and across multiple datasets, the proposed model obtains substantially stronger performance than baseline sequential neural models and previous work on automatic program translation.    Quality  The paper proposes a general framework for neural tree to tree transduction, and the model obtains high performance on program translation tasks.       Clarity  The model and experimental setup are define clearly.   If the terminal and non-terminal vocabularies are disjoint, is it necessary to generate end-of-sentence tokens explicitly, as they should always follow terminals?   Originality  Previous neural network architectures have been proposed for code generation (as cited in the paper), but have not been applied to program translation. The novelty of the proposed model lies in having both a tree-structured encoder and decoder. The proposed decoder generates trees breadth-first rather than depth-first, as it typical in many other architectures.   To show whether the tree decoder architecture proposed here has advantages over previously proposed tree decoders, experimental comparison with alternative architectures will be helpful.      Significance  Tree to tree transduction is an important problem, and this paper is the first to propose successful neural models that explicitly models the tree structure in both the input and output. It is further shown that the model obtains a strong improvement over previous models in the practical task of program translation. While there is still a lot of work to do done in this area, this paper makes a strong contribution and I would like to see it accepted.


Large-Scale Stochastic Sampling from the Probability Simplex
Stochastic gradient Markov chain Monte Carlo (SGMCMC) has become a popular method for scalable Bayesian inference. These methods are based on sampling a discrete-time approximation to a continuous time process, such as the Langevin diffusion. When applied to distributions defined on a constrained space the timediscretization error can dominate when we are near the boundary of the space. We demonstrate that because of this, current SGMCMC methods for the simplex struggle with sparse simplex spaces; when many of the components are close to zero. Unfortunately, many popular large-scale Bayesian models, such as network or topic models, require inference on sparse simplex spaces. To avoid the biases caused by this discretization error, we propose the stochastic Cox-Ingersoll-Ross process (SCIR), which removes all discretization error and we prove that samples from the SCIR process are asymptotically unbiased. We discuss how this idea can be extended to target other constrained spaces. Use of the SCIR process within a SGMCMC algorithm is shown to give substantially better performance for a topic model and a Dirichlet process mixture model than existing SGMCMC approaches.
contain simplex parameters. For example, latent Dirichlet allocation (LDA) is defined by a set of topic-specific distributions on words and document-specific distributions on topics. Probabilistic network models often define a link probability between nodes. More generally, mixture and mixed membership models have simplex-constrained mixture weights; even the hidden Markov model can be cast in this framework with simplex-constrained transition distributions. As models become large-scale, these vectors ω often become sparse-i.e., many ω j are close to zero-pushing them to the boundaries of the simplex. All the models mentioned have this tendency. For example in network data, nodes often have relatively few links compared to the size of the network, e.g., the number of friends the average social network user has will be small compared with the size of the whole social network. In these cases the problem of sampling from the simplex space becomes even harder; since many values will be very close to the boundary of the space. Patterson and Teh (2013) develop an improved SGMCMC method for sampling from the probability simplex: stochastic gradient Riemannian Langevin dynamics (SGRLD). The improvements achieved are through an astute transformation of the simplex parameters, as well as developing a Riemannian (see Girolami and Calderhead, 2011) variant of SGMCMC. This method achieved state-of-the-art results on an LDA model. However, we show that despite the improvements over standard SGMCMC, the discretization error of SGRLD still causes problems on the simplex. In particular, it leads to asymptotic biases which dominate at the boundary of the space and causes significant inaccuracy.
To counteract this, we design an SGMCMC method based on the Cox-Ingersoll-Ross (CIR) process. The resulting process, which we refer to as the stochastic CIR process (SCIR), has no discretization error. This process can be used to simulate from gamma random variables directly, which can then be moved into the simplex space using a well known transformation. The CIR process has a lot of nice properties. One is that the transition equation is known exactly, which is what allows us to simulate from the process without discretization error. We are also able to characterize important theoretical properties of the SCIR algorithm, such as the non-asymptotic moment generating function, and thus its mean and variance. We discuss how these ideas can be used to simulate efficiently from other constrained spaces, such as (0, ∞).
We demonstrate the impact of this SCIR method on a broad class of models. Included in these experiments is the development of a scalable sampler for Dirichlet processes, based on the slice sampler of Walker (2007); Papaspiliopoulos (2008); Kalli et al. (2011). To our knowledge the application of SGMCMC methods to Bayesian nonparametric models has not been explored. All proofs in this article are relegated to the Supplementary Material. All code for the experiments is available online 1 , and full details of hyperparameter and tuning constant choices has been detailed in the Supplementary Material.
1 Stochastic Gradient MCMC on the Probability Simplex

For the valuable problem of large-scale and sparse stochastic inference on simplex, the authors proposed a novel Stochastic gradient Markov chain Monte Carlo (SGMCMC) method, which is based on the Cox-Ingersoll-Ross (CIR) process. Compared with the commonly-used Langevin diffusion within the SGMCMC community, the CIR process (i) is closely related to the flexible Gamma distribution, and therefore more suitable for inferring a Dirichlet distribution on simplex, since a Dirichlet distribution is just the normalization of Gamma distributions; (ii) CIR has no discretization error, which is shown to be a clear advantage over the Langevin diffusion on simplex inference.  Besides, the author proved that the proposed SCIR method is asymptotically unbiased, and has improved performance over other SGMCMC method on sparse simplex problem via two experiments, namely inferring a LDA on a dataset of scraped Wikipedia documents and inferring a Bayesian nonparametric mixture model on Microsoft user dataset.  I think the quality is good; the presentation is clear; as far as I know the proposed technique is original and of great significance. Therefore I vote for acceptance. However, the experiments are okay, but not strong. More experiments would be better.   I have read the authors' response, I would suggest additional experiments, such as exploiting the presented SCIR for more complicated non-conjugate models like the PGBN [1]; (2) to make a direct comparison with the TLASGR-MCMC developed in [2], which is also developed for inference on simplex and is shown better than the SGRLD. Other potential reference methods include the ones in [3,4].   [1] The Poisson gamma belief network. NIPS 2015. [2] Deep latent Dirichlet allocation with topic-layer-adaptive stochastic gradient Riemannian MCMC. ICML 2017. [3] "A Complete Recipe for Stochastic Gradient MCMC. NIPS 2015" [4] "Stochastic gradient geodesic MCMC methods. NIPS 2016"


Learning Task Specifications from Demonstrations
Real-world applications often naturally decompose into several sub-tasks. In many settings (e.g., robotics) demonstrations provide a natural way to specify the sub-tasks. However, most methods for learning from demonstrations either do not provide guarantees that the artifacts learned for the sub-tasks can be safely recombined or limit the types of composition available. Motivated by this deficit, we consider the problem of inferring Boolean non-Markovian rewards (also known as logical trace properties or specifications) from demonstrations provided by an agent operating in an uncertain, stochastic environment. Crucially, specifications admit well-defined composition rules that are typically easy to interpret. In this paper, we formulate the specification inference task as a maximum a posteriori (MAP) probability inference problem, apply the principle of maximum entropy to derive an analytic demonstration likelihood model and give an efficient approach to search for the most likely specification in a large candidate pool of specifications. In our experiments, we demonstrate how learning specifications can help avoid common problems that often arise due to ad-hoc reward composition.

** update ** I increase the score I attribute to this paper since the authors' response to the various problems raised by the reviewers seems to me very satisfactory. However the authors promise several improvements and my score cannot be based solely on promises, so this score only increases by one. Finally, the practical motivations of this little explored field are extremely justified, so I also encourage authors to apply their work to more complex problems, which should lead to interesting results.  ** summary ** This paper sets up an algorithm capable from demonstrations (sequence of states and actions)  and from a set of specifications (set of trajectories), to infer the specifications  governing the actions of the agent and the visited states. These specifications can be seen as non-Markovian reward functions.  Thus, this work is related to inverse reinforcement learning (IRL)  which aims to infer the reward function of an agent  by observing these successive states and actions. By defining the probability of a trajectory knowing a specification  (using the maximum entropy principle) the development leads to a posterior distribution. Two algorithms result from this and allow to test the approach  on the system presented in introduction (motivating the paper).  ** quality ** Although its introduction is interesting, this paper is quite hard to follow,  especially from page 4.  The steps should be better justified or motivated by intuition or pedagogical explanations  of the ongoing development. Also some proofs only appears in the supplementary materials and not in the paper,  making the paper less self-contained. At least sketches of proof should be provided to let the reader study the main arguments.  ** clarity ** As mentioned above, clarity is not the strength of this paper.  This can also be illustrated by the more or less in-depth comments that follow.   page 3:  < Sec > Section  The Markov property is already present in Definition 2.1.  < no \phi =^{def} 1 - \phi(\xi) > no \phi(\xi) =^{def} 1 - \phi(\xi)  page 4:  In Equation 2, it seems that we multiply a probability of \xi  (trajectory probability given by the transition functions)  with another probability of \xi (given by the exponential function).  Some confusion in the notations X and \xi.  page 6: < the following simple algorithm. > the following simple algorithm (Algorithm 2).  page 8: < queries,\phi^{hat} > queries, \phi^{hat}  < assignments.\footnote{...} > assignments\footnote{...}.  The conclusion should be a Section.  ** originality ** To the best of my knowledge this paper is the first research  on inferring specifications from demonstrations in the MDP framework. The authors have done a good bibliographical work to position their study  in the relevant literature. I suggest authors read  (Drougard "Exploiting Imprecise Information Sources in Sequential Decision Making Problems  under Uncertainty." 2015) which adapts classical MDP models to Qualitative Possibility Theory. This theory is very close to logic  (and therefore could easily integrate the specifications framework)  while retaining properties similar to Probability Theory  (which allows planning with the same tools).  ** significance ** Although the idea and the algorithms developed are really interesting,  I doubt that the theoretical and applied contributions are worthy of NIPS.  I remain cautious however because the lack of clarity, the absence of sketch of proof  and the short time to review prevents me from checking developments in details.


Connectionist Temporal Classification with Maximum Entropy Regularization
Connectionist Temporal Classification (CTC) is an objective function for end-toend sequence learning, which adopts dynamic programming algorithms to directly learn the mapping between sequences. CTC has shown promising results in many sequence learning applications including speech recognition and scene text recognition. However, CTC tends to produce highly peaky and overconfident distributions, which is a symptom of overfitting. To remedy this, we propose a regularization method based on maximum conditional entropy which penalizes peaky distributions and encourages exploration. We also introduce an entropybased pruning method to dramatically reduce the number of CTC feasible paths by ruling out unreasonable alignments. Experiments on scene text recognition show that our proposed methods consistently improve over the CTC baseline without the need to adjust training settings. Code has been made publicly available at: https://github.com/liuhu-bigeye/enctc.crnn.
• Harm the training process. The positive feedback-like error signal means CTC lacks exploration and is prone to fall into worse local minima.
• Output overconfident paths. CTC tends to concentrate all its output distribution over one specific path. On the one hand, it is not suitable to handle the situation where the segmentation boundary is ambiguous, e.g. adjacent syllables in speech recognition and action switching in continuous sign language recognition. On the other hand, the low-entropy output distribution is a symptom of overfitting [31], leading to low prediction accuracy.
• Output paths with peaky distribution. The peaky distribution is not desirable for sequence segmentation tasks when the model needs to densely predict labels for each time-step. Even if only the temporal order of labels is required, learning the correct segmentation will improve model generalization and interpretation abilities.
Motivated by the maximum entropy principle [13], we propose a maximum conditional entropy based regularization for CTC (EnCTC). We consider the conditional distribution of feasible paths given the input sequence and label sequence. EnCTC prevents the entropy of feasible paths from decreasing too fast, thus alleviating the impact of CTC's positive feedback-like error signal and encouraging exploration during training. It also prevents the probability from being dominated by a single path and solves the peaky distribution problem. It mitigates overfitting and is more suitable for depicting ambiguous segmentation boundaries.
We further consider another solution to the problem of finding the reasonable feasible path-limit the size of the feasible set. We observe that in many sequence learning tasks, the intervals between the adjacent elements in the input sequence are almost the same, e.g. characters in text recognition, syllables in speech recognition and gestures in continuous sign language recognition. We summarize this phenomenon as the equal spacing prior. Therefore, we propose an algorithm to limit the size of the CTC feasible set by eliminating these unreasonable paths that seriously violate the equal spacing prior (EsCTC). Moreover, the equal spacing prior can give theoretical explanations from the perspective of maximum entropy, which indicates that equal spacing is the best prior without any additional subjective assumptions.
The main contributions of this paper can be summarized as: (1) We propose a maximum conditional entropy regularization for CTC (EnCTC), which encourages exploration for CTC training and prevents peaky output distributions. (2) We derive from equal spacing prior a pruning algorithm (EsCTC) to effectively limit the size of CTC feasible set and give theoretical explanations from the perspective of maximum entropy. (3) We provide polynomial-time dynamic programming algorithms for calculating EnCTC, EsCTC and their combination (EnEsCTC). (4) We validate the proposed methods on scene text recognition tasks and show that these methods are able to improve the baseline model without changing training settings.

This work presents a method for end-to-end sequence learning, and more specifically in the framework of Connectionist Temporal Classification (CTC). The paper has two main contributions: - The first is a regularization of the training of the CTC objective in order to reduce the over-confidence of the model. In order to do that, the authors propose a method based on conditional entropy. More specifically, the proposed regularization would encourages the model to explore paths that are close to the dominant one.  - The second contribution is a pruning method in order to reduce the space of possible alignments. In order to do so, they suppose that the consecutive elements of a sequence have equal spacing. The authors prove that this situation corresponds to the maximum of entropy over the possible segmentations of the sequence.  The paper mainly advocates for the combination of the two contributions. The authors also provide details about dynamic programming to optimize the proposed objective. They finally compare their methods to the state-of-the-art approaches on different datasets.   The contributions of the paper are well motivated, and their effect seems to be significant according to the experimental results. The two proposed regularization are also well explained intuitively, and through supporting theory. In addition, the effect of the regularization is well analyzed in the experiments section. For all these reasons I vote for acceptance of this paper.   Minor comment: There is a typo in equation (3) in the partial derivative notation.


Learning Others' Intentional Models in Multi-Agent Settings Using Interactive POMDPs
Interactive partially observable Markov decision processes (I-POMDPs) provide a principled framework for planning and acting in a partially observable, stochastic and multi-agent environment. It extends POMDPs to multi-agent settings by including models of other agents in the state space and forming a hierarchical belief structure. In order to predict other agents' actions using I-POMDPs, we propose an approach that effectively uses Bayesian inference and sequential Monte Carlo sampling to learn others' intentional models which ascribe to them beliefs, preferences and rationality in action selection. Empirical results show that our algorithm accurately learns models of the other agent and has superior performance than methods that use subintentional models. Our approach serves as a generalized Bayesian learning algorithm that learns other agents' beliefs, strategy levels, and transition, observation and reward functions. It also effectively mitigates the belief space complexity due to the nested belief hierarchy.

The paper describes a sampling method for learning agent behaviors in interactive POMDPs (I-POMDPs). In general, I-POMDPs are a multi-agent POMDP model which, in addition to a belief about the environment state, the belief space includes nested recursive beliefs about the other agents' models. I-POMDP solutions, including the one proposed in the paper, largely approximate using a finite depth with either intentional models of others (e.g., their nested beliefs, state transitions, optimality criterion, etc.) or subintentional models of others (e.g., essentially "summaries of behavior" such as fictitious play). The proposed approach uses samples of the other agent at a particular depth to compute its values and policy. Related work on an interactive particle filter assumed the full frame was known (b, S, A, Omega, T, R, OC). The main unique characteristic of the proposed approach is to relax this assumption: only S, A, Omega are known, with beliefs, T, O, and R learned from n samples. The algorithm essentially proceeds by sampling from these distributions instead to generate beliefs and thus compute value and policy. Experiments were conducted on the multi-agent tiger and UAV domains, comparing a fictitious play I-POMDP baseline versus 0-, 1-, 2-, 3-level intentional I-POMDPs. In summary, results show that the intentional outperforms the baseline.  The paper does a great job of explaining the I-POMDP and the issues surrounding the nested modeling of other agents' models. While the algorithm suffers from the necessary notational overhead, each step is well explained in the surrounding companion paragraphs. As there are no theoretical results, experiments attempt to serve as marginally convincing evidence that the approach works.  Here are some comments and questions:  1. The experiments require up to 1000 or 2000 samples to obtain solid performance to accurately model T, O, R, etc. These problems are quite small, with simple transition and observation functions to learn. Given this fact, and the long run times for basic 2-level nesting, what is the main motivation for the usefulness of this approach? Why not use a subintentional model for any larger I-POMDPs? The number of samples in this approach will need to grow rapidly to astronomical sizes, with likely large computation times. Perhaps a stronger motivation to contrast this with the much less sample-dependent---but knowingly lower performant---subintentional models.  2. Experiments show interesting behavior regarding the performance, number of samples, and even KL-divergence between learned model parameters and ground truth. However, in application, how do we know how many samples are sufficient to accurately model the problem? Are there any techniques for determining a stopping point, especially in larger I-POMDPs when there are many more samples?  Overall, the idea clearly is how one would solve the problem of not having T, O, and R for other agents a priori: Monte Carlo sample. The paper partially delivers on the promise of such a scalable algorithm, for which simple subintentional models like fictitious play already serve this role well. That said, the idea is good and broadly moving in this model-based direction is well-motivated.  Some minor comments: - Line 208: "...with an subintentional..." -> "...with a subintentional...". - Line 219: "Then we fixe the..." -> "Then we fixed the..." - There are a lot of figures, many of which could be saved for the supplement in favor of some theoretical result, perhaps regarding number of samples (as above), error bound, etc. - References have a few typographic errors. "POMDP" needs to be capitalized, "Markov", "Bayesian", etc.


Fully Neural Network Based Speech Recognition on Mobile and Embedded Devices
Real-time automatic speech recognition (ASR) on mobile and embedded devices has been of great interests for many years. We present real-time speech recognition on smartphones or embedded systems by employing recurrent neural network (RNN) based acoustic models, RNN based language models, and beam-search decoding. The acoustic model is end-to-end trained with connectionist temporal classification (CTC) loss. The RNN implementation on embedded devices can suffer from excessive DRAM accesses because the parameter size of a neural network usually exceeds that of the cache memory and the parameters are used only once for each time step. To remedy this problem, we employ a multi-time step parallelization approach that computes multiple output samples at a time with the parameters fetched from the DRAM. Since the number of DRAM accesses can be reduced in proportion to the number of parallelization steps, we can achieve a high processing speed. However, conventional RNNs, such as long short-term memory (LSTM) or gated recurrent unit (GRU), do not permit multi-time step parallelization. We construct an acoustic model by combining simple recurrent units (SRUs) and depth-wise 1-dimensional convolution layers for multi-time step parallelization. Both the character and word piece models are developed for acoustic modeling, and the corresponding RNN based language models are used for beam search decoding. We achieve a competitive WER for WSJ corpus using the entire model size of around 15MB and achieve real-time speed using only a single core ARM without GPU or special hardware.

Summary: This paper proposes an acoustic model by combining simple recurrent units (SRUs) and simple 1-D convolution layers for multi-time step parallelization, which significantly improves the speed by overcoming the step-by-step constraint of RNNs. The proposed AM performs substantially better than SRUs alone and even better than LSTMs with the same number of parameters. It is combined with an RNN LM as a fully neural network-based speech recognizer. Word pieces are investigated as the model output targets in addition to graphemes, for further speedup in decoding.  Quality: Strengths: - The idea of SRU + 1-D conv is simple but the improvement is impressive. - The multi-time step parallel processing for AM is shown to achieve nice speedup on a single core (6x with 8 parallel steps). - RNN LM with multi-stream processing is shown to be efficient enough to be used in beam search decoding, where it is typically used as second-pass rescoring due to efficiency constraint. - The paper shows a final model with total size 30MB is able to run in real time with a single core ARM.  Weaknesses: - Do all the baseline LSTM models in the paper use recurrent projection units (Sak et al. 2014, Long Short-Term Memory Recurrent Neural Network Architectures for Large Scale Acoustic Modeling)? It should be easy to implement and it is effective to improve performance with the same number of parameters. - LSTM + 1-D conv should be used as the baseline in Table 3 and 5 since it is shown to be better than LSTM alone in Table 1. - It is kind of surprising that SRU + 1-D conv achieves better accuracy than LSTM + 1-D conv. It is worth discussing the reason / hypothesis in the paper. - In Table 6, it would be useful to also measure the computation time of LSTM AM for comparison. - The proposed system should be compared to a standard LSTM AM + WFST decoder graph with a (pruned) n-gram LM, which is shown to be efficient for embedded speech recognition in previous works (e.g. McGraw et al. 2016, Personalized speech recognition on mobile devices), especially now that the RNN LM is the computation bottleneck of the entire system as is shown in Figure 2. In fact, the speed of RNN LM could be further improved by 8-bit quantization, which typically does not degrade the WER.  Clarity: In general, the presentation in the paper is clear and well-organized, except for the following questions: - In Table 2, what is the system performance with HCLM? - Does the system in Table 6 use characters or word pieces? Both should be shown for comparison. - How much is the WER different between different beam width in Figure 2? Does using half-precision in LM hurt WER? - It would be clearer to mention in Table 2 that 3.60 (for Deep Speech 2) is WER (instead of CER).  Originality: The problem is well-motivated: multi-time step parallel processing for AM would be very useful for embedded system recognition, but SRU alone does not work. It is novel to find that simply combining 1-D conv with SRU substantially improves the quality of SRU, leading to better performance than LSTM. Other tricks and analyses in the paper to improve speed and accuracy are also useful.   Significance: The results are very encouraging in terms of both accuracy and speed. Both the idea and the experiments would be of interest to the field and motivate future work.  It is a good submission. I recommend an accept, although the issues mentioned above should be addressed.


Efficient Convex Completion of Coupled Tensors using Coupled Nuclear Norms
Coupled norms have emerged as a convex method to solve coupled tensor completion. A limitation with coupled norms is that they only induce low-rankness using the multilinear rank of coupled tensors. In this paper, we introduce a new set of coupled norms known as coupled nuclear norms by constraining the CP rank of coupled tensors. We propose new coupled completion models using the coupled nuclear norms as regularizers, which can be optimized using computationally efficient optimization methods. We derive excess risk bounds for proposed coupled completion models and show that proposed norms lead to better performance. Through simulation and real-data experiments, we demonstrate that proposed norms achieve better performance for coupled completion compared to existing coupled norms.

The paper studies the problem of coupled tensor completion, which is originally motivated from multi-task settings. The proposed method, coupled nuclear norm, extends the previously proposed tensor nuclear norm to the setting where two tensors coupled in some mode-k. Such an extension is easy to think of and does not need a lot of effort. The authors show that, using the proposed coupled nuclear norm, the risk bound is reduced, saying from O(n^0.5(K-1)) to O(n^0.5 log(n)^K); this is a distinct improvement (but the criterion used to measure the risk is questionable).  Comments:  1. The proposed method as well as analysis techniques are some kind of mixture of existing papers. Thus, the innovation level of this paper is not high. But this is okay to me, as long as the authors can make a solid contribution to the community.   2. The major concern is about the clarify, in particular the confusing notations used throughout the paper. a) As the tensors are coupled in only one mode, it is necessary to use a superscript a to specify the coupled mode. Also, it is not a good idea to begin the method with some complicated notation with subscripts (b,c,d). I think it is beneficial to keep the notations as simple as possible.  b) The proposed coupled nuclear norm defined in Equation (1) seems a set, not a norm. In particular, what are \mathcal{X} and \mathcal{Y}? The are similar problems with Equation (2).  c) On the theoretical analysis, it is really wired to split the observed entries into a training set and a test set. Usually, the test data should be the missing entries.   d) The Rademacher complexity measures only the richness of the proposed norm. However, in general, more freedoms do not necessarily lead to good recovery performance. It will be more convincing if the authors can prove that the proposed coupled nuclear norm can reduce the sampling complexity required to restore the missing entries.


Optimization over Continuous and Multi-dimensional Decisions with Observational Data
We consider the optimization of an uncertain objective over continuous and multidimensional decision spaces in problems in which we are only provided with observational data. We propose a novel algorithmic framework that is tractable, asymptotically consistent, and superior to comparable methods on example problems. Our approach leverages predictive machine learning methods and incorporates information on the uncertainty of the predicted outcomes for the purpose of prescribing decisions. We demonstrate the efficacy of our method on examples involving both synthetic and real data sets.

This paper explores off-policy learning. It proposes an optimization framework that incorporates information on the uncertainty of the predicted outcomes to learn better policies. It is based on the fact that minimizing only the cost function would result in a low bias but high variance estimator. Their idea is to add the estimator’s variance and bias as penalty terms to the objective function to trade-off between learning estimators with small cost and those with high variance/bias.   However, it is not clear whether  the findings are novel, as Swaminathan and Joachims [21] also consider incorporating the variance of response into the objective function to attain models that exhibit lower variance. While the authors also mention that Swaminathan and Joachims [21] assume that the underlying mechanism of action selection (i.e., propensities) is known, their proposed Counterfactual Risk Minimization framework is independent of how the response variable (u in [21]) is defined (i.e., whether it requires the propensity score) and therefore does not rely on this assumption. The authors needs to better explain how their work differs from that of [21].  Unfortunately, many parts of the paper are quite hard to follow. The authors should clarify the following points: L22: Is the cost function known/given? e.g., Is it c = (y - \hat{y})^2, where y is the true outcome and \hat{y} is the predicted outcome by a regression fit? Or is it determined arbitrarily? It is very important to define the cost clearly, as the rest of the equations are heavily dependent on it … The experiments do not shed light on what the cost function is either. L117: “The first step …” → Are there any next steps? My understanding is that learning the predictive model is intertwined with optimizing for z since there is nothing else to learn other than c (and consequently w due to the way it’s defined in L149.5) in equation (2). L169-170: If we could find a model that predicts the counterfactual outcomes accurately, we could use that to select the best z; the problem is that, due to sample selection bias (i.e., dependence of z on x), we cannot be sure that the regression fit would give accurate estimates of the counterfactuals. Therefore, it is wrong to base the model selection on this. Unless the above mentioned points are clarified, it is hard to judge the quality of this work.   Minor points: L40: parameters are “learned”; objective function is “optimized”. L120-123: Unclear nomenclature: what is the “response variable”? What is “prediction of the algorithm”? and what is the “training response”? L128: “reduces bias of the predictions” → predictions of what? outcome or cost? L142.5: Did you mean B(x,z) := \sum_{i} || \hat{\mu(x,z)} - \mu(x,z) ||_2 ? L160-161: Isn’t this the role of the predictive model? Isn’t this already done?  Typos: L124: satisfies → to satisfy L211: probably → probable  ==== In light of the authors’ discussion in the rebuttal, I am convinced that the contribution of this paper is novel (compared to [21]).   Regarding my L169-170 comment -- although the authors indicated that they get good empirical results: I am still not convinced that this way of tuning parameters is valid. That is, tuning hyperparameters based on predicted counterfactuals, especially when the sample selection bias is not accounted for in prediction procedure. I expect this method to fail in case of a high selection bias in data and suspect that this was not the case in the paper’s experiments.  In general, it was difficult to understand some main parts of the submission. I believe the paper would benefit from being modified for more clarity and better flow.  Given the author’s comments, I am willing to revise my score to 5.