Data-Efficient Reinforcement Learning in Continuous State-Action Gaussian-POMDPs
We present a data-efficient reinforcement learning method for continuous stateaction systems under significant observation noise. Data-efficient solutions under small noise exist, such as PILCO which learns the cartpole swing-up task in 30s. PILCO evaluates policies by planning state-trajectories using a dynamics model. However, PILCO applies policies to the observed state, therefore planning in observation space. We extend PILCO with filtering to instead plan in belief space, consistent with partially observable Markov decisions process (POMDP) planning. This enables data-efficient learning under significant observation noise, outperforming more naive methods such as post-hoc application of a filter to policies optimised by the original (unfiltered) PILCO algorithm. We test our method on the cartpole swing-up task, which involves nonlinear dynamics and requires nonlinear control.

SUMMARY
In this paper, the authors propose to learn and plan with a model that represents uncertainty in a specific case of partial observability - that of noisy observations. Their method the analogue of a known model based method - PILCO - for this noisy observation case. It handles the noise by assuming a joint Gaussian distribution over the latent states and observations at each time step when making predictions (at execution time, observations are known and a filtered latent state is used). On the kind of dynamical system that these method are commonly evaluated on - the cart-pole swing-up in this case - the proposed methods seems to be better than relevant baselines at dealing with noise.
TECHNICAL QUALITY
The proposed algorithm is supported by a derivation of the update equation and experimental validation on the cart-pole swing-up task. The derivation seems technically solid. I just wondered why in (5) the covariance between M_t|t-1 and Z_t is 0, since in Fig. 2 it seems that Z_t is a noisy version of B_t|t-1 of which M_t|t-1 is the mean.
Although the experiments only consider a single system, the results seem quite convincing. The empirical loss per episode is significantly lower, and predictions of the cost per time step are much better than for the compared methods. I just wondered what is meant by the "empirical loss distribution", and whether the 100 independent runs that the error bars are based on are 100 evaluations of the same learned controller; or 100 independently trained controllers. In Figs. 7 and 8 I would be interested to see the n=0 case (or, if that causes numerical problems, n=0.01 or similar).
In terms of related work, the authors mainly restrict themselves to discussing Bayesian model learning and Bayesian RL techniques. It might be interesting to also discuss other styles of policy search methods for the partially observable setting.
It might be insightful to describe what about the current method limits us to the noisy observation case. If we observed positions but not velocities, coulnd't we filter that, too?
NOVELTY
I haven't seen earlier extensions of PILCO-type methods to the partially observable case. So I would consider the method quite novel.
SIGNIFICANCE AND RELEVANCE
Noisy dynamical systems occur in many (potential) application domain of reinforcement learning (robotics, smart grid, autonomous driving), so, this is a relevant research topic for the RL community. On the investigated system, the improvement relative to the state of the art seems quite significant, however, the method is only evaluated on a single system.
CLARITY
The paper is mostly well-written. However, in some part the clarity could be improved. For example, the paper does not mention explicitly they assume the reward function to be known. The policy improvement step (8 in the algorithm) is not explained at all. For readers already familiar with PILCO this is quite obvious but it would be good to give a short introduction from the others.
The technical section in 4.1 - 4.2 is very dense. I understand space is rather limited but it could be good to explain this part in more detail in the appendix. For example, the paper states V=E[V'] is assumed fixed, but doesn't state how this fixed value is computed or set.
Figures 1 and 2 are presented as analogues, yet the latent state X is not shown in the right image. It would be easier to understand the relationship if the underlying latent process was shown in both images.
There are quite many grammar issues. I'll point some out under minor comments.
MINOR COMMENTS
Grammar - please rephrase sentences in lines 7, 52, 86-87, 134,209,250, 283, 299-300
Contractions and abbreviations - please write out (doesn't, etc.)
Some sentence start with note but don't seem grammatical. I suggest writing "Note that, ..." or similar.
Line 55: Greater -> Larger
Line 65: Seems like the author name shouldn't appear here
Line 99: Less -> fewer
line 112: "Overfitting leads to bias". Overcapacity generally causes a high variance, not necessarily bias. Or is the bias here a result of optimizing the policy on the erroneous model? (similar in line 256)
line 118: I guess k(x_i, x_j) should have a superscript a.
line 118: are the phi_a additional hyperparameters? How are they set?
line 147, 148 "detailed section" -> detailed in Section"
line 162: it's not clear to me what you mean by "hierarchically random"
line 174: it seems you switched x and z in p(x)=N(z, Sigma).
line 177: Gaussian conditioning is not a product.
line 181: "the distribution f(b)". This seems a bit odd. Maybe assign f(b) to a random variable?
line 189: not sure what you mean by "in principle".
- you might want to introduce notation \mathbb{C} and \mathbb{V}
line 211: "a mixture of basis functions" -> do you mean it is a linear combination of basis functions?
line 245: "physically" -> "to physically"
line 273  "collect" -> to collect
line 274: trails -> trials
line 283: this seems a strong claim, you might want to rephrase (fPILCO doesn't predict how well other methods will do as is implied here )
line 290 - you might want to choose another symbol for noise factor than n (often number of data points).
297 : data - > data points ?
315: Give greater performance than otherwise -> improved performance with respect to the baselines ?
References: I would consistently use either full names or initials. Some references like 11 are quite messy (mentioning the year 3 times in this case). The words Gaussian and POMDPs in paper titles should be capitalized correctly.


Online to Offline Conversions, Universality and Adaptive Minibatch Sizes
We present an approach towards convex optimization that relies on a novel scheme which converts adaptive online algorithms into offline methods. In the offline optimization setting, our derived methods are shown to obtain favourable adaptive guarantees which depend on the harmonic sum of the queried gradients. We further show that our methods implicitly adapt to the objective's structure: in the smooth case fast convergence rates are ensured without any prior knowledge of the smoothness parameter, while still maintaining guarantees in the non-smooth setting. Our approach has a natural extension to the stochastic setting, resulting in a lazy version of SGD (stochastic GD), where minibathces are chosen adaptively depending on the magnitude of the gradients. Thus providing a principled approach towards choosing minibatch sizes.

This paper considers two striking problems in optimizing empirical loss in learning problems: adaptively choosing step size in offline optimization, and mini-batch size in stochastic optimizations.
With a black-box reduction to into AdaGrad algorithm and utilizing gradient normalization and importance weighting in averaging the intermediate solutions, this paper presents a variant of AdaGrad algorithm that attains same convergence rates as GD in convex and smooth optimization without the knowledge of smoothness parameter in setting the learning rate or any linear search (except strongly convex setting, where the algorithm still needs to know the strong convexity parameter H). In stochastic setting, the paper presents a LazySGD algorithm that adaptively chooses the mini-batch size according to the norm of sampled gradients.  The paper is reasonably well written, and the structure and language are good. The paper is technically sound and the proofs seem to be correct as far as I checked. The authors also included some preliminary experiments on a toy example to demonstrates some their theoretical insights. Overall, the paper is well written and has some nice contributions.


Few-Shot Adversarial Domain Adaptation
This work provides a framework for addressing the problem of supervised domain adaptation with deep models. The main idea is to exploit adversarial learning to learn an embedded subspace that simultaneously maximizes the confusion between two domains while semantically aligning their embedding. The supervised setting becomes attractive especially when there are only a few target data samples that need to be labeled. In this few-shot learning scenario, alignment and separation of semantic probability distributions is difficult because of the lack of data. We found that by carefully designing a training scheme whereby the typical binary adversarial discriminator is augmented to distinguish between four different classes, it is possible to effectively address the supervised adaptation problem. In addition, the approach has a high "speed" of adaptation, i.e. it requires an extremely low number of labeled target training samples, even one per category can be effective. We then extensively compare this approach to the state of the art in domain adaptation in two experiments: one using datasets for handwritten digit recognition, and one using datasets for visual object recognition.

Overall
The paper tackles the problem of domain adaptation when a few (up to 4) unlabeled samples exist in the target domain. An approach is presented based on the idea of adversarial training. The main contribution of the paper is in proposing an augmentation approach to alleviate the scarcity of target domain data, that is by taking pairs of source-target samples as input to combinatorially increase the number of available training samples. The approach proves to be very effective on two standard domain adaptation benchmarks, achieving new state-of-the-art results using only 4 labelled samples.
The problem is very relevant in general and the achieved results are substantial. This by itself warrants the paper to be published in NIPS. However, several important experiments are missing for a better and in-depth understanding of the proposed method. Should it have included those experiments, I’d have rated it as top 15% papers.
Related Works
+ A structured and thorough coverage of the related works regarding domain adaptation.
- related works are missing which have GAN’s discriminator to also predict class labels as in this work
- related works are missing on using pairs of samples as for augmenting low number of available data, especially for learning common embedding space
Approach
+ The incremental novelty of the augmentation technique along with the new loss for the context of deep adversarial domain adaptation is a plausible and effective approach.
- Many details are missing: batchsize, learning rate, number of general iterations, number of iterations to train DCD and g_t/h, number of trials in adversarial training of the embedding space before convergence, etc.
- introduction refers to the case where zero samples from a class is available, however this case is not discussed/studied neither in the approach section nor in the experiments.
Experiments
+ the results using only a few labelled samples are impressive. Specially, when compared with prior and concurrent works using similar approaches.
- Why not go above 4? It would be informative to see the diminishing return
- A systematic study of the components of the architecture is missing, specially comparing when g_t and g_s are shared and not, but also regarding many other architectural choices.
- In sec 3.1 it is mentioned a classification loss is necessary for the better working of the adaptation model. It seems conceptually redundant, except with the added loss more weight can be put on classification as opposed to domain confusion. In that respect,  the experimental setup lacks demonstrating the importance of this factor in the loss function.
- It seems the most basic configuration of an adversarial training is used in this work, plots and figures showing the stability of convergence using the introduced losses and architectures is important.
- An experiment where for some classes no target sample is available?


Predicting Scene Parsing and Motion Dynamics in the Future
The ability of predicting the future is important for intelligent systems, e.g. autonomous vehicles and robots to plan early and make decisions accordingly. Future scene parsing and optical flow estimation are two key tasks that help agents better understand their environments as the former provides dense semantic information, i.e. what objects will be present and where they will appear, while the latter provides dense motion information, i.e. how the objects will move. In this paper, we propose a novel model to simultaneously predict scene parsing and optical flow in unobserved future video frames. To our best knowledge, this is the first attempt in jointly predicting scene parsing and motion dynamics. In particular, scene parsing enables structured motion prediction by decomposing optical flow into different groups while optical flow estimation brings reliable pixel-wise correspondence to scene parsing. By exploiting this mutually beneficial relationship, our model shows significantly better parsing and motion prediction results when compared to well-established baselines and individual prediction models on the large-scale Cityscapes dataset. In addition, we also demonstrate that our model can be used to predict the steering angle of the vehicles, which further verifies the ability of our model to learn latent representations of scene dynamics.

The paper proposes a deep-learning-based approach to joint prediction of future optical flow and semantic segmentation in videos. The authors evaluate the approach in a driving scenario and show that the two components - flow prediction and semantic segmentation prediction - benefit from each other. moreover, they show a proof-of-concept experiment on using the predictions for steering angle prediction.
The paper is related to works of Jin et al. and Neverova et al. However, as far as I understand, both of these have not been officially published at the time of submission (and the work of Neverova et al. only appeared on arxiv 2 months before the deadline), so they should not be considered prior art.
Detailed comment:
Pros:
1) The idea seems sound: predicting segmentation and optical flow are both important tasks, and they should be mutually beneficial.
2) The architecture is clean and seems reasonable
3) Experimental evaluation on Citiscapes is reasonable, and some relevant baselines are evaluated: copying previous frame, warping with the estimated flow, predicting segmentation without flow, predicting flow without segmentation, the proposed model without the “transformation layer”. The improvements brought by the modifications are not always huge, but are at least noticeable.
Cons:
1) Steering angle prediction experiments are not convincing at all. The baseline is very unclear, and it is unclear what does the performance of “~4” mean. There is no obvious baseline - training a network with the same architecture as used by the authors, but trained from scratch. The idea of the section is good, but currently it’s not satisfactory at all.
2) I am confused by the loss function design. First, the weighting of the loss component is not mentioned at all - is there indeed no weighting? Second, the semantic segmentation loss is different for human-annotated and non-human-annotated frames - why is that? I understand why human-annotated loss should have a higher weight, but why use a different function - L1+GDL instead of cross-entropy? Have you experimented with other variants?
3) Figures 3 and 4 are not particularly enlightening. Honestly, they do not tell too much. Of course it is always possible to find a sample on which the proposed method works better. The authors could find a better way to represent the results - perhaps with a video, or with better-designed figures.
4) Writing looks rushed and unpolished.  For instance, caption of Table 2 is wrong.
Minor:
5) I’m somewhat surprised the authors use EpicFlow - it’s not quite state of the art nowadays
6) Missing relevant citation: M. Bai, W. Luo, K. Kundu and R. Urtasun: Exploiting Semantic Information and Deep Matching for Optical Flow. ECCV 2016.
7) Many officially published papers are cited as arxiv, to mention a few: Cityscapes, FlowNet, ResNet, Deep multi-scale video prediction, Context encoders
Overall, the paper is interesting and novel, but seems somewhat rushed, especially in some sections. Still, I believe it could be accepted if the authors fix some most embarrassing problems mentioned above, in particular, the part on steering angle prediction.


Variance-based Regularization with Convex Objectives
We develop an approach to risk minimization and stochastic optimization that provides a convex surrogate for variance, allowing near-optimal and computationally efficient trading between approximation and estimation error. Our approach builds off of techniques for distributionally robust optimization and Owen's empirical likelihood, and we provide a number of finite-sample and asymptotic results characterizing the theoretical performance of the estimator. In particular, we show that our procedure comes with certificates of optimality, achieving (in some scenarios) faster rates of convergence than empirical risk minimization by virtue of automatically balancing bias and variance. We give corroborating empirical evidence showing that in practice, the estimator indeed trades between variance and absolute performance on a training sample, improving out-of-sample (test) performance over standard empirical risk minimization for a number of classification problems.

The article "Variance-based Regularization with Convex Objectives" considers the problem of the risk minimization in a wide class of parametric models. The authors propose the convex surrogate for variance which allows for the near-optimal and computationally efficient estimation.
The idea of the paper is to substitute the variance term in the upper bound for the risk of estimation with the convex surrogate based on the certain robust penalty. The authors prove that this surrogate provides good approximation to the variance term and prove certain upper bounds on the overall performance of the method. Moreover, the particular example is provided where the proposed method beats empirical risk minimization.
The experimental part of the paper considers the comparison of empirical risk minimization with proposed robust method on 2 classification problems. The results show the expected bahaviour, i.e. the proposed method is better than ERM on rare classes with not significant loss on more common classes.
To sum up, I think that this is very strong work on the edge between the theoretical and practical machine learning. My only concern is that the paper includes involved theoretical analysis and is much more suitable for full journal publication than for short conference paper (the Arxiv paper is 47 pages long!!!).


Subset Selection under Noise
The problem of selecting the best k-element subset from a universe is involved in many applications. While previous studies assumed a noise-free environment or a noisy monotone submodular objective function, this paper considers a more realistic and general situation where the evaluation of a subset is a noisy monotone function (not necessarily submodular), with both multiplicative and additive noises. To understand the impact of the noise, we firstly show the approximation ratio of the greedy algorithm and POSS, two powerful algorithms for noise-free subset selection, in the noisy environments. We then propose to incorporate a noise-aware strategy into POSS, resulting in the new PONSS algorithm. We prove that PONSS can achieve a better approximation ratio under some assumption such as i.i.d. noise distribution. The empirical results on influence maximization and sparse regression problems show the superior performance of PONSS.

This paper considers the problem of maximizing a monotone set function subject to a cardinality constraint. The authors consider a novel combination of functions with both bounded submodularity ratio and additive noise. These setting have been considered separately before, but a joint analysis leads to a novel algorithm PONSS. This has improved theoretical guarantees and experimental performance when compared to previous noise-agnostic greedy algorithms. The paper flows well and is generally a pleasure to read.
Clarity
Small typo: "better on subclass problems" on line 26
The paper could benefit from a more thorough literature survey, including more recent results on the tightness of the submodularity ratio and applications to sparse regression [1] [2].
One comment on the beginning of Section 2: in this setting minimizing f subject to a cardinality constraint is not the same as maximizing -f subject to a cardinality constraint. Generally if -f is (weakly) submodular then f is (weakly) supermodular, and -f will not have the same submodularity ratio as f.
I would suggest including "i.i.d. noise distribution" when the assumptions for PONSS instead are first mentioned on page 2 ("fixed noise distribution"), instead of waiting until Section 5.
Impact
Influence maximization is an interesting application of noisy submodular optimization, since the objective function is often hard to compute exactly.
The authors should address that the PONSS guarantees hold with probability < 1/2 while the other guarantees are deterministic, and explain whether this affects the claim that the PONSS approximation ratio is "better".
O(Bnk^2) iterations can be prohibitive for many applications, and approximate/accelerated versions of the standard greedy algorithm can run in sublinear time.
Experiments
Most of my concerns are with the experiments section. In the sparse regression experiment, the authors report final performance on the entire dataset. They should explain why the did not optimize on a training set and then report final performance on a test set. Typically the training set is used to check how algorithms perform on the optimization, and the test set is used to check whether the solution generalizes without overfitting. Additionally, it appears that all of the R^2 values continue to increase at the largest value k. The authors should continue running for larger k or state that they were limited by large running times of the algorithms.
It would be more interesting to compare PONSS with other greedy algorithms (double greedy, random greedy, forward backward greedy, etc.), which are known to perform better than standard greedy on general matroid constraints
Questions
-What is the meaning of multiplicative domination in PONSS experiments if the noise estimate parameter \theta is set to 1? It would be interesting to quantify the drop in performance when 1 > \theta^' > \epsilon is used instead of \epsilon.
-For what range of memory B do the claims on line 170 hold?
-Could PONSS be parallellized or otherwise accelerated to reduce running time?
[1] A. A. Bian, J. M. Buhmann, A. Krause, and S. Tschiatschek. Guarantees for greedy maximization of non-submodular functions with applications, ICML 2017.
[2] E. R. Elenberg, R. Khanna, A. G. Dimakis, and S. Negahban. Restricted strong convexity implies weak submodularity, https://arxiv.org/abs/1612.00804


Partial Hard Thresholding: Towards A Principled Analysis of Support Recovery
In machine learning and compressed sensing, it is of central importance to understand when a tractable algorithm recovers the support of a sparse signal from its compressed measurements. In this paper, we present a principled analysis on the support recovery performance for a family of hard thresholding algorithms. To this end, we appeal to the partial hard thresholding (PHT) operator proposed recently by Jain et al. [IEEE Trans. Information Theory, 2017]. We show that under proper conditions, PHT recovers an arbitrary s-sparse signal within O(sκ log κ) iterations where κ is an appropriate condition number. Specifying the PHT operator, we obtain the best known results for hard thresholding pursuit and orthogonal matching pursuit with replacement. Experiments on the simulated data complement our theoretical findings and also illustrate the effectiveness of PHT.

Review prior to rebuttal:
The paper provides analytical results on partial hard thresholding for sparse signal recovery that are based on RIP and on an alternative condition (RSC and RSS) and that is applicable to arbitrary signals.
The description of previous results is confusing at times. First, [4] has more generic results than indicated in this paper in line 51. Remark 7 provides a result for arbitrary signals, while Theorems 5 and 6 consider all sufficiently sparse signals. Second, [13] provides a guarantee for sparse signal recovery (Theorem 4), and so it is not clear what the authors mean by “parameter estimation only” on line 59. If exact recovery is achieved, it is obvious that the support is estimated correctly too. Third, it is also not clear why Theorem 2 is described as an RIP-based guarantee - how does RIP connect to the condition number, in particular since RIP is a property that can go beyond specific matrix constructions?
The simulation restricts itself to s-sparse signals, and so it is not clear that the authors are testing the aspects of the results that go beyond those available in existing work. It is also not clear how the results test the dependence on the condition number of the matrix.
Minor comments follow.
Title: “A Towards” is not grammatically correct.
Line 128: the names of the properties are grammatically incorrect: convex -> convexity, smooth -> smoothness? or add “property” at the end of each?
Line 136: calling M/m “the condition number of the problem” can be confusing, since this terminology is already used in linear algebra. Is there a relationship between these two quantities?
Line 140: there is no termination condition given in the algorithm. Does this mean when the gradient in the first step is equal to zero? When the support St does not change in consecutive iterations?
Line 185: the paper refers to a comparison of the analysis of [4] being “confined to a special signal” versus the manuscript’s “generalization… [to] a family of algorithms”. It is not clear how these two things compare to one another.
Line 301: degrade -> degradation
Line 304: reduces -> reduce
Response to rebuttal: The authors have addressed some of the points above (in particular relevant to the description of prior work), but the impact of the contribution appears to be "borderline" for acceptance. Given that the paper is acceptable if the authors implement the changes described in the rebuttal, I have set my score to "marginally above threshold".


Multiresolution Kernel Approximation for Gaussian Process Regression
Gaussian process regression generally does not scale to beyond a few thousands data points without applying some sort of kernel approximation method. Most approximations focus on the high eigenvalue part of the spectrum of the kernel matrix, K, which leads to bad performance when the length scale of the kernel is small. In this paper we introduce Multiresolution Kernel Approximation (MKA), the first true broad bandwidth kernel approximation algorithm. Important points about MKA are that it is memory efficient, and it is a direct method, which means that it also makes it easy to approximate K −1 and det(K).

The authors consider the problem of large-scale GP regression; they propose a multiresolution approximation method for the Gram matrix K. In the literature, most approximation approaches assume either (1) a low rank representation for K, which may not be supported by the data, or (2) a block-diagonal form for K, the structure of which has to be identified by clustering methods, which is not trivial for high-dimensional data. The current paper proposes MKA, a novel approximation approach that uses captures local and global properties for K. The Gram matrix K is approximated as a Kronecker sum of low-rank and diagonal matrices, a fact that significantly reduces the computational complexity of the linear algebra calculations required in the context of GP regression.

The paper initiates a very interesting discussion on the nature of local and global kernel approximations, but I feel that certain aspects ofthe methodology proposed are not sufficiently clear. Below, I list some considerations that I had while reading the paper.

What is the effect of the clustering mentioned in step 1 of the methodology? Is the method less sensitive to the result of clustering than the local-based methods in the literature?
Does the approximate matrix \tilde{K} converge to true matrix K as the d_{core} parameter is increased? By looking at the experiments of Figure 2, it appears that MKA is rather insensitive to the d_{core} value.

The effect of approximating K in many stages as described in Section 3 is not obvious or trivial. The authors attribute the flexibility of MKA to the reclustering of K_l before every stage. However, I would suspect that a slight variation of the output at any stage would have dramatic consequences in the stages that follow.

It is not clear which compression method was used in the experiments. I would think that the cost of SPCA is prohibitive, given the objective of the current paper. It could still be worth mentioning SPCA if there was some experimental comparison with the use of MMF in the context of MKA.

I think that the experimental section would be stronger if there was also a demonstration of how well the MKA approximates the original GP. Although we get an idea of the predictive mean in Figure 1, there is no information of the predictive variance.

Minor comments:
The line for the Full GP method in Figure 2 is almost not visible.


Online Prediction with Selfish Experts
We consider the problem of binary prediction with expert advice in settings where experts have agency and seek to maximize their credibility. This paper makes three main contributions. First, it defines a model to reason formally about settings with selfish experts, and demonstrates that "incentive compatible" (IC) algorithms are closely related to the design of proper scoring rules. Second, we design IC algorithms with good performance guarantees for the absolute loss function. Third, we give a formal separation between the power of online prediction with selfish versus honest experts by proving lower bounds for both IC and non-IC algorithms. In particular, with selfish experts and the absolute loss function, there is no (randomized) algorithm for online prediction-IC or otherwise-with asymptotically vanishing regret.

In this paper, the standard (binary outcome) "prediction with expert advice" setting is altered by treating the experts themselves as players in the game.  Specifically, restricting to the class of weight-update algorithms, the experts seek to maximize the weight assigned to them by the algorithm, as a notion of "rating" for their predictions.  It is shown that learning algorithms are IC, meaning that experts reveal their true beliefs to the algorithm, if and only if the weight-update function is a strictly proper scoring rule.  Because the weighted-majority algorithm (WM) has a weight update which is basically the negative of the loss function, WN is thus IC if and only if the (negative) loss is strictly proper.  As the (negative) absolute loss function is not strictly proper, WM is not IC for absolute loss.  The remainder of the paper investigates the absolute loss case in detail, showing that WM with the spherical scoring rule performs well, and moreover performs better than WM with absolute loss; no other deterministic IC (or even non-IC) algorithm for absolute loss can beat the performance in the non-strategic case; analogous results for the randomized case.
I like the results, but find the motivation to be on loose footing.  First, it is somewhat odd that experts would want to maximize the algorithm's weight assigned to them, as after all the algorithm's weights may not correspond to "performance" (a modified WM which publishes bogus weights intermittently can still achieve no regret), and even worse, the algorithm need not operate by calculating weights at all.  I did find the 538 example compelling however.
My biggest concern is the motivation behind scoring the algorithm or the experts with non-proper scoring rules.  A major thrust of elicitation (see Making and Evaluating Point Forecasts, Gneiting 2011) is that one should always measure performance in an incentive-compatible way, which would rule out absolute loss.  The authors do provide an example in 538's pollster ratings, but herein lies a subtlety: 538 is using absolute loss to score *poll* outcomes, based on the true election tallies.  Poll results are not probabilistic forecasts, and in fact, that is a major criticism of polls from e.g. the prediction market community.  Indeed, the "outcome" for 538's scoring is itself a real number between 0 and 1, and not just the binary 0,1 of e.g. whether the candidate won or lost.
I would be much more positive about the paper if the authors could find a natural setting where non-proper losses would be used.
Minor comment:  Using "r" as the outcome may be confusing as it often denotes a report.  It seems that "y" is more common, in both the scoring rules and prediction with expert advice literature.


Alternating Estimation for Structured High-Dimensional Multi-Response Models
We consider the problem of learning high-dimensional multi-response linear models with structured parameters. By exploiting the noise correlations among different responses, we propose an alternating estimation (AltEst) procedure to estimate the model parameters based on the generalized Dantzig selector (GDS). Under suitable sample size and resampling assumptions, we show that the error of the estimates generated by AltEst, with high probability, converges linearly to certain minimum achievable level, which can be tersely expressed by a few geometric measures, such as Gaussian width of sets related to the parameter structure. To the best of our knowledge, this is the first non-asymptotic statistical guarantee for such AltEst-type algorithm applied to estimation with general structures.

Bounds on the estimation error in a multi-response structured Gaussian linear model is considered, where the parameter to be estimated is shared by all m responses and where the noise covariance may be different from identity. The authors propose an alternating procedure for estimation of the coefficient vector that essentially extends the GDS (generalized Dantzig selector). The new procedure accommodates general "low-complexity" structures on the parameter and non-identity covariance of the noise term.
The paper is outside my field of expertise, so I apologize in advance for any mistakes due to my misunderstanding.
The paper is very technical, and without the appropriate background, I was not able to really appreciate the significance of the contribution. That said, the theme of the results is quite clear, especially with the effort of the authors to supplement the results with intuitive explanations.
1. The paper is very technical, maybe too technical, and it is hard for a non-expert like myself to make sense of the conditions/assumptions made in the different theorems (despite the authors' evident effort to help with making sense of these conditions).
2. Again, I was not able to appreciate if the different assumptions and results are sensible or whether they are mostly ad-hoc or artifacts of the proofs, but overall the contribution over GDS and existing methods does not appear to be sufficiently significant for acceptance to NIPS.
3. The manuscript is not free of grammatical errors or bad wording, for example:
-l.168: "at the same order"
-caption of Fig 2: "The plots for oracle...implies..."
-caption of Fig 2: "keep unchanged" (better: "remain unchanged")
## comments after reading author rebuttal:
The authors have provided a clear non-technical summary of the paper, which is much appreciated. I would like to apologize for not being able to give a fair evaluation when originally reviewing the paper (this is because I did not have the technical background to appreciate the results).


On Tensor Train Rank Minimization: Statistical Efficiency and Scalable Algorithm
Tensor train (TT) decomposition provides a space-efficient representation for higher-order tensors. Despite its advantage, we face two crucial limitations when we apply the TT decomposition to machine learning problems: the lack of statistical theory and of scalable algorithms. In this paper, we address the limitations. First, we introduce a convex relaxation of the TT decomposition problem and derive its error bound for the tensor completion task. Next, we develop a randomized optimization method, in which the time complexity is as efficient as the space complexity is. In experiments, we numerically confirm the derived bounds and empirically demonstrate the performance of our method with a real higher-order tensor.

This paper looks at the tensor train (TT) decomposition as an efficient storage method for tensors with statistical guarantees.
Traditional methods for tensor decomposition include Tucker decomposition [29] and CP decomposition. The advantage of TT decomposition is that it can avoid the exponential space complexity of Tucker decomposition. However, its statistical performance is less understood, which the authors attempt to address in this paper.
In addition, the authors provide an efficient alternative to existing TT algorithms.
It’s interesting to note that Tucker and TT decompositions have complementary bottleneck - One has space complexity issues and the second has computational complexity issues.
The authors look at the problem of tensor completion which is an extension of the matrix completion problem that has been extensively researched in the literature.
In matrix completion problem, the standard assumption made is that the matrix is low-rank so that the problem becomes well-posed. This assumption is also observed to be the case in practical applications such as recommender system.
The authors make a similar observation for the tensor to be estimated (i.e. a low-rank tensor). Just as nuclear norm/schatten-p norm is used a relaxation to the rank matrix, the authors propose the Schatten TT norm as a convex relaxation of the TT rank of a tensor.
Comments:
- The authors mention that the TT rank is a tuple in section 2, while mentioning later in section 6 that it can be described by a single parameter, making the definition confusing.
- The problem formulation (4), the algorithmic approach (section 3.1) all mimick the ideas in matrix completion literature. The innovation is more to do with the appropriate representation of tensors while extending these ideas from the matrix completion literature.
- Even in the matrix factorization literature, the scalable methods are the ones that already assume a low-rank factor structure for the matrix to be completed to save space and computation. For tensors, it is all the more imperative to assume a low-rank factor structure. The authors do this in (6) to save space and time.
- Theorem 1 is quite interesting and provides a more tractable surrogate for the TT schatten norm.
- The statistical results are a new contribution for tensor completion, although the assumption of incoherence (Assumption 2) and subsequent proofs are by now very standard in the M-estimation literature.
- The authors use an ADMM algorithm again for solving (6) in section 4.2. It is not clear this would have a better advantage as compared to just the simpler alternating minimization over the TT factors of the tensor. It is not clear if the convergence of ADMM to a local minima for non-convex problems with multiple factors is proven in the literature. The authors don’t cite a paper on this either. On the other hand, it is easy to see that the simpler alternating minimization (keeping other factors fixed)
converges to a local optimum or a critical point. The authors don’t discuss any of these details in 4.2.
- The number of samples needed for statistical convergence is in the worst-case the the product of the dimensions of the tensor. Perhaps a better result would be to show that the sample complexity is proportional to the true TT rank of the tensor? The authors don’t mention any discussion on the lower-bound on the sample complexity in the paper, making the goodness of the sample complexity bound in Theorem 5 questionable.
Overall, the paper presents an interesting investigation of the TT decomposition for tensors and uses it in the context of tensor completion with statistical guarantees. It is not clear if the algorithm used for alternating optimization (ADMM) is actually  guaranteed to converge though it might work in practice. Also, the authors don’t discuss the goodness (what’s a good lower bound?) of the sample complexity bounds derived for their algorithm for tensor completion. The approach to modeling, algorithm development, scalability and statistical convergence mimicks that taken for matrix completion - However, the authors do a good job of combining it all together and putting their approach in perspective with respective to existing algorithms in the literature.


Robust Imitation of Diverse Behaviors
Deep generative models have recently shown great promise in imitation learning for motor control. Given enough data, even supervised approaches can do one-shot imitation learning; however, they are vulnerable to cascading failures when the agent trajectory diverges from the demonstrations. Compared to purely supervised methods, Generative Adversarial Imitation Learning (GAIL) can learn more robust controllers from fewer demonstrations, but is inherently mode-seeking and more difficult to train. In this paper, we show how to combine the favourable aspects of these two approaches. The base of our model is a new type of variational autoencoder on demonstration trajectories that learns semantic policy embeddings. We show that these embeddings can be learned on a 9 DoF Jaco robot arm in reaching tasks, and then smoothly interpolated with a resulting smooth interpolation of reaching behavior. Leveraging these policy representations, we develop a new version of GAIL that (1) is much more robust than the purely-supervised controller, especially with few demonstrations, and (2) avoids mode collapse, capturing many diverse behaviors when GAIL on its own does not. We demonstrate our approach on learning diverse gaits from demonstration on a 2D biped and a 62 DoF 3D humanoid in the MuJoCo physics environment.

This work deals with the problem of modeling joint action/state trajectory spaces of dynamical systems using a supervised imitation learning paradigm.  It approaches the problem by defining a Variational Autoencoder (VAE) that maps the state sequence, via a latent representation, into a (state,action) pair.
The approach is validated on three articulated body controller modeling problems: a robotic arm, a 2D biped, and a 3D human body motion modeling.
Prior work: the work closely follows (extends) the approach of Generative Adversarial Imitation Learning [11].  The difference here is that, through the use of VAE, the authors claim to have avoided the pitfall of GAN know as mode collapsing.
Summary
+ Addresses a challenging problem of learning complex dynamics controllers / control policies
+ Well-written introduction / motivation
+ Appealing qualitative results on the three evaluation problems. Interesting experiments with motion transitioning.
- Modeling formulation is somewhat different from GAIL (latent representation) but it rather closely follows GAIL
- Many key details are omitted (either on purpose, placed in appendix, or simply absent, like the lack of definitions of terms in the modeling section, details of the planner model, simulation process, or the details of experimental settings)
- Experimental evaluation is largely subjective (videos of robotic arm/biped/3D human motion)
- Paper appears written in haste or rewritten from a SIGGRAPH like submission
- Relies significantly on work presented in an anonymous NIPS submission
Detailed comments
In general, I like the focus of this paper; designing complex controllers for dynamic systems is an extremely challenging task and the approach proposed here looks reasonable.  The use of the VAE is justified and the results, subjectively, are appealing.
However, so many details are missing here and some parts of the discussion are unclear or not justified, in my view.
For instance, the whole modeling section (3), while not so obscure, still ends up not defining many terms.  Eg, what is \pi_E in the first eq of 3.2? (Not numbered, btw) What is \theta in eq3? (also used elsewhere) .  What are all the weight terms in the policy model, end of sec 3 (not numbered)?  You say you initialize theta to alpha wights, but what does that mean?
What is the purpose of Lemma 1?  The whole statement following it, l 135-145 sound unclear and contradictory.  Eg, you say that eq5 avoids the problem of collapse, yet you state in those lines that is also has the same problem?
The decoder models for (x,a) are not explicitly defined.
How is the trajectory simulation process actually accomplished?  This is somewhat tersely described in l168-172.
How are transitions between categories of different motion types modeled as you do not explicitly encode the category class?  It is obviously done in the z-space, but is it some linear interpolation?  It appears to be (l174-176) but is it timed or instantaneous?
What are all the numbers in the Action Decoder Sizes column of Tab 1, appendix?


Learning ReLUs via Gradient Descent
In this paper we study the problem of learning Rectified Linear Units (ReLUs) which are functions of the form x ↦ max(0, ⟨w, x⟩) with w ∈ R d denoting the weight vector. We study this problem in the high-dimensional regime where the number of observations are fewer than the dimension of the weight vector. We assume that the weight vector belongs to some closed set (convex or nonconvex) which captures known side-information about its structure. We focus on the realizable model where the inputs are chosen i.i.d. from a Gaussian distribution and the labels are generated according to a planted weight vector. We show that projected gradient descent, when initialized at 0, converges at a linear rate to the planted model with a number of samples that is optimal up to numerical constants. Our results on the dynamics of convergence of these very shallow neural nets may provide some insights towards understanding the dynamics of deeper architectures.

### Summary
The paper proves that projected gradient descent in a single-layer ReLU network converges to the global minimum linearly: ||w_t-w^*||/||w_0-w^*||=(1/2)^t with high probability. The analysis assumes the input samples are iid Gaussian and the output is realizable, i.e. the target value y_k for input x_k is constructed via y_k=ReLU(w^*.x_k). The paper studies a regression setting and uses least squares as the loss function.
### Pros and Cons
To the best of my knowledge, the result is novel; I am not aware of any proof of "convergence to global optimum" for the nonconvex loss resulted by fitting to a ReLu function via least squares. On top of that, the analysis considers a nonconvex regularization term with little assumptions on it. Of course, there still a clear gap between the results of this paper and models used in practice: notably, relaxing the Gaussian iid input assumption, and extending to multilayer setup. However, this result may become a stepping stone toward that goal.
### Proofs
I followed the building blocks of the proof at the coarse level, and the logic makes sense. However, I did not check at a very detailed level. I was wondering if the authors could include a simple simulation to confirm the correctness of the result? Given that the data is very simple (iid Gaussian) and the network is a single layer ReLU, using a very simple regularization scheme, it seems to take a few lines of Matlab to verify the empirical convergence rate and compare that with the result. In fact, I wonder why this is not attempted by the authors already?
### Regularization
The authors state that the result holds true regardless of regularization R being convex or nonconvex. What if R is such that it creates two far apart and disjoint regions, one containing the origin (initialization) and the other containing w^*. Wouldn't this create a problem for projected gradient descent initialized by w=0 to reach w^*?

### Minor Comments
1. There are typos, please proof read.
2. What is the meaning of A in L_A (Eq 5.12)?
3. What is the value of the constant gamma in line 180, or otherwise how is it related to other constants or quantities in the paper?


The Expressive Power of Neural Networks: A View from the Width
The expressive power of neural networks is important for understanding deep learning. Most existing works consider this problem from the view of the depth of a network. In this paper, we study how width affects the expressiveness of neural networks. Classical results state that depth-bounded (e.g. depth-2) networks with suitable activation functions are universal approximators. We show a universal approximation theorem for width-bounded ReLU networks: width-(n + 4) ReLU networks, where n is the input dimension, are universal approximators. Moreover, except for a measure zero set, all functions cannot be approximated by width-n ReLU networks, which exhibits a phase transition. Several recent works demonstrate the benefits of depth by proving the depth-efficiency of neural networks. That is, there are classes of deep networks which cannot be realized by any shallow network whose size is no more than an exponential bound. Here we pose the dual question on the width-efficiency of ReLU networks: Are there wide networks that cannot be realized by narrow networks whose size is not substantially larger? We show that there exist classes of wide networks which cannot be realized by any narrow network whose depth is no more than a polynomial bound. On the other hand, we demonstrate by extensive experiments that narrow networks whose size exceed the polynomial bound by a constant factor can approximate wide and shallow network with high accuracy. Our results provide more comprehensive evidence that depth may be more effective than width for the expressiveness of ReLU networks.

SUMMARY
* This paper studies how width affects the expressive power of neural networks. It suggests an alternative line of argumentation (representation of shallow by deep narrow) to the view that depth is more effective than width in terms of expressivity. The paper offers partial theoretical results and experiments in support of this idea.
CLARITY
* The paper expresses the objectives clearly and is easy to follow. However, the technical quality needs attention, including confused notions of continuity and norms.
RELEVANCE
* The paper discusses that too narrow networks cannot be universal approximators. This is not surprising (since the local linear map computed by a ReLU layer has rank at most equal to the number of active units). Nonetheless, it highlights the fact that not only depth but also width is important.
* The universal approximation results seem to add, but not that much to what was already known. Here it would help to comment on the differences.
* The paper presents a polynomial lower bound on the required number of narrow layers in order to represent a shallow net, but a corresponding upper bound is not provided.
* As an argument in favor of depth, the theoretical results do not seem to be conclusive. For comparison, other works focusing on depth efficiency show that exponentially larger shallow nets are needed to express deep nets. The paper also presents experiments indicating that a polynomial number of layers indeed allows deep narrow nets to express functions computable by a shallow net.
OTHER COMMENTS
* The paper claims that ``as pointed out in [2]: There is always a positive measure of network parameters such that deep nets can be realized by shallow ones without substantially larger size''.  This needs further clarification, as that paper states `` we prove that besides a negligible set, all functions that can be implemented by a deep network of polynomial size, require exponential size in order to be realized (or even approximated) by a shallow network.''
* In line 84, what is the number of input units / how do the number of parameters of the two networks compare?
* In related works, how to the bounds from [14,18] compare with the new results presented in this paper?
* In line 133. Lebesgue measurable functions are not generalizations of continuous functions.
* In line 134 the argumentation for L1 makes no sense.
* In line 163 ``if we ignore the size of the network ... efficient for universal approximation'' makes no sense.
* The comparison with previous results could be made more clear.
* Theorem 2 is measuring the error over all of R^n, which does not seem reasonable for a no-representability statement.
* Theorem 3 talks about the existence of an error epsilon, which does not seem reasonable for a no-representability statement. Same in Theorem 4.
* In line 235 the number of points should increase with the power of the dimension.
CONCLUSION
The paper discusses an interesting question, but does not answer it conclusively. Some technical details need attention.


The power of absolute discounting: all-dimensional distribution estimation
Categorical models are a natural fit for many problems. When learning the distribution of categories from samples, high-dimensionality may dilute the data. Minimax optimality is too pessimistic to remedy this issue. A serendipitously discovered estimator, absolute discounting, corrects empirical frequencies by subtracting a constant from observed categories, which it then redistributes among the unobserved. It outperforms classical estimators empirically, and has been used extensively in natural language modeling. In this paper, we rigorously explain the prowess of this estimator using less pessimistic notions. We show that (1) absolute discounting recovers classical minimax KL-risk rates, (2) it is adaptive to an effective dimension rather than the true dimension, (3) it is strongly related to the Good-Turing estimator and inherits its competitive properties. We use powerlaw distributions as the cornerstone of these results. We validate the theory via synthetic data and an application to the Global Terrorism Database.

SUMMARY: The paper discusses methods for estimating categorical distributions, in particular a method called "absolute discounting". The paper gives bounds for the KL-risk of absolute discounting, and discusses properties such as minimax rate-optimality, adaptivity, and its relation to the Good–Turing estimator.
CLARITY: The paper is presented in reasonably clear language, and is well-structured.
NOVELTY: The contributions of the paper seem good, but perhaps rather incremental, and not ground-breakingly novel. The paper advocates for the use of absolute discounting, and gives good arguments in favor, including theoretical properties and some experimental results. But the technique as such isn't especially novel, and more general versions of it exist that aren't referenced. The literature review might not be thorough enough: for example, there are many relevant techniques in Chen & Goodman's comprehensive and widely-cited report (1996, 1998) that aren't mentioned or compared to. The paper does cite this report, but could perhaps engage with it better.
Also, the paper does not give any insight on its relation to clearly relevant Bayesian techniques (e.g. Pitman-Yor processes and CRPs) that are similar to (and perhaps more general than) the form absolute discounting presented in this paper. These other techniques probably deserve a theoretical and/or experimental comparison. At the very least they could be mentioned, and it would be helpful to have their relationship to "absolute discounting" clarified.
SCORE:
Overall, I enjoyed reading the paper, and I hope the authors will feel encouraged to continue and improve their work. The main contributions seem useful, but I have a feeling that their relevance to other techniques and existing research isn't clarified enough in this version of the paper.
A few minor suggestions for improvements to the paper follow below.
SECTION 8 (Experiments):
Some questions.
Q1: Is 500 Monte-Carlo iterations enough?
Q2: If the discount value is set based on the data, is that cheating by using the data twice?
LANGUAGE:
Some of the phrasing might be a bit overconfident, e.g.:
  Line 1: "Categorical models are the natural fit..." -> "Categorical models are a natural fit..."?
  Line 48: "[we report on some experiments], which showcases perfectly the all-dimensional learning power of absolute discounting" (really, "perfectly"?)
  Line 291: "This perfectly captures the importance of using structure [...]"
NOTATION:
Totally minor, but perhaps worth mentioning as it occurs a lot: "Good-Turing" should be written with an en-dash ("–" in Unicode, "--" in LaTeX), and not with a hyphen ("-" in Unicode, "-" in LaTeX).
BIBLIOGRAPHY:
In the bibliography, "Good–Turing" is often written in lowercase ("good-turing"), perhaps as a result of forgetting the curly brace protections in the BibTeX source file. (Correct syntax: "{G}ood--{T}uring".)
The final reference (Zip35, Line 364) seems incomplete: only author, title and year are mentioned, no information is given where or how the paper was published.


Structured Bayesian Pruning via Log-Normal Multiplicative Noise
Dropout-based regularization methods can be regarded as injecting random noise with pre-defined magnitude to different parts of the neural network during training. It was recently shown that Bayesian dropout procedure not only improves generalization but also leads to extremely sparse neural architectures by automatically setting the individual noise magnitude per weight. However, this sparsity can hardly be used for acceleration since it is unstructured. In the paper, we propose a new Bayesian model that takes into account the computational structure of neural networks and provides structured sparsity, e.g. removes neurons and/or convolutional channels in CNNs. To do this we inject noise to the neurons outputs while keeping the weights unregularized. We establish the probabilistic model with a proper truncated log-uniform prior over the noise and truncated log-normal variational approximation that ensures that the KL-term in the evidence lower bound is computed in closed-form. The model leads to structured sparsity by removing elements with a low SNR from the computation graph and provides significant acceleration on a number of deep neural architectures. The model is easy to implement as it can be formulated as a separate dropout-like layer.
employs group-wise sparsity in convolutional filters, Perforated CNNs [3] drop redundant rows from the intermediate dataframe matrices that are used to compute convolutions, and Structured Sparsity Learning [24] provides a way to remove entire convolutional filters or even layers in residual networks. These methods allow to obtain practical acceleration with little to no modifications of the existing software. In this paper, we propose a tool that is able to induce an arbitrary pattern of structured sparsity on neural network parameters or intermediate data tensors. We propose a dropout-like layer with a parametric multiplicative noise and use stochastic variational inference to tune its parameters in a Bayesian way. We introduce a proper analog of sparsity-inducing log-uniform prior distribution [8,16] that allows us to formulate a correct probabilistic model and avoid the problems that come from using an improper prior. This way we obtain a novel Bayesian method of regularization of neural networks that results in structured sparsity. Our model can be represented as a separate dropout-like layer that allows for a simple and flexible implementation with almost no computational overhead, and can be incorporated into existing neural networks.
Our experiments show that our model leads to high group sparsity level and significant acceleration of convolutional neural networks with negligible accuracy drop. We demonstrate the performance of our method on LeNet and VGG-like architectures using MNIST and CIFAR-10 datasets.

Summary:
The paper addresses the actual problem of compression and efficient computation of deep NNs. It presents a new pruning technique for deep NNs, which is an enhancement of the Bayesian dropout method. Contrary to standard Bayesian dropout, the proposed method prunes NNs in a structured way. The NN-model is altered by dropout layers with an advanced probabilistic model (with truncated log-uniform prior distribution and truncated log-uniform posterior distribution). For the altered model, the authors propose a new "signal-to-ratio" pruning metric applicable to both individual neurons and more complex structures (e.g., filters).
The submission and supplementary files contain a theoretical explanation of the method and derivation of the corresponding formulas. Moreover, an experimental evaluation of the approach is presented, with promising results.
Quality:
The paper has very good technical quality. It contains a thorough theoretical analysis of the proposed model and its properties. It also presents several experimental results, that evaluate the qualities of the method and compare it with two concurrent approaches.
Clarity:
The presentation is comprehensible and well-organized, the method is described in detail including derivation of formulas and implementation details.
Originality and significance:
The paper addresses the actual problem of compression and efficient computation of deep NNs. Its main contribution is the presentation of a new general technique for structured pruning of NNs that is based on Bayesian dropout. Contrary to the original method,
1) it uses more sophisticated and proper noise distributions and it involves a novel pruning metric,
2) the subject of pruning are not weights, but more complex structures (e.g., filters, neurons).
3) The method can be applied to several NN architectures (CNN, fully-connected,...), and it seems that it can be easily added to existing NN implementations.
The presented approach proved in the experiments to massively prune the network and to speed up the computation on GPU and CPU, while inducing only small accuracy loss. However, based on the experimental results presented, the benefit over the concurrent "SSL" approach is relatively small.
Typo:
...that is has ... (line 117)
Comments:
- Table 1 and Table 2 contain rows with results for concurrent methods called "SSL" and "SparseVD", without a short description of the main principles of these methods in the text.
- It would be useful to assess also the efficiency of the pruning process itself.
- The method has several parameters that need to be set. An interesting question is, how does the actual setting of these parameters affect the balance between sparsity and accuracy of the final model.


Inverse Reward Design
Autonomous agents optimize the reward function we give them. What they don't know is how hard it is for us to design a reward function that actually captures what we want. When designing the reward, we might think of some specific training scenarios, and make sure that the reward will lead to the right behavior in those scenarios. Inevitably, agents encounter new scenarios (e.g., new types of terrain) where optimizing that same reward may lead to undesired behavior. Our insight is that reward functions are merely observations about what the designer actually wants, and that they should be interpreted in the context in which they were designed. We introduce inverse reward design (IRD) as the problem of inferring the true objective based on the designed reward and the training MDP. We introduce approximate methods for solving IRD problems, and use their solution to plan risk-averse behavior in test MDPs. Empirical results suggest that this approach can help alleviate negative side effects of misspecified reward functions and mitigate reward hacking.

The authors develop "Inverse Reward Design" which is intended to help mitigate unintended consequences when designing reward functions that will guide agent behaviour. This is a highly relevant topic for the NIPS audience and a very interesting idea.

        The main potential contribution of this paper would be the clarity of thought it could bring to how issues of invariance can speak to the IRL problem. However, I find that the current presentation to be unclear at many points, particularly through the development of the main ideas. I have done my best to detail those points below.

        My main concern is that the presentation leading up to and including 4.2 is very confused.

        The authors define P(\tilde w | w^*). (They always condition on the same \tilde M so I am dropping that.)

        At l.148, the authors state that through Bayes' rule, get  P(w^* | \tilde w) \propto P(\tilde w | w^*) P(w^*)

        In the text, the display just after l.148 is where things start to get confused, by the use of P(w) to mean P(w^* = w)

        The full Bayes' rule statement is of course P(w^* | \tilde w) = P(\tilde w | w^*) P(w^*) / P(\tilde w), or
        P(w^* | \tilde w) = P(\tilde w | w^*) P(w^*) / \int_\w^* P(\tilde w | w^*) P(w^*) dw^*

        I am confused because 1) the authors don't address the prior P(w^*) and 2) they state that the normalizing constant integrates "over the space of possible proxy rewards" and talk about how this is intractable "if \tilde{w} lies in an infinite or large finite set." Since w^* and \tilde{w} live in the same space (I believe) this is technically correct, but a very strange statement, since the integral in question is over w^*. I am concerned that there is substantial confusion here at the core idea of the whole paper. I suspect that this has resulted just from notational confusion, which often arises when shorthand [P(w^*)] and longhand [P(w^* = w)] notation is used for probabilities. I also suspect that whatever was implemented for the experiments was probably the right thing. However, I cannot assess this in the current state of the paper.

        Finally, I would suggest that the authors address two questions that I think are natural in this setting. 1) What do we do about the prior P(w^*)? (Obviously there won't be a universal answer for this, but I imagine the authors have some thoughts.) 2)

        Below are additional comments:

        l.26 - The looping example isn't given in enough detail to help the reader. The example should stand alone from the cited paper.
        Figure 1 - I understand the podium in the figure on the right, but the meaning of the numbers is not clear. E.g. is the first number supposed to convey [2][0] or twenty?

        l.50 - "Extracting the true reward is..." - Do you mean "We define..."

        Figure 2: uknown

        l.134 - I think somewhere *much* earlier it needs to be made clear that this is a feature-based reward function that generalizes over different kinds of states.

        l.138 - \xi and \phi(\xi) are not precisely defined. I don't know what it means for an agent to "select trajectory \xi"

        l.145 - Notation in (1) and (2) is confusing. I find conditioning on \xi drawn from ... inside the [] strange. Subscript the expectation with it instead?

        l.148 - P(w) should be P(w|\tilde{M}) unless you make additional assumptions. (true also in (3)) On the other hand, since as far as I can tell everything in the paper conditions on \tilde{M}, you may as well just drop it.

        l.149 - Probabilities are normalized. If you're going to talk about the normalizing constant, write it explicitly rather than using \propto.

        l.150 - Is this the same w? Looks different.

        l.151 - I think you want = rather than \propto. Something is wrong in the integral in (3); check the variable of integration. There are \bar\phi and \tilde\phi and it's not clear what is happening.

        l.158 - "constrained to lie on a hypercube" - Just say this in math so that it's precise.

        l.166 - Can't (4) just go from 1 to N?

        l.228 - I wouldn't open a figure caption with "Our proof-of-concept domains make the unrealistic assumption." Furthermore, I have no idea how the figure caption relates to the figure.

=== UPDATE
The authors have clarified section 4 and I understand much better what they were after. I have updated my score accordingly.


Variable Importance using Decision Trees
Decision trees and random forests are well established models that not only offer good predictive performance, but also provide rich feature importance information. While practitioners often employ variable importance methods that rely on this impurity-based information, these methods remain poorly characterized from a theoretical perspective. We provide novel insights into the performance of these methods by deriving finite sample performance guarantees in a high-dimensional setting under various modeling assumptions. We further demonstrate the effectiveness of these impurity-based methods via an extensive set of simulations.

The article tackles the problem of variable importance in regression trees. The strategy is to select the variables based on the impurity reduction they induce on label Y.  The main feature of this strategy is that the impurity reduction measure is based on the ordering of Y according to the ranking of the X variable under consideration, therefore it measures the  relationship between Y and any variable in a more robust way than simple correlation would. The authors prove that this strategy is consistent (i.e. the true explanatory variables are selected) in a range of settings. This is then illustrated on a simulated example where the results displayed are somewhat the ones one could have expected: the proposed procedure is able to account for monotone but non linear relationships between X and Y so it yields better results than simple correlations. In the same time, it cannot efficiently deal with interaction since explanatory variables are considered one by one - compared with procedures that consider a complete tree rather than only  a stump.
Overall the  idea that is presented is very simple but as mentioned by the authors the theoretical analysis of the strategy is not completely trivial. Although there are some weaknesses in the manuscript the ideas are well presented and the sketchs of proof presented in section 3.1 and 3.2 provide a good lanscape of the complete proof. However there are some weaknesses in the Simulation section, and it is a bit surprising (and disappointing) that there is no discussion section at all.
Regarding the Simulation section, first it seems that the quality criterion that is represented is actually not the criterion under study in section 2 and 3. The theoretical results state the consistency of the procedure, i.e. the probability that the complete set of s predictors is retrieved, so the authors could have displayed the proportion of runs where the set is correctly found. Instead, they display the fraction of the support that is recovered, and it is not clear whether this is done on a single simulated dataset or averaged. If only on a single simulation then it would be more relevant to repeat the experiment on several runs to have a clearer picture of what happens. Another remark is that the authors mention that they consider the model Y=X\beta +f(X_S) + w, but do they consider this model both to generate the data and perform the analysis, or just for the generation and then a purely linear model is used for analysis ? It seems that the second hypothesis is the good one but this should be clearly mentioned. Also it would be nice to have on an example the histogram of the criterion that is optimized by each procedure (when possible) to empirically evaluate the difficulty of the problem.
Regarding the proof itself, some parts of it should be more commented. For instance, the authors derive the exact distribution of \tilde{U}, but then they use a truncated version of it. It may seem weird to approximate the (known) true distribution to derive the final result on this distribution. The authors should motivate this step and also provide some insight about what is lost due to this approximation. Also, is there any chance that the results presented here can be easily extended to the classification case ? And lastly, since the number of active feature is unknown in practice, is there any way the strategy presented here could help estimating this quantity (also I understand it is not the core of the article) ?


Decoding with Value Networks for Neural Machine Translation
Neural Machine Translation (NMT) has become a popular technology in recent years, and beam search is its de facto decoding method due to the shrunk search space and reduced computational complexity. However, since it only searches for local optima at each time step through one-step forward looking, it usually cannot output the best target sentence. Inspired by the success and methodology of AlphaGo, in this paper we propose using a prediction network to improve beam search, which takes the source sentence x, the currently available decoding output y 1 , · · · , y t−1 and a candidate word w at step t as inputs and predicts the long-term value (e.g., BLEU score) of the partial target sentence if it is completed by the NMT model. Following the practice in reinforcement learning, we call this prediction network value network. Specifically, we propose a recurrent structure for the value network, and train its parameters from bilingual data. During the test time, when choosing a word w for decoding, we consider both its conditional probability given by the NMT model and its long-term value predicted by the value network. Experiments show that such an approach can significantly improve the translation accuracy on several translation tasks.

This paper addresses one of the limitation of NMT, the so-called
exposure bias, that results from the fact that each word is chosen
greedily. For this, the authors build on standard technique of
reinforcement learning and try to predict, for each outgoing
transition of a given state, the expected reward that will be achieved
if the system take this transition.
The article is overall very clear and the proposed ideas quite
appealing, even if many of the decisions seem quite ad hoc (e.g. why
doing a linear combination between the (predicted) expected reward and
the prediction of the NMT system and not decoding directly with the
former) and it is not clear whether the proposed approach can be
applied to other NLP tasks.
More importantly, several implementation "details" are not specified.
For instance, in Equation (6), the BLEU function is defined at the
sentence level while in the actual BLEU metric is defined at the
corpus level. The authors should have specified which approximation of
BLEU that have used.
The connection between structure prediction (especially in NLP) and
reinforcement learning has been made since 2005 (see for instance [1],
[2] or, more recently, [3]). It is, today, at the heart of most
dependency parsers (see, among others, the work of Goldberg) that have
deal with similar issues for a long time. The related work section
should not (and most not) be only about neural network works.
The MT system used in the experiments is far behind the performance
achieved by the best systems of the WMT campaign: the best system (a
"simple" phrase-base system as I remember) achieves a BLEU score of
~20 for En -> De and 35 for En -> Fr while the scores reported by the
authors are respectively ~15 and ~30. I wonder if the authors would
have observed similar gains had they considered a better baseline
system.
More generally, most of the reported experimental results are quite
disturbing. Figure 3.a  shows that increasing the beam actually
degrades translation quality. This counter-intuitive results should
have been analyzed. Figure 3.b indicates that taking into account the
predicted value of the expected BLEU actually hurts translation
performance: the less weight the predicted score gets in the linear
combination, the better the translation.
[1] Learning as Search Optimization: Approximate Large Margin Methods for Structured
Prediction, Hal Daumé III and Daniel Marcu, ICML'05
[2] Searched-based structured prediction, Hal Daumé III, John Langford
and Daniel Marcu, ML'09
[3] Learning to Search Better than Your Teacher, Kai-Wei Chang, Akshay
Krishnamurthy, Alekh Agarwal, Hal Daumé III and John Langford


Greedy Algorithms for Cone Constrained Optimization with Convergence Guarantees
Greedy optimization methods such as Matching Pursuit (MP) and Frank-Wolfe (FW) algorithms regained popularity in recent years due to their simplicity, effectiveness and theoretical guarantees. MP and FW address optimization over the linear span and the convex hull of a set of atoms, respectively. In this paper, we consider the intermediate case of optimization over the convex cone, parametrized as the conic hull of a generic atom set, leading to the first principled definitions of non-negative MP algorithms for which we give explicit convergence rates and demonstrate excellent empirical performance. In particular, we derive sublinear (O(1/t)) convergence on general smooth and convex objectives, and linear convergence (O(e −t )) on strongly convex objectives, in both cases for general sets of atoms. Furthermore, we establish a clear correspondence of our algorithms to known algorithms from the MP and FW literature. Our novel algorithms and analyses target general atom sets and general objective functions, and hence are directly applicable to a large variety of learning settings.

%%%% Summary
The authors consider constrained convex optimization problems where the constraints are conic and given by a bounded set of atoms. The authors propose linear oracle based method to tackle such problems in the spirit of Frank-Wolfe algorithm and Matching Pursuit techniques. The main algorithm is the Non-Negative Matching Pursuit and the authors propose several active set variants. The paper contains convergence analysis for all the algorithms under different scenarios. In a nutshel, the convergence rate is sublinear for general objectives and linear for strongly convex objectives. The linear rates involve a new geometric quantity, the cone width. Finally the authors illustrate the relevance of their algorithm on several machine learning tasks and different datasets.
%%%% Main comments
I did not have time to check the affine invariant algorithms and analyses presented in the appendix.
The papers contains interesting novel ideas and extensions for linear oracle based optimization methods but the technical presentation suffer some weaknesses:
- Theorem 2 which presentation is problematic and does not really provide any convergence guaranty.
- All the linear convergence rates rely on Theorem 8 which is burried at the end of the appendix and which proof is not clear enough.
- Lower bounds on the number of good steps of each algorithm which are not really proved since they rely on an argument of the type "it works the same as in another close setting".
The numerical experiments are numerous and convincing, but I think that the authors should provide empirical evidences showing that the computational cost are of the same order of magnitude compared competing methods for the experiments they carried out.
%%%% Details on the main comments
%% Theorem 2
The presention and statement of Theorem 2 (and all the sublinear rates given in the paper) has the following form:
- Given a fixed horizon T
- Consider rho, a bound on the iterates x_0 ... x_T
- Then for all t > 0 the suboptimality is of the order of c / t where c depends on rho.
First, the proof cannot hold for all t > 0 but only for 0 < t <= T. Indeed, in the proof, equation (16) relies on the fact that the rho bound holds for x_t which is only ensured for t <  = T.
Second the numerator actually contains rho^2. When T increases, rho could increase as well and the given bound does not even need to approach 0.
This presentation is problematic. One possible way to fix this would be to provide a priori conditions (such as coercivity) which ensure that the sequence of iterates remain in a compact set, allowing to define an upper bound independantly of the horizon T.
In the proof I did not understand the sentence "The reason being that f is convex, therefore, for t > 0 we have f (x t ) < = f (0)."
%% Lemma 7 and Theorem 8
I could not understand Lemma 7.
The equation is given without any comment and I cannot understand its meaning without further explaination. Is this equation defining K'? Or is it the case that K' can be chosen to satisfy this equation? Does it have any other meaning?
Lemma 7 deals only with g-faces which are polytopes. Is it always the case? What happens if K is not a polytope? Can this be done without loss of generality? Is it just a typo?
Theorem 8:
The presentation is problematic. In Lemma 7, r is not a feasible direction. In Theorem 8, it is the gradient of f at x_t. Theorem 8 says "using the notation from Lemma 7". The proof of Theorem 8 says "if r is a feasible direction". All this makes the work of the reader very hard.
Notations of Lemma 7 are not properly used:
- What is e? e is not fixed by Lemma 7, it is just a variable defining a maximum. This is a recurent mistake in the proofs.
- What is K? K is supposed to be given in Lemma 7 but not in Theorem 8.
- Polytope?
All this could be more explicit.
"As x is not optimal by convexity we have that < r , e >  > 0". Where is it assumed that $x$ is not optimal? How does this translate in the proposed inequality?
What does the following mean?
"We then project r on the faces of cone(A) containing x until it is a feasible direction"
Do the author project on an intersection of faces or alternatively on each face or something else?
It would be more appropriate to say "the projection is a feasible direction" since r is fixed to be the gradient of f. It is very uncomfortable to have the value of r changing within the proof in an "algorithmic fashion" and makes it very hard to check accuracy of the arguments.
In any case, I suspect that the resulting r could be 0 in which case the next equation does not make sense. What prevents the resulting r from being null?
In the next sentences, the authors use Lemma 7 which assumes that r is not a feasible direction. This is contradictory with the preceeding paragraph. At this point I was completely confused and lost hope to understand the details of this proof.
What is r' on line 723 and in the preceeding equation?
I understand that there is a kind of recursive process in the proof. Why should the last sentence be true?
%% Further comments
Line 220, max should be argmax
I did not really understand the non-negative matrix facotrization experiment. Since the resulting approximation is of rank 10, does it mean that the authors ran their algorithm for 10 steps only?


Learning Efficient Object Detection Models with Knowledge Distillation
Despite significant accuracy improvement in convolutional neural networks (CNN) based object detectors, they often require prohibitive runtimes to process an image for real-time applications. State-of-the-art models often use very deep networks with a large number of floating point operations. Efforts such as model compression learn compact models with fewer number of parameters, but with much reduced accuracy. In this work, we propose a new framework to learn compact and fast object detection networks with improved accuracy using knowledge distillation [20] and hint learning [34]. Although knowledge distillation has demonstrated excellent improvements for simpler classification setups, the complexity of detection poses new challenges in the form of regression, region proposals and less voluminous labels. We address this through several innovations such as a weighted cross-entropy loss to address class imbalance, a teacher bounded loss to handle the regression component and adaptation layers to better learn from intermediate teacher distributions. We conduct comprehensive empirical evaluation with different distillation configurations over multiple datasets including PASCAL, KITTI, ILSVRC and MS-COCO. Our results show consistent improvement in accuracy-speed trade-offs for modern multi-class detection models.
drop [3,20,34]. However, those results are shown only for problems such as classification, using simpler networks without strong regularization such as dropout.
Applying distillation techniques to multi-class object detection, in contrast to image classification, is challenging for several reasons. First, the performance of detection models suffers more degradation with compression, since detection labels are more expensive and thereby, usually less voluminous. Second, knowledge distillation is proposed for classification assuming each class is equally important, whereas that is not the case for detection where the background class is far more prevalent. Third, detection is a more complex task that combines elements of both classification and bounding box regression. Finally, an added challenge is that we focus on transferring knowledge within the same domain (images of the same dataset) with no additional data or labels, as opposed other works that might rely on data from other domains (such as high-quality and low-quality image domains, or image and depth domains).
To address the above challenges, we propose a method to train fast models for object detection with knowledge distillation. Our contributions are four-fold:
• We propose an end-to-end trainable framework for learning compact multi-class object detection models through knowledge distillation (Section 3.1). To the best of our knowledge, this is the first successful demonstration of knowledge distillation for the multi-class object detection problem.
• We propose new losses that effectively address the aforementioned challenges. In particular, we propose a weighted cross entropy loss for classification that accounts for the imbalance in the impact of misclassification for background class as opposed to object classes (Section 3.2), a teacher bounded regression loss for knowledge distillation (Section 3.3) and adaptation layers for hint learning that allows the student to better learn from the distribution of neurons in intermediate layers of the teacher (Section 3.4).
• We perform comprehensive empirical evaluation using multiple large-scale public benchmarks.
Our study demonstrates the positive impact of each of the above novel design choices, resulting in significant improvement in object detection accuracy using compressed fast networks, consistently across all benchmarks (Sections 4.1 -4.3).
• We present insights into the behavior of our framework by relating it to the generalization and under-fitting problems (Section 4.4).

Overall
This paper proposes a framework for training compact and efficient object detection networks using knowledge distillation [20]. It is claimed that this is the first successful application of knowledge distillation to object detection (aside from a concurrent work noted in Section 2 [34]), and to the best of my knowledge I confirm this claim. In my view, the main contributions of this work are the definition of a learning scheme and associated losses necessary to adapt the ideas of [20] and [32] for object detection. This requires some non-trivial problems to be solved - in particular, the definition of the class-weighted cross entropy to handle imbalance between foreground and background classes and the definition of a teacher bounded regression loss to guide the bounding box regression. Another minor contribution is the adaptation layer proposed in 3.4 for hint learning. On a skeptical note, these contributions can be viewed as rather straightforward technical adaptations to apply knowledge distillation to a related but novel task, object detection. In this regard, the goal may be considered a low-hanging fruit and the contributions as incremental.
I have two main criticisms to the manuscript. First, I am not convinced that knowledge distillation is particularly well suited to the problem of object detection (see argument below). This leads to my second concern: there are no experiments demonstrating advantages of the proposed model compression method to other model compression approaches.
The manuscript is well written and easy to read, with only a few minor typos. The related work is well structured and complete, and the experiments seem fairly rigorous.
Related Works
+ The related works is well structured and seems to be comprehensive. The related work section even identifies a recent concurrent work with on the same topic and clearly points out the difference in the two approaches.
Approach
- It seems to me that knowledge distillation is not particularly well suited to the problem of object detection where there are only a few classes (and quite often, only foreground and background). One of the main features of [20] is the ability to distill knowledge from an ensemble containing generalist and specialist models into a single streamlined model. The specialist models should handle rare classes or classes that are easily confused, usually subsets of a larger class (such as dog breeds). There is no use of ensemble learning in the proposed approach, and there would be little benefit as there are relatively few classes. My intuition is that this will give much less power to the soft targets and knowledge distillation.
- The problem of class balancing for object detection needs to be more clearly motivated. What are the detection rates from the RPN and the ratio of foreground to background?
Experiments
+ Good experimental design clearly demonstrates the advantages of the method.
+ Important commonly used benchmark datasets are used (PASCAL, MS COCO, ILSVRC)
+ Experiments use several teacher and student models in a variety of configurations, allowing the reader to clearly identify trends (deeper teachers = better performance, bigger datasets = smaller gains)
+ Ablation study justifies the choice of weighted cross-entropy and bounded regression loss functions (note: In my opinion ‘ablation’ is a slight misuse of the terminology, although it is used this way in other works)
-  A comparison against other compressed models is lacking.
- It would have been interesting to see results with a deeper and more modern network than VGG-16 as teacher.
Other comments
L29 - Deeper networks are easier to train - clarify what you mean by this. Problems like vanishing gradients make it difficult to train deeper networks.
L40-41 - This line is unclear. Should you replace ‘degrades more rapidly’ with ‘suffers more degradation’?
L108 - Typos: features, takes
L222 - Typo: Label 1


Riemannian approach to batch normalization
Batch Normalization (BN) has proven to be an effective algorithm for deep neural network training by normalizing the input to each neuron and reducing the internal covariate shift. The space of weight vectors in the BN layer can be naturally interpreted as a Riemannian manifold, which is invariant to linear scaling of weights. Following the intrinsic geometry of this manifold provides a new learning rule that is more efficient and easier to analyze. We also propose intuitive and effective gradient clipping and regularization methods for the proposed algorithm by utilizing the geometry of the manifold. The resulting algorithm consistently outperforms the original BN on various types of network architectures and datasets.

Paper Summary
Starting from the observation that batch-normalization induces a particular
form of scale invariance on the weight matrix, the authors propose instead to
directly learn the weights on the unit-sphere. This is motivated from
information geometry as an example of optimization on a Riemannian manifold, in
particular the Stiefel manifold V(1,n) which contains unit-length vectors.  As
the descent direction on the unit sphere is well known (eq 7), the main
contribution of the paper is in extending popular optimization algorithms
(SGD+momentum and Adam) to constrained optimization on the unit-sphere.
Furthermore, the authors propose orthogonality as a (principled) replacement
for L2 regularization, which is no longer meaningful with norm constraints.
The method is shown to be effective across two families of models (VGG, wide
resnet) on CIFAR-10, CIFAR-100 and SVHN.
Review
I like this paper, as it proposes a principled alternative to batch
normalization, which is also computationally lightweight, compared to other
manifold optimization algorithms (e.g. natural gradient). That being said a few
flaws prevent me from giving a clear signal for acceptance.
(1) Clarity. I believe the paper would greatly benefit from increased
simplicity and clarity in the exposition of the idea. First and foremost,
concepts of element-wise scale invariance and optimization under unit-norm
constraints are much simpler to grasp (especially to people not familiar with
information geometry) than Riemannian manifolds, exponential maps and parallel
transport. I recommend the authors preface exposition of their methods with
such an intuitive explanation. Second, I also found there was a general lack of
clarity about what type of invariance is afforded by BN, and which direction
were the weights norms constrained in (column vs row). On line 75, the authors
write that "w^T x is invariant to linear scaling", when really this refers to
element-wise scaling of the columns of the matrix W \in mathbb{R}^{n x p},
where $p$ is the number of hidden units. In Algorithm 4, there is also a discrepancy
between the product "W x" (implies W is p x n) and the "column vectors w_i"
being unit-norm.
(2) Experimental results. As this is inherently an optimization algorithm, I
would expect there to be training curves. Does the algorithm simply converge
faster, does it yield a better minimum, or both ? Also, I would like to see
a controlled experiment showing the impact of the orthogonality regularizer:
what is the test error with and without ?
(3) Baselines and related work. This work ought to be discussed in the more
general context of Riemannian optimization for neural networks, e.g. natural
gradient descent algorithms (TONGA, K-FAC, HF, natural nets, etc.). Path-SGD also
seems closely related and should be featured as a baseline.
Finally, I am somewhat alarmed and puzzled that the authors chose to revert to
standard SGD for over-complete layers, and only use their method in the
under-complete case. Could the authors comment on how many parameters are in this
"under-complete" regime vs total number of parameters ? Are there no appropriate
relaxations of orthogonality ? I would have much rather seen the training curves
using their algorithm for *all* parameters (never mind test error for now) then
this strange hybrid method.