Deep Dynamical Modeling and Control of Unsteady Fluid Flows
The design of flow control systems remains a challenge due to the nonlinear nature of the equations that govern fluid flow. However, recent advances in computational fluid dynamics (CFD) have enabled the simulation of complex fluid flows with high accuracy, opening the possibility of using learning-based approaches to facilitate controller design. We present a method for learning the forced and unforced dynamics of airflow over a cylinder directly from CFD data. The proposed approach, grounded in Koopman theory, is shown to produce stable dynamical models that can predict the time evolution of the cylinder system over extended time horizons. Finally, by performing model predictive control with the learned dynamical models, we are able to find a straightforward, interpretable control law for suppressing vortex shedding in the wake of the cylinder.

Update: after author response. I think that the authors have done good job in addressing my concerns in their response. Prior to receiving the feedback, my main concern could be summarized as: the paper proposes a modification to an existing method (optimize multi-step prediction error instead of one-step prediction error) in an effort to improve accuracy and stability of the model during open-loop simulations. However, there was no direct comparison to the existing method that was modified, and so in the absence of any theoretical or empirical evidence, it is difficult to say whether the proposed approach is effective. In their response, the authors provided a comparison with Takeishi, which suggests that the modification does indeed (at least in this particular experiment) have the desired effect of improving multi-step prediction accuracy. In light of these results (and with the cost function specified more clearly) I'm happy to revise my 'score' to a 6.   Summary ------- This paper concerns data-driven modeling and control of fluid flows. The dynamics describing such flows are nonlinear, making modeling and control challenging. The approach adopted in this paper is based on spectral analysis of the Koopman operator (a popular approach that has received considerable attention in the literature), which (roughly speaking) models the finite-dimensional nonlinear dynamics with infinite-dimensional linear dynamics. For tractable analysis it is desirable to find a finite-dimensional approximation of these infinite-dimensional dynamics (i.e., a Koopman invariant subspace). The strategy presented in this paper is dynamic mode decomposition (DMD), another popular approach that has received much attention in the literature. The key challenge in DMD is coming up with a basis for the Koopman invariant subspace (i.e., a collection of nonlinear functions referred to as ‘observables’). Here lies the main contribution of the paper: a data-driven way to learn appropriate observables, where each observable function is modeled by a deep neural network (encoder). The resulting finite-dimensional linear model is then used for control design (standard MPC for linear systems). It is observed that the MPC strategy coincides with proportional feedback (i.e. control action proportional to the velocity), *as long as* the velocity used for feedback is measured at a specific point (spatially). This proportional control then constitutes a ‘simple, interpretable’ policy.   Originality ----------- The idea of learning a basis for the Koopman invariant subspace (using deep neural networks) seems very promising. Such an approach was proposed in Takeishi (NIPS ’17, ref [19]). As the authors themselves state, the approach in this paper is a ‘modified form of the training algorithm proposed by Takeishi’, c.f., pg. 2 - this raises the question: how significant are the modifications? In my view, the modifications are rather minor. in the authors’ own words:  ‘One main difference between our algorithm and that proposed by Takeishi et al. is that we force the model to simulate the time evolution of the system during training in a manner that mirrors how the model will be deployed at test time.’ Specifically, the error term for the decoder includes mismatch for both the measured states and the predicted states (where the predicted states are obtained by simulating the linear model forward in time, recursively). While this does not directly effect the estimation of A (which is estimated by least squares) the argument is (presumably) that this indirectly (via the nonlinear function g) leads to better prediction performance (specifically, longer-term stability of predictions, c.f. e.g., Figure 3).  A few comments: i) while it is plausible that this could improve predictive performance, it is not obvious; e.g., this does not directly influence the estimation of A (which is still fit by least squares, i.e., minimization of one-step prediction error), and it is A that affects stability of the long-term predictions. ii) in my view, this is a rather minor modification of the algorithm proposed in Takeishi. iii) further, this appears to be the only difference? (i.e., ‘one main difference’ could more accurately be described as ‘the only difference’?). The other difference is that Takeishi has an additional step to estimate a linear delay embedding (because Takeishi do not assume that the states are observable, unlike the present paper, which makes the stronger assumption that the states are observable).  The material on MPC is entirely standard, given the finite-dimensional linear model.  Quality ------- There are no theoretical claims made in the paper, so quality can only be assessed via the numerical illustrations. The method is evaluated against a 'deep variational Bayes filter'. A natural question is: why not compare performance to the method proposed by Takeishi? After all, the proposed method is a modification of Takeishi, would it not make sense to see if the modification had the intended effect, namely, better long-term predictive performance and stability?   Clarity ------- The paper is fairly well written, however, the main contribution (namely, a method for training deep Koopman dynamical models) is only really presented via Figure 1. It might be desirable to describe the method with more precision, e.g., specifically state the cost functions that are minimized during training (particularly for the encoder/decoder; in fairness, the least squares cost is stated), along with any regularization penalties.   Significance  ------------ I think the paper tackles a highly relevant/significant problem (i.e., learning basis functions for DMD in spectral analysis of the Koopman operator), however, the differences compared to previous work are unclear at best. The modifications to existing algorithms (that of Takeishi) appear minor, and the numerical studies do not evaluate against the method of Takeishi, making it very difficult to say whether the modification is effective or not.  Further comments ---------------- I think numerical comparison with Takeishi is quite important/necessary.


Reinforcement Learning with Multiple Experts: A Bayesian Model Combination Approach
Potential based reward shaping is a powerful technique for accelerating convergence of reinforcement learning algorithms. Typically, such information includes an estimate of the optimal value function and is often provided by a human expert or other sources of domain knowledge. However, this information is often biased or inaccurate and can mislead many reinforcement learning algorithms. In this paper, we apply Bayesian Model Combination with multiple experts in a way that learns to trust a good combination of experts as training progresses. This approach is both computationally efficient and general, and is shown numerically to improve convergence across discrete and continuous domains and different reinforcement learning algorithms.

UPDATE: I have read the authors' response, and it was particularly helpful to me in showing that I misinterpreted some of the reward modeling.  I now understand better that the learning from experts is actually learning potentials, and yes in that case there is a theoretical convergence guarantee, so those parts of my criticism have been addressed.  I do feel there is still an experimental gap here though, because no other approaches from the Bayesian RL or other literature were tested against,a and the number of variants of the current algorithm tested are relatively small and in toy domains.  Summary: The paper describes a Bayesian method for combining the potential functions of multiple experts over time in a reinforcement learning scenario.  A method is given for updating both the posterior probabilities of the potentials and calculating the weighted reward function used in choosing an action.  Empirical results in a gridworld and cart pole are given.  Review:  I think the problem setting the authors have chosen is very rich and poses an interesting problem for a reinforcement learning agent.  I also like that the expert advice is provided in the form of potentials, rather than trajectories or exact reward functions, because it makes the objective clearly defined (maximize the agent’s own reward) and the role of the experts (to speed up exploration) clear as well.  However, I thought the solution wasn’t quite worked out or tested to the level of detail needed for a NIPS paper and comparisons to other Bayesian approaches were not attempted.  Specifically, the paper could be substantially improved based on the following:  The algorithm itself is Bayesian in that given the data already collected, it calculates a posterior on the potential expert potentials.  But the algorithm does not really make use of that knowledge in its action selection.  That is, the weighted potentials do certainly affect exploration, but the algorithm does not take actions to explore *which* expert might be right, which is a hallmark of Bayesian RL algorithms.    A good, and very relevant, example of such an algorithm is the BOSS algorithm from “A Bayesian Sampling Approach to Exploration in Reinforcement Learning” (UAI 2009).  That algorithm has a space of potential models, and performs exploration in such a way that either explicitly refutes some models or follows an optimal strategy.  One could perform similar exploration of the potentials in order to better explore the weight space, and I was surprised to not see a Bayesian RL approach here that did so.  It seems like a comparison to a Bayesian RL algorithm that treats the experts as potential models is needed.  In addition, the empirical results did not cover enough scenarios to be completely convincing.  The current results definitely show the mixed potentials are quickly converging to the performance of a good potential function.  But what if the mixture did not contain a good potential?  Or contained 9 bad functions and 1 good one?  How does the performance of the algorithm vary as that good/bad mixture varies?   If this algorithm is going to be used in larger environments, there will likely be many more experts and potentials, and we need to know how the algorithm will behave under these conditions.  From a theoretical perspective, the paper lacks a rigorous theoretical result proving convergence or ensuring optimal asymptotic behavior.  Without such guarantees, and only the limited experiments, the algorithm does not feel like it has been proven enough.   I also found the result at the end of Section 3.1 confusing.  Doesn’t the result that the true rewards can be found by a weighted combination of the experts require that the value function be in the convex hull of their individual functions?  Why is that assumption not stated outright?  Again, it is unclear how the algorithm will behave with all very incorrect potentials.  Minor points:  The paper makes a number of very general claims about how it can be applied with any exploration policy, any RL algorithm, etc., but then goes on to state there are no theoretical guarantees and provides an implementation algorithm that may only be applicable for model free learning.  Would Rmax work like this?  I think the general claims may be a little overreaching, though I think a large class of algorithms are indeed covered.  Lines 142-145 make reference to equation numbers that do not appear above  The derivation in lines 141-142 is somewhat hard to follow and could be improved by including a numeric example of the computation.


Minimax Estimation of Neural Net Distance
An important class of distance metrics proposed for training generative adversarial networks (GANs) is the integral probability metric (IPM), in which the neural net distance captures the practical GAN training via two neural networks. This paper investigates the minimax estimation problem of the neural net distance based on samples drawn from the distributions. We develop the first known minimax lower bound on the estimation error of the neural net distance, and an upper bound tighter than an existing bound on the estimator error for the empirical neural net distance. Our lower and upper bounds match not only in the order of the sample size but also in terms of the norm of the parameter matrices of neural networks, which justifies the empirical neural net distance as a good approximation of the true neural net distance for training GANs in practice.

This paper considers the minimax estimation problem of neural net distance. The problem originates from the generative adversarial networks (GANs). In particular, the paper established the lower bound on the minimax estimation for neural net distance which seems to the first result of this kind.  The authors then derived an upper bound on the estimation error which matches the minimax lower bound in terms of the order of sample size, and the norm of the parameter matrices. However, there is a gap of $\sqrt{d}$ or $\sqrt{d+h}$ which remains as an open question. The main techniques root from a critical lemma called Le Cam's methods in nonparametric statistics and the recent paper [26].    The upper bound used the Rademacher complexity and the extension of McDiarmid's inequality for unbounded support distributions and sub-gaussian.   Overall, the results are novel and very interesting in the setting of neural net distance. Several comments below:   1. The form of neural net distance is very similar to the MMD work in [26] where the same techniques (Le Cam's method) were used for deriving the minimax lower bound. It would be nice to explain the difference between this study and the techniques used in [26].    2. There is a gap between the minimax lower bound and the upper bound used. Any comments on this?  Is this because  the hypothesis space of deep network is too large for the neural net distance?     3.  The notation $\mathal{F}_{nn}$ for representing the function class of neural network could be confusing since here  $n$ denotes the number of samples    I am quite satisfied with the authors' feedback.  It is a good paper.


Learning Invariances using the Marginal Likelihood
Generalising well in supervised learning tasks relies on correctly extrapolating the training data to a large region of the input space. One way to achieve this is to constrain the predictions to be invariant to transformations of the input that are known to be irrelevant (e.g. translation). Commonly, this is done through data augmentation, where the training set is enlarged by applying hand-crafted transformations to the inputs. We argue that invariances should instead be incorporated in the model structure, and learned using the marginal likelihood, which correctly rewards the reduced complexity of invariant models. We demonstrate this for Gaussian process models, due to the ease with which their marginal likelihood can be estimated. Our main contribution is a variational inference scheme for Gaussian processes containing invariances described by a sampling procedure. We learn the sampling procedure by backpropagating through it to maximise the marginal likelihood.

The paper deals with incorporating invariances in Gaussian processes. Specifically, the authors propose a method for making Gaussian process models less sensitive to transformations (affine transformations, rotations etc). In their set-up, the transformations under consideration are described by a probability distribution and the invariance is achieve by averaging the kernel of the Gaussian process wrt. this distribution of transformations.   For this class of models, both the kernel of invariant functions as well as the posterior distributions of interest are intractable. However, the authors deal with both of these issues using the so-called inter-domain inducing points approach. One major advantage of the proposed method is that the only requirement for the distribution of the transformations is that it can be sampled from. Importantly, the proposed method also allows learning parameters of the transformations from data through the marginal likelihood. The drawback is that they are only able to derive lower bounds for the marginal likelihood of the invariant model for Gaussian likelihoods and logistic likelihoods.  The paper is concluded by applying the proposed method to the MNIST dataset and some of its variations.    The submission appears technically sound and correct. The claims of the paper are not supported by any theoretical analysis, but the paper does contain experimental results for the proposed method showing the benefits of incorporating invariance.  My prime concern is that the only included datasets are MNIST and variations of MNIST and yet, their results still seem quite sensitive. It would be a significant improvement, if the authors could demonstrate the benefit of the model on more complex datasets. Also, it would be interesting if the authors would compare their method against data augmentation.  The paper is in general well-structured, well-written and easy to follow.  Their idea for incorporating invariance in Gaussian processes in original and novel.  Incorporating invariance into probabilistic models is an important problem. Despite the fact that experimental results could be more convincing and it is not possible to do multi-class classification in a proper probabilistic fashion, I still believe that the results are important and that other researchers will most likely build upon and improve their method.    Further comments: Line 26: “...likelihood cannot be used to compare to other model.” typo  Line 23, 36: The following two sentences are contradictions: “Data augmentation is also unsatisfactory from a Bayesian perspective” (line 23) and “We believe this work to be exciting, as it simultaneously provides a justification for data augmentation from a Bayesian point of view” (line 36)   Line 134: “Data augmentation attempts to do the same, only by considering a general set points for each input x.” typo  Line 189: “The intractability stems less the bound than the approximate posterior q(f ), …” Something seems to be missing here   Line 228: How many samples S is needed in practice?  Line 234: “For example, the logistic likelihood can be written as an expectation over a Pòlya-Gamma variable ω [REF]”. Missing reference   Line 275: “However, all invariant models outperform a GP model with simple RBF kernel by a large margin both in terms of error and marginal likelihood (not shown), see Fig. 2 (right).” If the right-most panel does not show the marginal likelihood, then what does it show?  After author feedback ----------------------------- I've read the rebuttal, but it has not changed my opinion: it is a good submission with significant contributions to the field.


Q-learning with Nearest Neighbors
We consider model-free reinforcement learning for infinite-horizon discounted Markov Decision Processes (MDPs) with a continuous state space and unknown transition kernel, when only a single sample path under an arbitrary policy of the system is available. We consider the Nearest Neighbor Q-Learning (NNQL) algorithm to learn the optimal Q function using nearest neighbor regression method. As the main contribution, we provide tight finite sample analysis of the convergence rate. In particular, for MDPs with a d-dimensional state space and the discounted factor 2 (0, 1), given an arbitrary sample path with "covering time" L, we establish that the algorithm is guaranteed to output an "-accurate estimate of the optimal Q-function using e O L/(" 3 (1 ) 7 ) samples. For instance, for a wellbehaved MDP, the covering time of the sample path under the purely random policy scales as e O 1/" d , so the sample complexity scales as e O 1/" d+3 . Indeed, we establish a lower bound that argues that the dependence of e ⌦ 1/" d+2 is necessary.

This work analyzes the sample complexity of a nearest-neighbor Q-learning algorithm. It tackles the case of continuous state space and discrete action space, where discretization of the state space is applied. The paper is well written and relatively easy to follow. The theory and definitions are sound, and the proofs seem to be valid (though I did not verify all of the appendix). I also appreciate the elegancy of the NN Bellman operator in (5). Nonetheless, the work suffers from several weaknesses. Given bellow is a list of remarks regarding these weaknesses and requests for clarifications and updates to the manuscript.  - The algorithm’s O(1/(\esiplon^3 (1-\gamma)^7)) complexity is extremely high. Of course, this is not practical. Notice that as opposed to the nice recovery time O(\epsilon^{-(d+3)}) result, which is almost tight, the above complexity stems from the algorithm’s design. - Part of the intractability of the algorithm comes from the requirement of full coverage of all ball-action pairs, per each iteration. This issue is magnified by the fact that the NN effective distance, h^*, is O(\epsilon (1-\gamma)). This implies a huge discretized state set, which adds up to the above problematic complexity.  The authors mention (though vaguely) that the analysis is probably loose. I wonder how much of the complexity issues originate from the analysis itself, and how much from the algorithm’s design.  - In continuation to the above remark, what do you think can be done (i.e. what minimal assumptions are needed) to relax the need of visiting all ball-action pairs with each iteration? Alternatively, what would happen if you partially cover them? - Table 1 lacks two recent works [1,2] (see below) that analyze the sample complexity of parametrized TD-learning algorithms that has all ‘yes’ values in the columns except for the ‘single sample path’ column. Please update accordingly.  - From personal experience, I believe the Lipschitz assumption is crucial to have any guarantees. Also, this is a non-trivial assumption. Please stress it further in the introduction, and/or perhaps in the abstract itself. - There is another work [3] that should also definitely be cited. Please also explain how your work differs from it. - It is written in the abstract and in at least one more location that your O(\epsilon^{-(d+3)}) complexity is tight, but you mention a lower bound which differs by a factor of \epsilon^{-1}. So this is not really tight, right? If so, please rephrase.  - p.5, l.181: ‘of’ is written twice. p.7, l.267: ‘the’ is written twice.  References: [1] Finite Sample Analyses for TD (0) with Function Approximation, G Dalal, B Szörényi, G Thoppe, S Mannor, AAAI 2018 [2] Finite Sample Analysis of Two-Timescale Stochastic Approximation with Applications to Reinforcement Learning G Dalal, B Szörényi, G Thoppe, S Mannor, COLT 2018 [3] Batch Mode Reinforcement Learning based on the Synthesis of Artificial Trajectories, Raphael Fonteneau, Susan A. Murphy, Louis Wehenkel, and  Damien Ernst, Annals of Operations Research 2013


Uplift Modeling from Separate Labels
Uplift modeling is aimed at estimating the incremental impact of an action on an individual's behavior, which is useful in various application domains such as targeted marketing (advertisement campaigns) and personalized medicine (medical treatments). Conventional methods of uplift modeling require every instance to be jointly equipped with two types of labels: the taken action and its outcome. However, obtaining two labels for each instance at the same time is difficult or expensive in many real-world problems. In this paper, we propose a novel method of uplift modeling that is applicable to a more practical setting where only one type of labels is available for each instance. We show a mean squared error bound for the proposed estimator and demonstrate its effectiveness through experiments.

This paper proposes an approach to heterogeneous treatment effect estimation (what it calls “uplift modeling”) from separate populations. A simple version of the setup of this paper is as follows. We have two populations, k = 1, 2, with different probabilities of treatment conditional on observed features, Pk[T|X] (the paper also allows for the case where these need to be estimated). We have access to covariate-outcome pairs (X, Y) drawn from both populations, so we can estimate Ek[Y|X]. We assume potential outcomes Y(-1), Y(1), and assume that E[Y(T) | X] doesn’t depend on setup k.  What we would really want is to estimate a conditional average treatment effect tau(x) = E[Y(1) - Y(-1) | X = x]. However, we don’t observe assigned treatments for each sample, i.e., we don’t see (X, T, Y); rather we only know marginal treatment probabilities Pk[T|X] along with data (X, Y). The main result of the paper is that we can still identify tau(x) in terms of observable moments,  (A) tau(x) = 2 (E2[Y|X=x] - E1[Y|X=x]) / (E2[T|X=x] - E1[T|X=x]),  and it then derives estimators for tau(x).  Overall, I found this to be a fascinating paper, in that it states a simple, seemingly difficult problem, and pushes quite far in its results. However, it appears that the setup discussed in this paper is closely related to instrumental variables regression, and that the authors have essentially re-derived (fairly sophisticated forms of) two-stage least-squares.  As background, in instrumental variables regression, we usually observe an instrument Z that “nudges” individuals towards treatment; we then observe a tuple (X, Y, T, Z). There is a concern, however, that the treatment T itself may not be unconfounded. The solution, then, is to use the effect of Z on Y to identify the causal effect of T (because we assume that Z nudges on T in a random way), resulting in an identification result:  (B) tau(x) = 2 (E[Y|X=x, Z=2] - E[Y|X=x, Z=1]) / (E[T|X=x, Z=2] - E[T|X=x, Z=1]).  Notice that (A) and (B) are exactly the same, provided we use “setup” in case (A) as our instrument Z in case (B). Several methods for heterogeneous treatment effect estimation via instrumental variables have recently been considered, including  Athey, Susan, Julie Tibshirani, and Stefan Wager. "Generalized random forests." arXiv preprint arXiv:1610.01271 (2016). Lewis, Greg, and Vasilis Syrgkanis. "Adversarial Generalized Method of Moments." arXiv preprint arXiv:1803.07164 (2018). Hartford, Jason, Greg Lewis, Kevin Leyton-Brown, and Matt Taddy. "Deep iv: A flexible approach for counterfactual prediction." In International Conference on Machine Learning, pp. 1414-1423. 2017.  For a review of fundamental results, see  Imbens, Guido W. "Instrumental Variables: An Econometrician’s Perspective." Statistical Science 29.3 (2014): 323-358.  The approach presented here (based on solving a min-max optimization problem) appears to be novel, and potentially interesting, in the context of the existing literature. However, because the paper seems unaware of its nearest neighbors in the literature, it is very difficult to accurately assess its merit.  Thus, although I enjoyed reading this paper, I am now voting for rejection. However, this should really be taken as an encouragement for the authors to re-visit their research question in light of the IV literature, and then resubmit to ICML or JMLR.  Furthermore, I think this paper has potential for considerable impact; but as it’s currently written, it seems to mostly just target a niche problem in marketing, and so would probably not get the readership it deserves even if published at NIPS. Relatedly, for broader scientific impact, I’d recommend the authors use nomenclature around heterogeneous treatment effect estimation rather than uplift (which is, as far as I can tell, a term only used in marketing). Some recent references include:  Hill, Jennifer L. "Bayesian nonparametric modeling for causal inference." Journal of Computational and Graphical Statistics 20.1 (2011): 217-240. Imai, Kosuke, and Marc Ratkovic. "Estimating treatment effect heterogeneity in randomized program evaluation." The Annals of Applied Statistics 7.1 (2013): 443-470. Künzel, Sören R., et al. "Meta-learners for Estimating Heterogeneous Treatment Effects using Machine Learning." arXiv preprint arXiv:1706.03461 (2017). Wager, Stefan, and Susan Athey. "Estimation and Inference of Heterogeneous Treatment Effects using Random Forests." arXiv preprint arXiv:1510.04342 (2015).


Policy-Conditioned Uncertainty Sets for Robust Markov Decision Processes
What policy should be employed in a Markov decision process with uncertain parameters? Robust optimization's answer to this question is to use rectangular uncertainty sets, which independently reflect available knowledge about each state, and then to obtain a decision policy that maximizes the expected reward for the worst-case decision process parameters from these uncertainty sets. While this rectangularity is convenient computationally and leads to tractable solutions, it often produces policies that are too conservative in practice, and does not facilitate knowledge transfer between portions of the state space or across related decision processes. In this work, we propose non-rectangular uncertainty sets that bound marginal moments of state-action features defined over entire trajectories through a decision process. This enables generalization to different portions of the state space while retaining appropriate uncertainty of the decision process. We develop algorithms for solving the resulting robust decision problems, which reduce to finding an optimal policy for a mixture of decision processes, and demonstrate the benefits of our approach experimentally.

The authors consider distributionally robust finite MDPs over a finite horizon.  The transition probabilities conditionally to a state-action pair should remain at L1-bounded distance from a base measure, which is feasible as being generated using a given reference policy.  This is a nice idea. A few comments are mentioned next.  * One aspect I did not understand is how the reference policy should be chosen.  Related to that question, why the requirement of staying "close" to this policy would be beneficial. In safe RL for instance, we could want to remain close to "safe" states, but I can't make the connection with this and the identification of a single "safe" reference policy. In the first experiment, (line 253) the authors use a uniform policy as the reference. In the second experiment, there are only 2 actions and the authors use a state-independent randomized policy (line 301). What is the rationale? Is the choice of a "uniform" reference policy specific to these two problems?  * How should we read the results of Table 1, when p varies ? Is there a value of p that leads to a better policy? In which sense?  * I liked (line 274)  that the authors provided a sensitivity analysis of the stepsize they use to discretize a [0,1]-valued extra state variable.   * What is the exact definition of the softmin in equations (6) and (11)  * There are too few details to understand the benchmark policies such as LR, RECT, MLE to be able to reproduce the experiments (depicted in Figure 1). I would think that those methods have more parameters than those given in the supplementary material.  * (line 225) "converges when the statistics match", but "when statistics match" is unclear, can the authors explain.  * Section 2 introduces the notion of "Directed information theory for processes", but I interpret this as describing the fact that policies should be in the class of progressively measurable processes in discrete-time -- which is a normal math framework for history-dependent policies.  * The significance is somewhat difficult to assess because the test problems seem relatively low-dimensional, which, granted, facilitates benchmarking, but does not allow to demonstrate scalability of the proposed method.


Fast greedy algorithms for dictionary selection with generalized sparsity constraints
In dictionary selection, several atoms are selected from finite candidates that successfully approximate given data points in the sparse representation. We propose a novel efficient greedy algorithm for dictionary selection. Not only does our algorithm work much faster than the known methods, but it can also handle more complex sparsity constraints, such as average sparsity. Using numerical experiments, we show that our algorithm outperforms the known methods for dictionary selection, achieving competitive performances with dictionary learning algorithms in a smaller running time.

This paper proposes an algorithm for the dictionary selection problem.  In this problem the goal is to find a small subset of "atoms" so it possible to approximate the whole data using the selected subset in sparse representation. The proposed algorithm is an extension to "Replacement Greedy", an algorithm for two-stage submodular maximization. This is a similar problem to the paper's problem, however the underline function in this paper is not submodular. The algorithm uses techniques in Orthogonal matching pursuit (OMP). This technique allows them to provably improve the running time of the algorithm and be able to pose global constraints. Experimental results show improvement in runtime and tests residual variance in comparison with other existing algorithms. The technical parts appear to be correct. -In the experiment section, the author used test residual variance to compare between the algorithms. I don’t understand why? Isn't it better comparing the value of the computed subset? After all the goal is to maximize this function. In my opinion, I think the authors should at least explain their choice.  -The algorithm requires the smooth parameter M_{s,2}, this parameter can be bounded. According to the author, this value can be computed in O(n^2 * d) time. This sounds much higher than the run time of this algorithm and other existing algorithms. What am I missing? -The paper is generally well written and constructed. There was one notation that I think could at least be explained better- equation (5) the \_Z_t'\Z_t. It took me a while to understand it. -The p-replacement definition was a little bit confusing. At first I thought it is just a chain of replacement, that in each step approaches (Z_1*...Z_t*). It is possible to write a small explanation before the definition. -An intuition for line 179-180 is needed. "we propose a proxy of this quantity by borrowing an idea from orthogonal matching pursuit".  -The main contribution of this paper, in my opinion, is the extension of OMP, which gives a better runtime and the possibility to pose global constraints.  -The paper also proposes an online variant of their algorithm. This variant is the end of the supplementary, which I didn’t fully verify.  -It seems the paper has shown better  results on the dictionary selection problem both in theory and application.      Other comments and typos:  -Table 1: The approximation ratio of the replacement greedy is from your paper. I think it should be stated somewhere in the caption.  -Typo Eq(3)- You should replace the big T with small t: Z_t' \subset Z_t+a.   -Line 370 in the supplementary: "there exists a bijection"- what is the properties of this bijection? Maybe it is clear from the context but I still think you should add it.   ---Edit---- I have read the author response. The clarification about the upper bounds of M_{s,2}  satisfied me (the second comment). Therefore, I am updating my rating of this paper from 6 to 7.


Moonshine: Distilling with Cheap Convolutions
Many engineers wish to deploy modern neural networks in memory-limited settings; but the development of flexible methods for reducing memory use is in its infancy, and there is little knowledge of the resulting cost-benefit. We propose structural model distillation for memory reduction using a strategy that produces a student architecture that is a simple transformation of the teacher architecture: no redesign is needed, and the same hyperparameters can be used. Using attention transfer, we provide Pareto curves/tables for distillation of residual networks with four benchmark datasets, indicating the memory versus accuracy payoff. We show that substantial memory savings are possible with very little loss of accuracy, and confirm that distillation provides student network performance that is better than training that student architecture directly on data.

Main idea: this paper proposes to tackle the task of speeding up any CNN by replacing the "higher level blocks" that typically consist of stacked convolutions with cheaper blocks consisting of 1x1 convolutions and potentially group convolutions. Thus far this is not super-novel, as this idea has been studied before. However, the authors also employ knowledge distillation (again this idea has been studied before in isolation) to match the inner layer block activations between the teacher and the student (cheaper) networks. This shows very promising results over the baseline which only replaces the blocks with their cheaper version without distillation.   The main selling point of this paper is that the ideas have been proven before independently, and this paper ties them together nicely. Therefore, it should be relatively easy for other researchers to hopefully replicate these results. Another nice plus is that the authors included a segmentation task besides classification. This is important because in the literature most of the work done in speeding up networks seems to be centered around classification, and it's not clear whether such methods generalize beyond this niche domain. In fact, from personal experience, we have also experienced a huge quality degradation when changing the convolutions to be depthwise, but not doing distillation in the context of non-classification problems.  On the negative side, I would have wished to see a discussion about runtime. Most of the focus of the paper is in comparing the number of parameters against the various errors. For this to be truly useful in practice it would be great to discuss how does the number of FLOPs (or forward pass time) change (did it increase?, did it decrease?). I would strongly suggest the authors clarify this in the final version of the paper.  Overall score: I think the paper has some good practical applications, and I think we should accept it.  Confidence: I am not super familiar with the field of making networks faster besides the ideas I mentioned above, and I did do a bit of research on Google Scholar trying to find similar papers that discuss both ideas, but I didn't find anything matching quite exactly (e.g., MobileNets seem to use both distillation and cheaper convolutions but they don't match the internal features).


Approximate Knowledge Compilation by Online Collapsed Importance Sampling
We introduce collapsed compilation, a novel approximate inference algorithm for discrete probabilistic graphical models. It is a collapsed sampling algorithm that incrementally selects which variable to sample next based on the partial compilation obtained so far. This online collapsing, together with knowledge compilation inference on the remaining variables, naturally exploits local structure and contextspecific independence in the distribution. These properties are used implicitly in exact inference, but are difficult to harness for approximate inference. Moreover, by having a partially compiled circuit available during sampling, collapsed compilation has access to a highly effective proposal distribution for importance sampling. Our experimental evaluation shows that collapsed compilation performs well on standard benchmarks. In particular, when the amount of exact inference is equally limited, collapsed compilation is competitive with the state of the art, and outperforms it on several benchmarks.

This paper proposes an approximate inference algorithm that combines SDDs with importance sampling. Previously, SDDs have been used for exact inference, by compiling probabilistic graphical models into a tractable SDD representation for answering queries. However, this compilation may lead to an exponentially large representation. Instead, the authors of this paper propose to interleave the compilation process with sampling assignments for individual variables in order to reduce the SDD and keep its size small.  The partially compiled SDD itself serves as a proposal distribution for sampling.  The algorithm is similar to other importance sampling algorithms, but I thought the way that it used SDDs and sampled variables dynamically was clever. The experiments comparing different variable selection strategies and baseline proposal distributions were useful. It makes sense that the frontier method would work well, since variables that participate in many factors could have their probabilities greatly changed by each factor. In fact, you could design really bad cases where factors contradict the included ones, leading to a proposal distribution that's much worse than uniform. I'd be interested to see additional discussion of limitations and when these methods will work particularly well or poorly compared to other algorithms.  I do not follow the literature on approximate inference closely enough to know if the baselines used are still state-of-the-art and represent the best comparison. SampleSearch and EDBP have been around for a while... is there anything more recent that's worthy of discussion or empirical comparison? That said, both algorithms have performed well on these benchmarks in the past, and the benchmarks are from inference competitions designed specifically to test approximate inference algorithms. In my review, I will assume that the empirical comparison is as strong as reported. Additional discussion of related work (even if it's not compared empirically) would be welcome, as would more explanation of why these inference tasks are meaningful on these datasets and how to interpret the metrics. For example, collapsed inference means that probabilities of very unlikely events can still be estimated. Are there appropriate experiments that could test predictions of rare events?  One suggested piece of related work:    Gogate and Domingos, "Approximation by Quantization," UAI 2011  This work performs exact inference on a model where the parameters have been approximated to make reasoning tractable. I suspect this paper is less general-purpose and less accurate than what is proposed here, but there are some common themes of connecting exact and approximate inference and using representations that support rich structure (algebraic decision diagrams, in this case, instead of SDDs).  (This is not a serious problem -- just a suggestion for the authors which they are free to disregard.)  Overall summary: Clever combination of different inference methods with good empirical results. If a longer version of this paper is ever published, it would help to have even more analysis and context.  After the author response: I've read the author response and the other reviews, and I remain enthusiastic about this paper. The comments by Reviewer 1 and the author response increase my confidence that this paper is sound and state-of-the-art. The comments by Reviewer 3 do not convince me that anything is lacking or inadequate about this paper. I have changed my rating from 7 to 8.


Banach Wasserstein GAN
Wasserstein Generative Adversarial Networks (WGANs) can be used to generate realistic samples from complicated image distributions. The Wasserstein metric used in WGANs is based on a notion of distance between individual images, which induces a notion of distance between probability distributions of images. So far the community has considered 2 as the underlying distance. We generalize the theory of WGAN with gradient penalty to Banach spaces, allowing practitioners to select the features to emphasize in the generator. We further discuss the effect of some particular choices of underlying norms, focusing on Sobolev norms. Finally, we demonstrate a boost in performance for an appropriate choice of norm on CIFAR-10 and CelebA.

Update: After reading the authors' response, my opinion remains that this is a strong paper and I argue for its acceptance. The addition of FID and emphasis of the L^p norms further improve it. I do still suggest that the authors run experiments on at least one other dataset: I agree that more datasets would "contribute relatively little to the reader’s understanding of the proposed method", but they do serve a purpose of convincing readers of the practical effectiveness and versatility of that method. I also don't think the authors need to present the same number of results for each dataset; one experiment per dataset demonstrating improvement seems sufficient. --- The popular WGAN-GP method of GAN training minimizes a loss which is defined in terms of an L^2 ground metric. This paper generalizes that method to arbitrary Banach spaces and in particular explores training WGAN-GPs with Sobolev norms, which are perhaps a more sensible choice for image modeling. Experimentally, the proposed BWGAN achieves very good Inception scores on CIFAR-10.  This work directly addresses an important but under-explored aspect of WGANs, namely the importance of the ground metric. The choice of L^2 was made originally entirely because it was convenient, and prior to now nobody had carefully studied (a) whether the choice of ground metric matters or (b) how to implement WGANs with other ground metrics. In my opinion this paper is a solid contribution toward both.   The theoretical contribution seemed sound to me, but I don't have a very extensive theory background so I defer to the other reviewers' opinions there.  The experiments are generally carefully carried out and convincing. The authors achieve very competitive Inception scores with just a faithful (and careful, judging by the description in the supplementary) reimplementation of the WGAN-GP CIFAR-10 model, plus the method in the paper. An important caveat is that Inception scores are known to vary substantially between different random seeds of the same model hyperparameters [3], but the authors do re-train the model several times from scratch and plot the distribution in Fig 2. The paper could use additional experiments on different datasets and architectures. In particular I'd be very interested in a 128x128 image synthesis result, since it's plausible that the choice of norm is even more important in higher dimensions.  Regarding figure 1, I'm not totally convinced by the visual assessment of samples or Fréchet derivatives. In particular, it's hard to tell whether the derivatives for s=-2 are have relatively greater low-frequency components than s=2, or whether the magnitudes for s=2 are just lower overall. Were the gradients rescaled to the same interval before visualization?  The paper relies heavily on Inception scores for evaluation. This is understandable (they're a de-facto standard) but somewhat fraught: the experiments in this paper amount to evaluating different objectives (the BWGAN losses induced by different metrics) on the basis of their correlation with another objective (Inception score), which seems risky when the ``evaluation'' objective is heuristic. Unlike most other GAN papers, which argue (explicitly or implicitly) that improved scores result from more stable training dynamics, here the improvements are attributed mostly to the perceptual alignment of the loss. As such, it would be good to see at least one other evaluation metric reported; I suggest also reporting FID [1] scores.  As a side result, the authors derive a heuristic for choosing hyperparameters which applies to both their method and WGAN-GP. Their baseline WGAN-GP outperforms the one in the original paper (which was state of the art at the time). It would be ideal to validate the heuristic empirically, for examply by sweeping over hyperparameters and showing that the heuristic achieves scores close to the optimal found by the sweep. In particular, in order for Figure 2 to support the conclusion that the choice of s matters, the other hyperparameters must be set to optimal or near-optimal values, which might themselves vary with s.  The result (in the supplemental) that performance varies substantially between different L^p norms is significant and I feel should be included in the main paper, especially given that they basically matched the Sobolev norms in Inception score. Seeing that result changed my conclusion that Sobolev norms are an especially good choice for image modeling.  The writing is generally clear and thorough. The math is easy to follow and the experiments are described in enough detail that reimplementation shouldn't be hard. It wasn't obvious before fully reading the text that the computation of the Fréchet derivative could be done ``implicitly'' by the trick on L172; the authors might want to highlight this in an algorithm box somewhere. The authors do a good job of providing background and placing their work within a larger context.  Overall this paper explores an important topic, presents sound theory and a simple method, shows convincing (if slightly limited) experimental results, and is written clearly. Therefore I recommend acceptance.


Explaining Deep Learning Models -A Bayesian Non-parametric Approach
Understanding and interpreting how machine learning (ML) models make decisions have been a big challenge. While recent research has proposed various technical approaches to provide some clues as to how an ML model makes individual predictions, they cannot provide users with an ability to inspect a model as a complete entity. In this work, we propose a novel technical approach that augments a Bayesian non-parametric regression mixture model with multiple elastic nets. Using the enhanced mixture model, we can extract generalizable insights for a target model through a global approximation. To demonstrate the utility of our approach, we evaluate it on different ML models in the context of image recognition. The empirical results indicate that our proposed approach not only outperforms the state-of-the-art techniques in explaining individual decisions but also provides users with an ability to discover the vulnerabilities of the target ML models.

I think the rebuttal is prepared very well. Although the assumption of a single component approximating the local decision boundary is quite strong, the paper nonetheless offers a good, systematic approach to interpreting black box ML systems. It is an important topic and I don't see a lot of studies in this area.  Overview  In an effort to improve scrutability (ability to extract generalizable insight) and explainability of a black box target learning algorithm the current paper proposes to use infinite Dirichlet mixture models with multiple elastic nets (DMM-MEN) to map the inputs to the predicted outputs. Any target model can be approximated by a non-parametric Bayesian regression mixture model. However, when samples exhibit diverse feature correlations, sparsity, and heterogeneity standard non-parametric Bayesian regression models may not be sufficients. The authors integrate multiple elastic nets to deal with diverse data characteristics. Extensive experiments are performed on two benchmark data sets to evaluate the proposed method in terms of scrutability and explainability. For scrutability a bootstrap sample of the training set is used to replace the most important pixels (as determined by the proposed algorithm) in images with random values. The intuition behind this exercise is that if the selected pixels are indeed strong features then the classification accuracy should suffer significantly and the degree of impact should outweigh other scenarios with randomly manipulated features. This intuition is demonstrated to be correct on both data sets. For explainability the proposed method is compared against two other methods from the literature (LIME and SHAP). The proposed method can identify parts of images that contribute to decisions with greater level accuracy than the other two techniques. Results suggest that the proposed technique can also be used to identify anomalies (inconsistent predictions of the target system) with some level of success.     Technical Quality  I would consider the technical quality to be good with some reservations.    The main idea relies on the assumption that one component in the mixture model can be sufficient to explain a single prediction. What happens if a single component cannot approximate the local decision boundary near a single instance with an acceptable accuracy? In reality one would expect that many Normal components might be needed to explain a single prediction. So using a single component can make the prediction uninterpretable. What aspect of the model guarantee or stipulate sparse mixture components?   The motivation behind using Orthant Gaussian prior on regression coefficients is not well justified. Does this really serve for its purpose (data dimensionality and heteregoneity)? What would happen if standard Gaussian prior was used as a prior?  I also do not follow the need for truncation when the finite number of components can be easily determined during inference without any truncation especially when MCMC inference is used.      Clarity  The paper reads well with some minor issues outlined below (list is not exhaustive).  Line 48 Most of the works that related to, "that" here is redundant Line 81 any learning models, "model"  Line 320 when it comes specific datasets, "to" missing Line 320 it dose not affect, "does"    Originality  I would consider both the method (Dirichlet mixture models with multiple elastic nets) and the approach (evaluation in terms of scrutinability and explainability) quite original.  Significance:  I expect the significance of this work to be high as there is a dire need in the ML literature for models that can make outputs of complex learning systems more interpretable. Toward achieving this end the current paper proposes a systematic approach to explain and scrutinize the outputs of deep learning classifiers with promising results.     Other Comments:  Please define PCR (Principal Component Regression) and explain why PCR is used as opposed to classification accuracy


TADAM: Task dependent adaptive metric for improved few-shot learning
Few-shot learning has become essential for producing models that generalize from few examples. In this work, we identify that metric scaling and metric task conditioning are important to improve the performance of few-shot algorithms. Our analysis reveals that simple metric scaling completely changes the nature of few-shot algorithm parameter updates. Metric scaling provides improvements up to 14% in accuracy for certain metrics on the mini-Imagenet 5-way 5-shot classification task. We further propose a simple and effective way of conditioning a learner on the task sample set, resulting in learning a task-dependent metric space. Moreover, we propose and empirically test a practical end-to-end optimization procedure based on auxiliary task co-training to learn a task-dependent metric space. The resulting few-shot learning model based on the task-dependent scaled metric achieves state of the art on mini-Imagenet. We confirm these results on another few-shot dataset that we introduce in this paper based on CIFAR100.

** After rebuttal   Thank you for the author response. I have read the rebuttal and will maintain my score. The submission and rebuttal motivate three orthogonal improvements (metric scaling, task-conditioning, auxiliary task co-training) to Prototypical Networks. The paper would be much better written as three distinct units that evaluate the three components separately and in more depth.  The mathematical analysis section of the paper consists of writing out the gradient of the objective function with the additional temperature parameter. Although this provides intuition **about the limiting cases of \alpha**, I would hesitate to call this a significantly "non-trivial" derivation. I also recommend that the authors provide empirical validation (e.g., plotting the contribution of \alpha-transformed gradients) for the non-limiting cases of \alpha in order to validate the intuition that varying \alpha in non-limiting cases trades off between "minimizing the overlap of the sample distributions or correcting cluster assignments sample-wise" (rather than just providing the final test performance as in Figure 3).  I also do not agree that the generalization to a learned temperature parameter is exactly analogous to the development of Prototypical Networks from Matching Networks: While the Prototypical Network is itself a slight modification to the Matching Network, it touches on the important tradeoff between parametric and non-parametric density estimation (e.g., see the section on mixture density estimation in the Prototypical Networks paper). I do not see substantial evidence for an analogous claim that the form of the gradient with the additional learnable temperature parameter "has implications for future work in few-shot learning (FSL) and metric learning more generally." The submission needs more argumentation in order to convince the reader that the derivation would "drive algorithm design and empirical studies" more than the provided study that varies \alpha.  Lastly, while it is nice to see evaluation on more datasets, the episodic training setup of CIFAR-100 is not sufficiently different from the few-shot miniImageNet dataset to be considered a major contribution to the community.  ** Before the rebuttal  This work proposes improvements to the prototypical network approach to few-shot classification introduced in [25]. The improvements are three-fold: (i) the addition of a learned temperature parameter in the computation of the softmax-normalized centroid distance; (ii) a modification to conditional normalization [16, 17] in which the conditioning is done as a function of the task-specific dataset as a whole; and (iii) an auxiliary prediction task implemented as an additional head in the architecture that outputs the logits for a 64-way classification task.  The method gives better performance on the standardized miniImageNet benchmark as well as on a few-shot split of CIFAR100 introduced by the authors.  QUALITY  The theoretical contribution of this paper is to write out the gradient of the objective function from [25] with the addition of the temperature scaling parameter in the softmax computation.  The limiting cases (zero and infinite temperature)  are discussed for intuition even though they do not occur in practice as I assume the temperature parameter is transformed to be positive finite (although this is not stated).  There are some flaws in quality throughout the paper:  Several technical claims / hypotheses made in the paper are not evidenced: "Neural network initialization and batch norm encourages this regime." "If Df is large, the network may have to work outside of its optimal regime to be able to scale down the feature representation." "We hypothesize that the importance of TEN is not uniformly distributed over the layers in the feature extractor. The lower layers closer to the similarity metric need to be conditioned more than the shallower layers closer to the image because pixel-level features extracted by shallow level layers are not task specific."  A conceptual point is made in the conclusion unnecessarily: it does not appear that the fact that "the scaled cosine similarity...does not belong to the class of Bregman divergences" is used anywhere in the paper, yet it is mentioned in the conclusion.  CLARITY   The paper is reasonably clear. Some points of are omitted: - Why not report results with temperature scaling and matching networks (instead of prototypical networks)? - What is the architecture of the TEN? What motivated this architecture (specifically, why use the centroids, which enable parameter sharing amongst the feature extractors, but still requires the TEN with additional parameters)? - What is the reason that the TEN "underperformed"? Was it overfitting?  The experimental comparison in and the discussion of Table 1 does not identify the differences in architectural complexity between the reported methods (i.e., [1, 14, 15] and the proposed work employ a ResNet architecture while the other methods employ a shallow convolutional network).  ORIGINALITY  The architecture and algorithm are a combination of previously proposed methods (see "SIGNIFICANCE" below).  The problem setup is not novel, although the authors apply the few-shot episode generation procedure of [30] to the CIFAR100 dataset.  SIGNIFICANCE  The work is incremental as a combination of previously proposed methods applied to prototypical networks for the task of few-shot classification. In particular: - "Metric scaling" is the addition of a learnable temperature parameter to the normalized distance computation in the regime of Matching Networks [30] or Prototypical Networks [25]. - "Task conditioning" makes use of a task-dataset-conditioning network to predict the scale and offset parameters of batch normalization layers as in [3, 16, 17].  - Auxiliary tasks are known to be beneficial to few-shot learning [*,**] and learning in general.  Moreover, I disagree with the argument (in the conclusion section) that the task sampling technique should be modified for improved performance on a few-shot task, as we should be hesitant about tuning the dataset (which is a feature of the problem) to an algorithm.  SPECIFIC COMMENTS  pg. 1: "...one aims to...learn a model that extracts information from a set of labeled examples (sample set)..." Introduce the terminology "support set" alongside "sample set". I believe the restriction to the labelled support/unlabelled query setting is not representative of recent works in few-shot learning; e.g., consider [***, ****], which deal with learning with unlabelled data in the support set.  pg. 1: It is strange to introduce few-shot learning with Ravi & Larochelle as the first citation, then to claim that the problem has subsequently been reframed by Ravi & Larochelle as meta-learning -- they are the same paper! This needs to be rewritten to correctly capture the nuanced difference between few-shot and meta-learning.  pg. 1: The claim that certain approaches to few-shot learning and meta-learning are "influential" and "central to the field" is subjective and   pg. 1: "a feature extractor (or more generally a learner)" A feature extractor is more general than a neural network with learned parameters, so this relationship needs to be reversed. Since you consider models with learned parameter, it would be sufficient to rewrite this as "a feature extractor with learned parameters."  line 57: "parameterized by \phi, mapping x to z, a representation space of dimension D_z" z is an element of the representation space, not the representation space itself.  line 59: "can directly be used to solve the few-shot learning classification problem by association" This needs a more thorough explanation and a more thorough description of the differences between [30] and [25]. In particular, the training criterion for [25] is discussed in lines 62-63 but [30]'s is not.  line 60: "proposed to introduce inductive bias" Framing the computation of the class centroid as an "inductive bias" is not useful in this context unless it is identified why it is a useful inductive bias.  line 61: "a unique feature representation" The mean of embeddings is not necessarily unique.  line 61: "for each class k" It is confusing to use k to index classes when above K have been used to count examples in each class.  line 77-8: "This is the case of Matching Networks [30], which use a Recurrent Neural Network (RNN) to accumulate information about a given task." The vanilla version of MNs does NOT use an RNN; only the "full-context embedding" version requires it.  line 108-109: "We observed that this improvement could be directly attributed to the interference of the different scaling of the metrics with the softmax." This needs an experimental result, and so likely does not belong in the "Model Description" section.  lines 224: "We re-implemented prototypical networks..." Do the experimental results remain the same when employing the authors' original code (https://github.com/jakesnell/prototypical-networks) with the addition of the temperature scaling parameter?  [*] Alonso, Héctor Martínez, and Barbara Plank. "When is multitask learning effective? Semantic sequence prediction under varying data conditions." arXiv preprint arXiv:1612.02251 (2016). [**] Rei, Marek. "Semi-supervised multitask learning for sequence labeling." arXiv preprint arXiv:1704.07156 (2017). [***] Finn, Chelsea, Tianhe Yu, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. "One-shot visual imitation learning via meta-learning." In CoRL, 2017. [****] Metz, Luke, Niru Maheswaranathan, Brian Cheung, and Jascha Sohl-Dickstein. "Learning Unsupervised Learning Rules." arXiv preprint arXiv:1804.00222 (2018).


Efficient Loss-Based Decoding on Graphs for Extreme Classification
In extreme classification problems, learning algorithms are required to map instances to labels from an extremely large label set. We build on a recent extreme classification framework with logarithmic time and space [19], and on a general approach for error correcting output coding (ECOC) with loss-based decoding [1], and introduce a flexible and efficient approach accompanied by theoretical bounds. Our framework employs output codes induced by graphs, for which we show how to perform efficient loss-based decoding to potentially improve accuracy. In addition, our framework offers a tradeoff between accuracy, model size and prediction time. We show how to find the sweet spot of this tradeoff using only the training data. Our experimental study demonstrates the validity of our assumptions and claims, and shows that our method is competitive with state-of-the-art algorithms.

1. Summary  The paper concerns a problem of extreme classification. The authors provide an in-depth analysis of the recently proposed LTLS algorithm. They reformulate the problem in the framework of loss-based decoding for ECOC. Based on their theoretical insights they introduce valid extensions to the original algorithm. They discuss the trade-off between the computational and predictive performance, which is then validated experimentally. The introduced W-LTLS obtains promising results in the experiments.  2. Relation to previous work  The authors build their work on the LTLS algorithm and the loss-based decoding for ECOC. Both are very nicely introduced in a formal way at the beginning of the paper. The authors also include a section with a more general overview of related work.   Below I give only one minor remark in this regard. Since the authors also discuss the tree-based methods in the related work, the following algorithms (with logarithmic complexity)  should be mentioned: - Hierarchical softmax ("Hierarchical probabilistic neural network language model", AISTATS 2005) and FastText ("Bag of tricks for efficient text classification", CoRR 2016) - Conditional probability estimation trees ("Conditional probability tree estimation analysis and algorithms", UAI 2009) - Classification Probability Trees (the generalization of the above) ("Consistency of probabilistic classifier trees", ECMLPKDD 2016) - Filter trees ("Error- correcting tournaments", ALT 2009) - Probabilistic label trees ("Extreme F-measure maximization using sparse probability estimates", ICML 2016)  3. Quality:  3.1 Strengths  This is a solid submission that extends previous work. The new algorithm has been analyses from theoretical and empirical point of view. The authors also try to establish in a formal way the trade-off between predictive accuracy and computational complexity. This is one of the first steps into a general reduction framework for this kind of trade-off.  4. Clarity  4.1 Strengths  The paper is well and clearly written. The appendix contains almost all necessary elements for which there were no place in the main text.  4.2 Weaknesses  The empirical results should also be given in tabular form with numeric values, not only in the form of plots. This is important for accurate comparison of algorithms in the future references.  5. Originality:  5.1 Strengths  The algorithm and the theoretical results are original.  6. Significance:  6.1 Strengths  The work is significant as it will push the research in the desired direction of establishing the framework for the trade-off between predictive and computation performance.  7. Minor remarks:  References: Reference 1 has been doubled, please check also other references as they do not contain all relevant information.  8. After rebuttal  I thank the authors for their rebuttal. I do not change my evaluation, but give one additional comment that could be useful for the authors.  As the paper compares W-LTLS to LomTree and FastXML, it could also include the comparison to a label tree approach being either represented by HSM, PLT, or Parabel. All these algorithms are very similar to each other. PLT and Parabel are multi-label generalizations of HSM. Parabel additionally uses a specific tree building strategy that gets very good results for multi-label problems. To compress the model size in label tree approaches one can use a simple shallow network as it is done in the FastText implementation.


Autowarp: Learning a Warping Distance from Unlabeled Time Series Using Sequence Autoencoders
Measuring similarities between unlabeled time series trajectories is an important problem in domains as diverse as medicine, astronomy, finance, and computer vision. It is often unclear what is the appropriate metric to use because of the complex nature of noise in the trajectories (e.g. different sampling rates or outliers). Domain experts typically hand-craft or manually select a specific metric, such as dynamic time warping (DTW), to apply on their data. In this paper, we propose Autowarp, an end-to-end algorithm that optimizes and learns a good metric given unlabeled trajectories. We define a flexible and differentiable family of warping metrics, which encompasses common metrics such as DTW, Euclidean, and edit distance. Autowarp then leverages the representation power of sequence autoencoders to optimize for a member of this warping distance family. The output is a metric which is easy to interpret and can be robustly learned from relatively few trajectories. In systematic experiments across different domains, we show that Autowarp often outperforms hand-crafted trajectory similarity metrics.

summary: 6/10 This paper proposes a warping procedure in an unsupervised manner. Specifically, it has two separate stages. First it uses an auto encoder to learn a latent embedding for trajectories. Second, it uses a conventional metric betaCV and optimizes the average distance of trajectories within the same cluster determined by their latent representations. A few experiments were conducted on both synthetic and real data. The results can be more convincing.  quality: 5/10 There are places which are a bit hand wavy. For example, in the `Warping distances` section, it is mentioned `distances are generally well-suited to trajectories, this serves to regularize the process of distance metric learning, and generally produces better distances than using the latent representations directly `. It is nice to investigate/elaborate this as one natural questions is why not simply use the Euclidian distance induced by the learned latent representation as the distance metric but still using the second stage?  A proposition is provided suggesting the latent betaCV may be a reliable metric for the ground truth. But it would be nice if this proposition has any practical indications, e.g., how long the trajectory length needs to be for a set of trajectories with C clusters. As in the current work, there’s not much insight from the proposition.  3 experiments were conducted: 1 synthetic, 1 real, 1 semi-real (real data injected with noise). However, there are some questionable parts about each of the experiment. In the synthetic data, only additive noise and outliers are added; sampling rate is not mentioned but this is a main motivating point for this work. Quantitively addressing the robustness to sampling rate should be an important justification. In the real data, the metric is a bit heuristic, i.e., the closeness within same clusters varying with number of clusters. There are only 500 trajectories in total and tens of clusters. It is not convincing how strong this metric correlates with the quality of the warping procedure. Not to mention SSPD is better than Autowarp when number of neighbors is relatively small, which is a more reasonable regime to care about (e.g., <50).  In the semi-real data where noise is injected. However, clearly those injected noise follow the model assumptions used in the proposed procedure, which is an unfair comparison for other procedures. This makes this ‘real data’ experiment much less convincing.   clarity: 8/10 This paper is in general clearly presented.   originality: 7/10 It is nice that the distance metrics are summarized with a common parametrization. The tools (auto encoders, optimization method) are conventional, but they are used in this context in an interesting way.  significance: 7/10 Comparing time series is an important problem. If the proposed method is investigated in a more rigorous way, it can be more impactful.  Update: The authors addressed my concerns reasonably well, though the experiments can be much stronger than presented in the paper.


The promises and pitfalls of Stochastic Gradient Langevin Dynamics
Stochastic Gradient Langevin Dynamics (SGLD) has emerged as a key MCMC algorithm for Bayesian learning from large scale datasets. While SGLD with decreasing step sizes converges weakly to the posterior distribution, the algorithm is often used with a constant step size in practice and has demonstrated successes in machine learning tasks. The current practice is to set the step size inversely proportional to N where N is the number of training samples. As N becomes large, we show that the SGLD algorithm has an invariant probability measure which significantly departs from the target posterior and behaves like Stochastic Gradient Descent (SGD). This difference is inherently due to the high variance of the stochastic gradients. Several strategies have been suggested to reduce this effect; among them, SGLD Fixed Point (SGLDFP) uses carefully designed control variates to reduce the variance of the stochastic gradients. We show that SGLDFP gives approximate samples from the posterior distribution, with an accuracy comparable to the Langevin Monte Carlo (LMC) algorithm for a computational cost sublinear in the number of data points. We provide a detailed analysis of the Wasserstein distances between LMC, SGLD, SGLDFP and SGD and explicit expressions of the means and covariance matrices of their invariant distributions. Our findings are supported by limited numerical experiments.

Review after rebuttal:  I thank the author(s) for their response. While I still believe that this paper is a minor increment beyond what has already been done on SGLD, I agree that the message might be useful for some.  I also appreciate the effort the authors have made in improving the manuscript based on reviews' suggestions, particularly their efforts to include relevant numerical experiments to ML scenarios, and recommendations beyond the CV approach which has been studied to exhaustion and rarely applicable in practice.  Based on this, I've adjusted my decision to marginally above threshold.  Original review: In the paper "The promises and pitfalls of Stochastic Gradient Langevin Dynamics" the authors revisit the Stochastic Langevin Gradient Dynamics (SGLD) approach to approximately sampling from a probability distribution using stochastic gradients (specifically subsampling).   The authors compare a number of different classes of approximate inference method, including SGLD, LMC (known by some as Unadjusted Langevin Algorithm or ULA) and Stochastic Gradient Langevin Dynamics Fixed Point (SGLDFP) -- the latter being a variant of SGLD with a control variate exploiting the unimodality of the distribution, similar to what has been presented in [3, 25 and others].  In the usual context of strong log-concave models, geometric convergence can be readily shown for all these algorithms via synchronous coupling arguments (Lemma 1)  The authors investigate the asymptotic and non-asymptotic behaviour of these processes in the big-data setting (N = data size), under the frequently adopted assumption that the step size is proportional to 1/N and the mini-batch size is fixed independently of N.  T  They show that while the invariant measures for LMC and SGLDFP will converge to each other and the target distribution in Wasserstein-2 with rate O(N^{-1/2}),   SGLD and SGD will converge to each other O(N^{-1/2}).  They demonstrate an example (Bayesian linear regression) in which the distance to the invariant measure remains strictly bounded above from 0 as N -> infinity.    To further understand the behaviour, the authors then obtain asymptotic formula for the first two moments of each respective invariant distribution -- this makes evident the bias with respect to the correct moments for SGD and SGLD.  Finally, toy numerical experiments demonstrate the theory they have just described.  Firstly, I commend the authors for being so precise. I found to be very clearly written.  In my opinion, the main novelty in this paper is the demonstration that the equilibrium behaviour of (1) SGLD and SGD are so similar as N -> infinity and (2) LMC/ULA approaches the correct distribution as N->\infty (albeit this could have already been seen from other works).  While the analysis is illuminating, the scope of applicability is quite limited, the assumption of strongly log-concave models does not put us too far from the Gaussian case, and has already been studied very well following the familiar lines of  [A. Durmus and E.Moulines. High-dimensional Bayesian inference via the Unadjusted Langevin]  [Dalalyan, Arnak S., and Avetik G. Karagulyan. "User-friendly guarantees for the Langevin Monte Carlo with inaccurate gradient." arXiv preprint arXiv:1710.00095 (2017).] [Dalalyan, Arnak S. "Further and stronger analogy between sampling and optimization: Langevin Monte Carlo and gradient descent." arXiv preprint arXiv:1704.04752 (2017).] and various other papers.  Indeed, the proofs in the supplementary material follow almost identically similar proofs in these papers.    The conclusion of this paper is that SGLD diverges from the correct distribution in the large N limit and to use control variates with SGLD.  While the precise results differ, very similar observations using different approaches have been made in other papers, as the authors have mentioned themselves.  My strongest criticism is that I feel that other avenues could have been addressed beyond introducing control variates which is now well established and which are unlikely to work beyond unimodal distributions, and thus strongly limited their usage in the ML community.  For example, would another scaling of step size with N have fixed this asymptotic bias, or perhaps scaling the mini-batch size p with N -- indeed it appears that perhaps choosing p \propto N^{alpha} for some alpha > 0 would also be a solution?  A negative or positive statement along these lines could have been easily considered in this work and provided additional useful recommendations to a practitioner.  Moreover, given the typical style of NIPs papers I also believe the numerical experiments would have provided an opportunity to demonstrate the theory presented in a more realistic context outside the assumptions given.   Perhaps demonstrating examples where the strong log-concavity assumption does not hold, or the model has multiple modes.  In conclusion, while I believe this paper is clearly and precisely written,  given the plethora of previous works investigating very similar ideas using very similar methods, and given the limited applicability of the conclusions for typical problems in ML, I would recommend that perhaps this work is not appropriate for NIPs in its current form, but would strongly recommend it be submitted to a more appropriate journal/conference.    Finally, as a general comment that has no bearing on my decision on this paper,  I find it curious that the authors claim that "SGLD has demonstrated spectacular successes in machine learning tasks".  I am curious evidence this is based on -- I stand to be corrected but I have yet to see SGLD being used or applied in practice.


Query Complexity of Bayesian Private Learning
We study the query complexity of Bayesian Private Learning: a learner wishes to locate a random target within an interval by submitting queries, in the presence of an adversary who observes all of her queries but not the responses. How many queries are necessary and sufficient in order for the learner to accurately estimate the target, while simultaneously concealing the target from the adversary? Our main result is a query complexity lower bound that is tight up to the first order. We show that if the learner wants to estimate the target within an error of ✏, while ensuring that no adversary estimator can achieve a constant additive error with probability greater than 1/L, then the query complexity is on the order of L log(1/✏) as ✏ ! 0. Our result demonstrates that increased privacy, as captured by L, comes at the expense of a multiplicative increase in query complexity. The proof builds on Fano's inequality and properties of certain proportional-sampling estimators.

- Summary & Strength  The paper studies a sequential learning with binary feedback with a privacy concern where the learning objective is to locate a latent value with some confidence while hiding it from a spying adversary who observes all the queries but not the responses. The authors obtain a tight fundamental limit of the query complexity using Fano’s inequality, which is a standard tool for the lower bound analysis. The fundamental limit matches the query complexity of an upper bound algorithm (Replicated Bisection strategy) in the asymptotic order which is originally proposed in [11, 15]. The query complexity analysis shows a privacy-efficiency trade-off. The proof seems sound, and the paper is easy to follow.   - Weakness & Comment  The proposed model is somewhat artificial. In particular, all the responses are assumed to be correct, whereas noisy response is common in practice. Hence, it limits to connect the considering problem to a motivating example in practice. In addition, there is a line of works e.g., [BA 2008, 13], that study the query complexity even under noisy responses (which lead to interesting applications) although there is no privacy concern. However, there is no justification or motivation on the noiseless response.   - Minor comments  I’m not sure that the notion of the random seed Y is necessary.  Line 34: identity function > indicator function Line 44: $R^i := \emptyset$ > $R^1 := \emptyset$ Line 46: “guess’ > “guess” Line 61: any an adversary > any adversary Line 163: $L \log (1/\epsilon)- L (\log L -1)$ > $L \log (1/\epsilon)- L (\log L -1) - 1$ Eq. 5.7: ${\cal E}_{i, j}$ is used without defining it.  -- [BA 2008]: Ben-Or, Michael, and Avinatan Hassidim. "The bayesian learner is optimal for noisy binary search (and pretty good for quantum as well)." Foundations of Computer Science, 2008. FOCS'08. IEEE 49th Annual IEEE Symposium on. IEEE, 2008.


How SGD Selects the Global Minima in Over-parameterized Learning: A Dynamical Stability Perspective
The question of which global minima are accessible by a stochastic gradient decent (SGD) algorithm with specific learning rate and batch size is studied from the perspective of dynamical stability. The concept of non-uniformity is introduced, which, together with sharpness, characterizes the stability property of a global minimum and hence the accessibility of a particular SGD algorithm to that global minimum. In particular, this analysis shows that learning rate and batch size play different roles in minima selection. Extensive empirical results seem to correlate well with the theoretical findings and provide further support to these claims.

This paper examines, theoretically and empirically, how does SGD select the global minima it converges to. It first defines two properties ("sharpness" and "non-uniformity") of a fixed point, and how these determine, together with batch size, the maximal learning rate in which the fixed point is stable under SGD dynamics (both in mean and variance). It is then demonstrated numerically how these results relate affect the learning rate and batch size affect the selection of minima, and the dynamics of "escape" from sharp minima".   Clarity: This paper is nicely written, and quite clear. Quality: Seems correct, except some fixable errors (see below), and the numerical results seem reasonably convincing.  Originality: The results are novel to the best of my knowledge. Significance: The results shed light on the connections between sharpness, learning rate, batch size, and highlight the importance of "non-uniformity". These connections are not well understood and have received attention since the recent work by Keskar et al. From the empirical side, I especially found it interesting that GD selects solutions with maximal sharpness and the strong correlation between sharpness and non-uniformity. I also agree with the authors that the fast escape from sharp minima (e.g., Figure 7, left column) favors this paper's "instability" explanation over the SDE interpretations (e.g., Jastrzebski et al.).  Main Comments: 1) The linearization x_{t+1}=x_t - \eta A_{\xi_t}x_t is wrong: the zeroth order term G(0;\xi) is not necessarily zero (x^* is only a fixed point of G(x), not G(x;\xi)). In other words we need to keep constant term  G(0;\xi) in the dynamics.  These constants do not affect the dynamics of E[x_t] (since their mean is zero), but they do affect E[x^2_t] . This implies that the variance generally does not go to zero. An indeed, SGD generally doesn't converge to a fixed point without decreasing the learning rate to zero.  A way to fix this would be to assume we converge to a global minima where \nabla G(x;\xi) for all \xi . This seems reasonable in overparameterized neural networks.  2) The numerical results focus on quadratic loss, since the authors claim that the Hessian vanishes for cross-entropy. This indeed happens, but only for minima at infinity. However, if we use cross-entropy with L2 regularization (as is commonly done), then all minima are finite and the Hessian would not vanish.  Therefore, I'm not sure I understand this argument. Do the reported results also hold for cross-entropy with L2 regularization?  Minor Comments: 1) I think eqs. 13 and 14 would be more helpful before eq. 8 (with A_i=H_i) than in their current location. 2) It would be helpful to see the derivation of eq. 6 in the appendix.  3) Figure 3 could use larger fonts (especially in labels) 4) Figure 6(a): "2/\eta" should not be there, right? 5) I recommend the authors to make their code available online to enable other to easily reproduce their results.  Some Typos: line 217 - "the extend" line 224 - "is easily" line 292 - "generlizes" equation below line 314 - equality should be \leftarrow (since both sides have v^k).  %% After author feedback %% I thank the authors for their reply. Some futher comments to improve the quality of the paper:  First, I suggest the authors add complete proofs (in the SM), and the main Theorem they mention in the rebuttal.  Second, please include citation and discussion related to the works mentioned by Reviewer 3. For example, with regard to Ma et al., which seems most relevant (the results in the first two papers assume strong convexity, which makes them generally inapplicable to over-parameterized models), I think the main differences are: 1) Ma et al. focused on how to select the learning rate and batch size to achieve an optimal convergence rate. In contrast, this paper focuses on how the batch size and learning rate determine when do we get dynamical instability and SGD escapes away from the minimum.  2) I think the resulting expressions are rather different. For example, the bounds in Ma et al. (e.g. Theorem 2) depend on the eigenvalues of the Hessian, while in this paper they depend on the maximal eigenvalues of H and Sigma. The latter expressions expose nicely the important factors affecting stability (sharpness and non-uniformity). Also, (and this is more minor) in Ma et al. the derived upper bound is for the expected loss, while here the bound is the first two moments of the optimized parameters. 3) I believe the results in this paper are more easily generalizable (to realistic models) since we only care about dynamical stability near the fixed point, and not the optimal learning rate/batch size (Which depend on the entire optimization path).  4) The derivation in this paper is quite simple and intuitive (at least for me).  Third, the definitions of non-uniformity vary somewhat in the paper. Specifically: a. line 24 (intro): "the variance of the Hessian (see below for details)" b. line 66 (1D case): non-uniformity = s = std of the (1D) Hessian c. lines 90-91 (General case, matrix version): \Sigma (variance of Hessian matrix) is a "measure of the non-uniformity"  d. lines 97-98 (General case, scalar version): non-uniformity = s, where s^2 is the largest eigenvalue of \Sigma e. Appendix C (what they used in practice): same as the previous definition (up to the approximation accuracy of the power iteration method). So definitions (b),(d), and (e) are consistent, and I think it is pretty clear that this what they actually mean by "non-uniformity".  (c) is different, but they only say this is a "measure of .." and not the definition. I think this matrix version was mainly pointed out to give some intuition. (a) indeed seems to be inaccurate, and should be corrected, but I think it is a small mistake. I think these issues should be corrected, and a formal definition should be given, as non-uniformity is one of the key ingredients of this paper.  Fourth, I think the authors should try to emphasize and clarify more the important parts in their experiments. I'll try to summarize the interesting observations I found in the experiments: 1) The sharpness value is nearly always near the maximum value (2/eta) for GD (Table 2, Figure 6a); Non-uniformity also has values with a similar magnitude as the bound (Figure 6a). 2) The linear relation between sharpness and non-uniformity (Figure 6) 3) Observations (1-2) above (as well as figure 3), suggest that smaller mini-batches decrease sharpness due to the non-uniformity bound (Figure 6). 4) Non-uniformity seems to predict well when we escape from sharp minima (Figure 7). 5) The escape from sharp minima is fast (Figure 7). I think this is quite a lot of new interesting information that I've learned here. Previous high impact papers suggested that * Keskar et al.: (a) "large batch size implies large sharpness"  (b) "small sharpness is important for generalization" * Hoffer et al., Goyal et al., Smith and Le: (c) "changing the learning rate at large batch sizes can improve generalization" However, so far it was not clear why these statements should be true (e.g. counterexample in Appendix A for the statement b). I think the observations (1-4) above make significant progress in understanding statements (a-c), and suggests we should also focus on non-uniformity (instead of just sharpness). Also, observation 5 suggests that previous SDE models for this phenomena are wrong. Other previous explanations (e.g., Yin et al.) seems not be applicable to over-parameterized case, as explained in the discussion. In terms of math, I am not aware of other works that examined this specific over-parameterized case, where all sample gradient are zero at the minimum, except for Ma et al. (which focused on different questions).   Fifth, I think that there is an issue in the conditions relying only on the maximal eigenvalues. For example, in at line 98 the author state that "a sufficient condition for (9) to hold is eq. (10)", and condition (10) only depends on the maximum eigenvalue of H.  However, I don't see why small (possibly even zero) eigenvalues of H can't affect stability. For example, suppose that B=1 and \eta=1, that v is the eigenvector corresponding the max eigenvalue of (9), and also that H*v=c*v, \Sigma*v = q*v for two positive scalars c and q.  In this case, we get an eigenvalue of the matrix in (9) which is equal to (1-c)^2+q. This value can be larger than 1, if c is sufficiently small (e.g. c=0), relatively to q. The authors should explain how this issue could be avoided, or correct their statement. In any case, I recommend adding more mathematical details for all calculations in the supplementary.