Forecasting Treatment Responses Over Time Using Recurrent Marginal Structural Networks
Electronic health records provide a rich source of data for machine learning methods to learn dynamic treatment responses over time. However, any direct estimation is hampered by the presence of time-dependent confounding, where actions taken are dependent on time-varying variables related to the outcome of interest. Drawing inspiration from marginal structural models, a class of methods in epidemiology which use propensity weighting to adjust for time-dependent confounders, we introduce the Recurrent Marginal Structural Network -a sequence-to-sequence architecture for forecasting a patient's expected response to a series of planned treatments. Using simulations of a state-of-the-art pharmacokinetic-pharmacodynamic (PK-PD) model of tumor growth [12], we demonstrate the ability of our network to accurately learn unbiased treatment responses from observational data -even under changes in the policy of treatment assignments -and performance gains over benchmarks.

Post-rebuttal comment: In their response, the authors answered my primary question about breaking apart their model (which consists of two RNN components) to identify more specifically where the gains of this model come from. They also explained how Figure 4 contains similar comparisons, and clarified my question about the gamma = 0 setting. I think the paper would benefit from a little more clarity around Figure 4 to minimise the confusion I experienced, but otherwise I remain confident in my assessment of the quality of the paper. I note also that their response to me also addressed one of the concerns of Reviewer 3.  Summary of work: This work is focused on estimating treatment responses through time in the presence of time-dependent confounders using marginal structural networks. Marginal structural networks have been used in epidemiology and other statistical fields, and exploit inverse probability of treatment weighting (IPTW) to adjust for bias. They require two components - the covariate-dependent probability of treatment assignment (propensity) and the treatment-dependent response. In this work, they use RNNs (LSTMs) to approximate each of these components. They use a commonly-used synthetic dataset to develop and test their model, comparing against standard marginal structural networks and a Bayesian approach.  Explanation of overall score: This is a strong submission overall - the experiments are reasonably convincing, the architecture and approach is well explained, and the problem seems important. I also think the idea of bringing approaches used elsewhere (e.g. epidemiology) for causal inference to a NIPS audience, and demonstrating how neural networks can be beneficial, is a worthy endeavour.  Explanation of confidence score: I am not very familiar with one of the primary comparisons in this work (the BTRC model) and therefore can’t guarantee that the comparison was fair.  Quality:  * The analyses include two important experiments: varying the level of time-dependent confounding (specifically, how treatment assignment depends on the patient’s current state, which is the tumour diameter in these experiments), and varying the prediction horizon (for different levels of confounding). * In the first experiment they also compare their proposed approach with a hybrid where the IPTWs are derived from the “standard” MSM model. Further along this vein, I would be interested to know how well a “vanilla” RNN would perform at this task. That is, if the task in Figure 4 is simply to predict the treatment response at the next time-point, how well can you do if you neglect the MSM modelling entirely, and treat it as a standard regression problem? I think, for readers not familiar with MSMs, this comparison might help to motivate why the MSM framework is necessary. * Equivalently, it would be interesting to see how well a MSM performs if the IPTWs are estimated using RNNs, but the treatment response itself is given by the standard regression model. In essence, this work proposes an upgrade of MSMs with two new parts, and demonstrates that this works better. But then the question is where did the greatest performance gain come from? * I am slightly confused by the performance in Figure 4 when there is no time-dependent confounding. In this case, shouldn’t all IPTWs (if estimated correctly) be the same? If this is true, then shouldn’t the performance of the R-MSN and the Seq2Seq/Logistic baselines be the same? My intuition in this case is that the RMSE should be driven by the baseline difficulty to predict if a treatment will be assigned (since, according to (9), gamma = 0 indicates random assignments), on top of the error of the treatment-dependent response prediction. If the latter part uses the same model, then how does the RMSE differ? * To thoroughly demonstrate the power of the RNNs to adapt to different types of response models, I would have liked to see experiments on a different dataset.  Clarity:  * The submission is well-organised and fairly clearly written. My biggest complaint would be that there is relatively less detail about how the final model specifically looks - it would perhaps be sufficient to refer back to equation (4) after the R-MSN has been defined (around equation (5)), to make it clear how the RNNs are fitting in to the MSM model. * There are some small errors in the text of 4.3 - in Step 2, it says Figure 3(c) instead of Figure 3(b), and later says “decoder” when it should probably say “encoder”. (I only point these out because these descriptions are quite important!). * As far as I know, there is no limit on the length of the references for the main paper, so I would recommend to move all references appearing in the main text to the main bibliography, it’s a little confusing otherwise. * The appendix contains a good amount of detail about training and hyperparameter optimisation, making this paper likely easy to reproduce 🙂 (I hope the authors will also provide code when feasible)   Originality:   * To the best of my knowledge the idea to parametrise elements of a MSM using RNNs is original. * The appendix contains an extended related work section, helpfully placing the work in context.  Significance:  * The problem of accounting for time-dependent confounders is important in various domains of applied machine learning. * This idea to me seems like a nice application of methods from deep learning towards more traditional statistical sciences, which may also help to publicise models such as MSMs to the NIPS community.


Joint Active Feature Acquisition and Classification with Variable-Size Set Encoding
We consider the problem of active feature acquisition where the goal is to sequentially select the subset of features in order to achieve the maximum prediction performance in the most cost-effective way at test time. In this work, we formulate this active feature acquisition as a joint learning problem of training both the classifier (environment) and the reinforcement learning (RL) agent that decides either to 'stop and predict' or 'collect a new feature' at test time, in a cost-sensitive manner. We also introduce a novel encoding scheme to represent acquired subsets of features by proposing an order-invariant set encoding at the feature level, which also significantly reduces the search space for our agent. We evaluate our model on a carefully designed synthetic dataset for the active feature acquisition as well as several medical datasets. Our framework shows meaningful feature acquisition process for diagnosis that complies with human knowledge, and outperforms all baselines in terms of prediction performance as well as feature acquisition cost.

The authors consider the problem of active feature acquisition (i.e., the setting where each feature includes a cost to see the value, often motivated by diagnostic tests in medical settings), where the goal is to maximize classifier performance with a minimum cost subset of features. Methodologically, they start by modifying the objective function to include empirical risk minimization and feature acquisition cost minimization. The acquisition mechanism is done via a RL formulation in a method similar to RL/imitation learning methods for feature selection (e.g., [15-18]), where the state is the observable features, the action is to acquire a new feature (or classify), and the reward is the (negative) feature acquisition cost. The interesting aspect of their approach is that the classifier and RL agent are jointly optimized (RL via deep Q-learning), requiring a clever set encoding method to manage the large search space (similar to [Vinyals, Bengio & Kudlur; ICLR 2016]). The joint optimization is completed via a joint encoding of the classification and policy with shared encoder layers. Experiments are first conducted on synthetic data sets to elucidate specific properties (e.g., the value of joint training vs. a pre-trained classifier, the benefits of set encoding). Secondly, experiments are conducted on PhysioNet (four classification tasks) and a private Medical check-up dataset. Evaluating using AUC, they show that they are able to generate competitive results while observing ~20% of the features.  In summary, this is an interesting approach to feature value acquisition that builds on recent work wrt meta-learning with RL-based approaches and adopting this work to this particular setting with the joint learning and set encoding methods that are entirely sensible.  Additionally, they provide sufficient training details where it seems likely that others would be able to produce this work. Experiments are conducted on synthetic sets, ‘real’ (bake-off) datasets, and an actual real-world problem. The only technical detail I find a bit surprising is that they are able to train these algorithms with 1000s of examples, when policy gradient is generally fairly data hungry, etc. Thus it would also be nice to see more ‘traditional’ baseline results (i.e., no deep learning, just an expected gain test) — particularly for the real-world problems. While I am not a true expert on this particular problem, this is ostensibly a novel approach to a potentially important problem with reasonably convincing results. However, the writing needs significant work (and I am not entirely certain it followed the latest style files) including structurally, grammatically, and even some of the precision of claims (covered in more detail below). Accordingly, I am ambivalent wrt accepting this work in its current form, learning toward rejection given the likely number of strong papers submitted.  [Quality]: Overall, the paper is well-motivated, specifies a sensible model, provides a reasonable level of implementation details, and generates promising experiments. While the writing needs work, I will leave this to ‘clarity’. I do have a few questions that are a bit nit-picky: (1) the authors state a ‘cost-sensitively (sic) regularized optimization problem’; however, it isn’t clear to me that this is a regularizer per se as I generally think of regularization as a variance reduction mechanism — not all penalty functions within the objective will do this (and I’m not certain this does), (2) the authors make a point that ‘To search this huge space efficiently and stably, we carefully devise the architecture with the recently proposed set encoding method’ — as best as I can tell, this is exactly the same method, correct?, (3) Lines 131-132: “the hyperparameter balancing the relative importances of two terms…is absorbed in the predefined cost” — isn’t this difficult to do while keeping the costs interpretable (or is this just notational)?.  Overall, precisely clarifying these differences wrt existing work would strengthen the paper.  [Clarity]: The paper needs significant writing work. I will give examples of each issue (in my opinion), but it is an overall issue. - Structure: The introduction starts with a discussion around deep learning and proceeds to feature acquisition — it isn’t entirely clear why deep learning is a significant aspect of this paper. I would think one would start with the feature acquisition aspect and then proceed to deep learning (although I think it really is secondary). The related work section is basically a ‘laundry list’ of papers. Without reading them, they are mostly just citations — while this is an opportunity to provide a general framework and demonstrate how this fits together. - Grammar: The paper is full of simple errors like missing determiners, syntactic errors, odd word choice, tc.  (i.e., Line 37: “Given a new dataset”; Line 36: “to be sufficient confident”; Line 47: “It is worth noting that we do not assume a fixed classifier that is pre-trained during a separate phase.; Line 57: “We formulate the feature acquisition problem as a cost-sensitive penalty in the objective function and optimize with a joint…”; Line: 69: “…we cannot guarantee that all of the features are available at the start”; Line 98: “Also, our”).  These are all over the place. Finally, I think there should be more discussion around the empirical results in general.  [Originality]: While I am not a true expert in this area, while it seems to draw significantly from related work, it also put together multiple relevant works and ostensibly thought through the details — showing promising results. While I think many could have come up with this formulation, I also don’t believe it trivial and is an advance over more directly formulations.  [Significance]: While I would like to see additional experiments contrasting with non-deep learning methods, this work does seems like a promising approach to active feature value acquisition. This is a practical problem (even if somewhat narrow in scope) — so I would say that it meets the bar in this regard.  Overall, this paper has good ideas, solid execution, and promising results. However, given the competitiveness of NIPS, I think the paper need significant writing work to be competitive in its current form.  Comments after author rebuttal: I do appreciate Table A and in reading the other reviewers comments, I may have over-indexed on the writing (which does need work, but it is difficult for me to validate) and under-appreciated the effort to get this validated in a practical setting. Thus, I am a bit more positive wrt this work, but still believe it should be somewhat borderline and wish we had more confident reviewers.


Representation Learning for Treatment Effect Estimation from Observational Data
Estimating individual treatment effect (ITE) is a challenging problem in causal inference, due to the missing counterfactuals and the selection bias. Existing ITE estimation methods mainly focus on balancing the distributions of control and treated groups, but ignore the local similarity information that provides meaningful constraints on the ITE estimation. In this paper, we propose a local similarity preserved individual treatment effect (SITE) estimation method based on deep representation learning. SITE preserves local similarity and balances data distributions simultaneously, by focusing on several hard samples in each mini-batch. Experimental results on synthetic and three real-world datasets demonstrate the advantages of the proposed SITE method, compared with the state-of-the-art ITE estimation methods.

# Summary  This paper proposes a method for estimating individual treatment effects. The basic approach is to propose a regression y_i = g(f(x_i), t_i) where f(x_i) is a learned representation of the covariates, and g predicts the response using this representation and the treatment. The main novelty is in the latent representation g(x_i). The given motivation is that points that are close in the covariate space should also be close in the representation space. This is achieved by adding two terms to the mini-batch loss function.   # Strengths  The problem is self evidently important, the idea is creative, and seems to work well under to an (apparently) broad battery of empirical tests.     # Weaknesses  I had trouble understanding this paper, but I'm not sure if that's due to the exposition or because it's a little ways outside my normal area of expertise. In particular, I don't immediately see the answer to:  1. When, if ever, is SITE a consistent estimator of the true average ITE? What assumptions are required for the estimator to be a valid estimator of the causal effect? In particular, the loss function is only defined over minibatches, so the objective function for the regression is left implicit. Presupposing that SGD does in fact converge, does this oddball objective function cause any issues?  2. Is the intuitive motivation (representation preserves local similarity) actually supported by the empirical results? It's clear that SITE does somewhat better than the comparitor methods, but it's not clear to me why this is---could it attributed simply to higher model capacity?  Experiments showing that SITE actually learns representations that do a better job of preserving local similarity, and that this matters, would strengthen the paper.  I note that the authors claim (line 263-265) that the experiments show this already, but I don't understand how.    More broadly, the whole procedure in this paper feels a bit ad hoc. It's not clear to me which parts of the model are really important for the results to hold.   # Overview   Based on the above critiques, I've voted (weakly) to reject. However, if good answers to the above points of confusion can be incorporated into the text, I'd be happy to reconsider.    # Further questions and comments 1. The learning procedure described in the paper seems to take propensity scores as inputs. E.g. triplet pair selection relies on this. How are the required propensity scores determined? 2. Does the SGD actually converge? It seems possible that selecting triplets within the mini-batch may lead to issues. 3. What is the summation (\Sum) in equation 7 over? I thought there was exactly one \hat{i}, \hat{j}, etc. per minibatch 4. Is the correct interpretation of the experiment in section 3.2 that the covariates x are generated according to one of the two normal distributions, the response is generated independent of T, and the ground truth ITE is 0? Assuming this is correct, the text should be rewritten to be clearer. I'm also unclear what the inputs to the KL distance are, and why KL is chosen as the distance       #### Update  I thank the authors for responding to my comments and questions.  1. The ablation study with PDDM is helpful. I think the paper would be improved by adding this, and also one for MPDM loss term.   The presented results show that PDDM improves the ITE estimate, but, as far as I can see, it does not show that the PDDM term actually causes the learned representations to better preserve local similarity.      2. I'm still confused by when, if ever, the method described in the paper is a correct estimator for ITE (defining 'correct' sensibly is part of the challenge here). The author response didn't cleared this up for me. For example, it was still unclear why assumptions 2.1 and 2.2 are included given that, as far as I can tell, neither is actually referenced again in the paper.   I was sufficiently confused by this that I went back and read the related prior work ("Learning Representations for Counterfactual Inference" Johansson, Shalit, Sontag, and "Estimating Individual Treatment Effect: Generalization Bounds and Algorithms", ref [28] in the paper). These papers establish a clear meaning for the use of representation learning for causal inference: the PEHE is bounded by an expression that includes the distance between P(r | t=0) and P(r | t=1), the distributions over the representations for the treated and untreated. I'm convinced this could provide a clear motivation and (hand wavy) justification for SITE. However, I don't think the paper explains this well---if the method is justified by ref [28] then the key results and the connection should be summarized in the paper!  There is presumably some analogous idea making use of the assumption that similar units will have similar responses, but I'm again unsure what such a result is or exactly how it might apply to the proposed method.  Although I'm now convinced that the method can probably be justified properly, I don't think this is done in the paper, and I view the the confusing presentation as a serious problem.  For this reason, I'm still voting to reject. However, I note that reviewer 1 has essentially the same criticism (to wit: this method doesn't seem to actually estimate ITE), but doesn't view this is a serious issue.


The Description Length of Deep Learning Models
Solomonoff's general theory of inference (Solomonoff, 1964) and the Minimum Description Length principle (Grünwald, 2007;Rissanen, 2007) formalize Occam's razor, and hold that a good model of data is a model that is good at losslessly compressing the data, including the cost of describing the model itself. Deep neural networks might seem to go against this principle given the large number of parameters to be encoded. We demonstrate experimentally the ability of deep neural networks to compress the training data even when accounting for parameter encoding. The compression viewpoint originally motivated the use of variational methods in neural networks (Hinton and Van Camp, 1993;Schmidhuber, 1997). Unexpectedly, we found that these variational methods provide surprisingly poor compression bounds, despite being explicitly built to minimize such bounds. This might explain the relatively poor practical performance of variational methods in deep learning. On the other hand, simple incremental encoding methods yield excellent compression values on deep networks, vindicating Solomonoff's approach.

Summary :   The paper considers the problem of minimum description length of any model - resting on the conjecture (as supported by Solomonoff’s general theory of inference and the Minimum Description Length) that a good model must compress the date along with its parameters well. Thus generalization is heuristically linked to compressive power of the model. With their experimental results, the authors surprise showing good overall compressive power of Deep Learning models using the prequential coding approach.  Relating to the "Chaitin’s hypothesis [Cha07] that “comprehension is compression” any regularity in the data can be exploited both to compress it and to make predictions", the authors focus only on the compression problem .  Detailed Comments:  Clarity: The paper in well written and presents a good survey of the related work.  Originality: There is no theory in the paper. The paper presents an empirical study of the possible confirmation of the Chaitin’s hypothesis [Cha07]  that DL generalizes as it compresses well too, i.e., it has small MDL. For whatever it is worth, it is an interesting approach to understand an important problem in DL community, viz. the generalization issue despite the huge number of parameters in a DL model.  Quality and Significance: I have the following concerns towards the acceptance of this paper:  (a) No Sound Theory : This is the biggest drawback. Beyond the two experiments presented it's not rigorously clear, that describing a model with less description complexity has direct connection to generalization. You can always use a over-parametrized model, and still get very good generalization (see https://arxiv.org/abs/1710.05468), and that over-parametrized model may not have good MDL. Perhaps, what we need to quantify is some inter layer quantity layer (like Tishby's paper: https://arxiv.org/abs/1503.02406), so that the representation is learned to be well-suited for classification. So the connection with MDL is not clear.   (b) Experiments not reproduced carefully : There is further concern about Section 3.4. Prequential coding applying to non-sequential data (e.g MNIST) is equivalent to training ConvNet without shuffling the data batches. The result with MNIST at 99.5% is not state of the art. State of the art is around 99.8% (https://github.com/hwalsuklee/how-far-can-we-go-with-MNIST).  (c) Lack of enough experiments : For the claim supporting prequential coding causing good generalization, a better experiment will be using other dataset, which is not well studied as MNIST and CIFAR. Comparing Vanilla ConvNet model and Prequential Coding ConvNet model without any explicit regularization will be fair. Prequential code may not work very well, since it doesn't have shuffling dataset ability, which might make neural network overfit in some order-based rules.  Overall, I like the idea of applying Information Theory to Deep Learning, but description length may not be the right way, or atleast has not been explained and exposited rigorously with this submission. A good revision may be required.   ****** In the light of the author-feedback after the review process, I agree that quite a few of my concerns have been addressed and hence the overall score and the score of reproducibility has been updated.


A Smoothed Analysis of the Greedy Algorithm for the Linear Contextual Bandit Problem
Bandit learning is characterized by the tension between long-term exploration and short-term exploitation. However, as has recently been noted, in settings in which the choices of the learning algorithm correspond to important decisions about individual people (such as criminal recidivism prediction, lending, and sequential drug trials), exploration corresponds to explicitly sacrificing the well-being of one individual for the potential future benefit of others. In such settings, one might like to run a "greedy" algorithm, which always makes the optimal decision for the individuals at hand -but doing this can result in a catastrophic failure to learn. In this paper, we consider the linear contextual bandit problem and revisit the performance of the greedy algorithm. We give a smoothed analysis, showing that even when contexts may be chosen by an adversary, small perturbations of the adversary's choices suffice for the algorithm to achieve "no regret", perhaps (depending on the specifics of the setting) with a constant amount of initial training data. This suggests that in slightly perturbed environments, exploration and exploitation need not be in conflict in the linear setting.

Post Rebuttal:  Thanks for the clarification. You are right about the lemma. Sorry about this. I raised my score correspondingly.  In my defense, it does seem like the lemma can be improved, however. Using the method of mixtures the dependence on d in the square root should disappear. Perhaps there is a small price to pay in the logarithmic term. Here is the brief sketch:  Let eta_1,...,eta_n be a sequence of conditionally 1-subgaussian random variables adapted to filtration (F_t). Then let X_1,...,X_n be a sequence of random elements with X_t in R^d and assume that X_t is F_{t-1}-measurable (that is, (X_t) is (F_t)-predictable).  Let us fix 1 <= i <= d. Since X_{ti} is F_{t-1}-measurable and eta_t is conditionally 1-subgaussian it follows that X_{ti} eta_t is conditionally X_{ti}-subgaussian. Hence  exp(sum_t (L X_{ti} eta_t - L^2 X_{ti}^2 / 2)) is a supermartingale for any L > 0.  Hence Prob(sum_t X_{ti} eta_t >= L sum_t X_{ti}^2 / 2 + log(1/delta) / L) <= delta for any L > 0.  If the X_{ti} where non-random, then you could choose L \approx sqrt(log(1/delta) / sum_t X_{ti}^2). But you can't do this, because it is random. The solution is either the method of mixtures or a covering argument. The first is cleaner (Google it). The covering argument is easier to explain simply.  Since 0 <= sum_t X_{ti}^2 <= n we can take a union bound over all L in some covering set of [0,n] shows that  Prob(sum_t X_{ti} eta_t >= something_1 + sqrt(sum_t X_{ti}^2 log(something_2 /delta)) where something_2 should be a lower degree polynomial of n and something_1 is small due to the approximation error in the covering set.  Now take a union bound over the coordinates and argue symmetrically for the lower tails and you should have something like  |sum_t X_{ti} eta_t| lesssim sqrt(sum_t X_{ti}^2 log(d n / delta))  Hence || sum_t eta_t X_t ||_2^2 <= sum_i sum_t X_{ti}^2 log(dn / delta) = n log(dn/delta).  And so you should have a bound that is O(sqrt(n log(dn / delta))). I did not go back to check how much this would improve the results.  I also encourage you to work hard on motivating the assumptions and correcting the minor notational issues.  Summary.  This paper is about contextual linear bandits. The authors are trying to understand the phenomenon observed in practice that exploration seems much less important than the theory usually suggests. To make a start understanding this they analyze the greedy policy that chooses the arm for which the inner product with the least squares estimator and the context is largest.  With no further assumptions this leads to linear regret. The authors make a 'perturbation' assumption where the contexts are first chosen by an adversary and then perturbed using zero-mean noise. Under carefully chosen assumptions on the noise they show that the greedy algorithm now enjoys O(sqrt(T)) regret.  The two standard settings are studied. (a) where there is one unknown parameter vector and (b) where there is an unknown parameter vector for each action.  The results setting (b) are unsurprisingly much weaker than (a) where some forced exploration of all arms is required. A lower bound is given proving that this is unavoidable for the greedy algorithm. In setting (a) the assumptions on the perturbation distribution ensure the minimum eigenvalue of the design matrix is increasing linearly, which ensures sufficient accuracy of the least-squares estimator.   Quality:  The results seem reasonable in principle. I think there is at least one bug in the proof (see 'Question about Lemma A1' below). It should be fixable with limited loss in the logarithms.  Originality:  Perturbation analysis of this kind is not new, but I think the modifications are interesting enough.   Clarity:  This is a weak point of the paper. There are issues with row/column vectors being mismatched, notation used in multiple different ways. Many unexplained details. Regret bounds and conditions under which they hold are given in Big-O notation and there is no way to see what is the variable of interest. The supplementary material is in better shape than the main body, which is a bit unfortunate. Statements involving probabilities are often a little imprecise and there are many typos. The full paper is also very long. I have to say I did not read all proofs.   Significance:  The difference between practice and theory for contextual bandits is quite large and any attempt to understand this is welcome and potentially significant. For the present article the main issue is the validity of the assumptions. Is the perturbation model reasonable? I did not see that this is addressed anywhere, which is quite unfortunate.   Other comments:  I leave the philosophy to others, but is it not also repugnant and short-sighted to explore suboptimally in clinical trials and then waste the results because the data does not statistically support the hypothesis that the drug works? I thought the point of sequential design was to optimize this process.  Perhaps a better argument is that explicit exploration may actually not be required. That is, explicit exploration is not the enemy, but rather excessive exploration.  Question about Lemma A.1:  The proof of this result does not seem correct. You are right that the coordinates of S_t are rs-subgaussian, but this is much weaker than you need because summing over the squares will give you a bound of sqrt(2trsd log(d/delta)), which has an extra d relative to what you have.  Really what you want to show is that with high probability  sum_{s=1}^t eta_s x^s_i < sqrt(sum_{s=1}^t eta_s (x^s_i)^2 log(1/delta))  Then summing the squares of the coordinates and taking the sqrt will yield your bound. The problem now is that showing the above for arbitrary choices of x the above might not hold. You will need a more sophisticated method such as the method of mixtures and something extra will appear in the logarithm.    Overall I find the paper to be interesting, but suffers from several issues in terms of clarity and motivation for the assumptions. Having found so many issues in the first 20 pages or so it seems the authors should spend more time polishing the article and perhaps consider submitting to a journal where a work of this length might be more appreciated.   Minors:  * Is mathcal H^* defined?  * Supp. First para of proof of Lemma 3.2 there is a missing \Vert  * There seems to be some indecision about what is a column/row vector. Eg., L215 there is ^beta^t x_i^t  * Display after L185. beta e -> beta . e.  * Should say that s-subgaussian means exp(lambda X) <= exp(lambda^2 s / 2) not exp(lambda^2 s^2 / 2)? Unfortunately different authors use different defs (I see it in the supp, but not main)  * Sometimes you write || . || and sometimes || . ||_2 for the Euclidean norm.  * Condition 1 (and elsewhere). You can help the reader by giving domains for variables.  * The wide hats over r-good and r-auspicious don't seem to add anything and look bad.  * Theorem 4.1. The definition of low perturbation regime is given in terms of Big-O notation and the regret here is also Big-O notation. But what is the variable? T? d? The theorem statement should be made rigorous.  * Expectation operators are often used when the underlying measure is unclear and sometimes the notation is really strange (Proof of Lem 3.2 for example).  * Second to last equality in Proof of Lem 3.2. Why is this an equality? Diversity provides a lower bound, no?  * Why is there a t in logarithm in Lemma A.1? The union bound is done coordinate-wise, not over time so it should be log(d/delta).  * Lemma A.2. This proof is missing some expectations. The LHS is a number, but the RHS after the second inequality is a random variables.  * Proof of Lemma 3.1. Is eta^t defined? I suppose it must be the vector of noise up to time t. Note that this lemma probably needs changing if Lemma A.1 is indeed not true. Elsewhere it seems that eta^t is the vector of noise over the coordinates, so this seems like a bit of a mismatch.  * The statement of Corollay 3.1 is a bit weird. Whether or not a certain number of rounds are r-auspicious is itself a random quantity, so the probabalistic quantifier should not come after. Perhaps you need to define an indicator function IND that this number of rounds occurs and then claim that with probability at least 1 - delta it holds that  IND ||beta - ^beta^t|| \leq 32 \sqrt ....  * L223. The probability measure here should be given. This is over the perturbations of the adversary in round t only.


Multi-Agent Generative Adversarial Imitation Learning
Imitation learning algorithms can be used to learn a policy from expert demonstrations without access to a reward signal. However, most existing approaches are not applicable in multi-agent settings due to the existence of multiple (Nash) equilibria and non-stationary environments. We propose a new framework for multi-agent imitation learning for general Markov games, where we build upon a generalized notion of inverse reinforcement learning. We further introduce a practical multiagent actor-critic algorithm with good empirical performance. Our method can be used to imitate complex behaviors in high-dimensional environments with multiple cooperative or competing agents.

This paper proposes several alternative extensions of GAIL to multi-agent imitation learning settings. The paper includes strong, positive results on a wide range of environments against a suitable selection of baselines. However, insufficient details of the environments is provided to reproduce or fully appreciate the complexity of these environments. If accepted, I would request the authors add these to the appendix and would appreciate details (space permitting) to be discussed in the rebuttal - particularly the state representation.  The more pressing point I would like to raise for discussion in the rebuttal is with regard to the MACK algorithm proposed for the generator. The authors make a justified argument for the novelty of the algorithm, but do not thoroughly justify why they used this algorithm instead of an established MARL algorithm (e.g. Multi-Agent DDPG). It seems that MACK is in itself also a contribution to the literature, but it is not directly compared to a suitable existing baseline. Its use also convolutes whether the gains seen by multi-agent GAIL were due to the different priors used or the different RL algorithm used for the generator. Ideally it would be interesting to see an ablation study showing how multi-agent GAIL performs with the different priors proposed in Figure 1 but with TRPO as in the original paper. Additionally, given the choice to use MACK instead of an established multi-agent RL algorithm, a comparison of multi-agent GAIL with MACK to multi-agent GAIL with multi-agent DDPG would provide empirical evidence of the benefit of MACK.  For future work, I would suggest the authors revisit an assumption of the paper that the number of experts matches directly the number of agents. However, I don't think this assumption is strictly necessary for some of the multi agent extensions proposed and would be an interesting further empirical study as it enables unique use cases where expert data is limited but could generalise across agents or provide partial insight into how some of the agents should act.


Demystifying excessively volatile human learning: A Bayesian persistent prior and a neural approximation
Understanding how humans and animals learn about statistical regularities in stable and volatile environments, and utilize these regularities to make predictions and decisions, is an important problem in neuroscience and psychology. Using a Bayesian modeling framework, specifically the Dynamic Belief Model (DBM), it has previously been shown that humans tend to make the default assumption that environmental statistics undergo abrupt, unsignaled changes, even when environmental statistics are actually stable. Because exact Bayesian inference in this setting, an example of switching state space models, is computationally intensive, a number of approximately Bayesian and heuristic algorithms have been proposed to account for learning/prediction in the brain. Here, we examine a neurally plausible algorithm, a special case of leaky integration dynamics we denote as EXP (for exponential filtering), that is significantly simpler than all previously suggested algorithms except for the delta-learning rule, and which far outperforms the delta rule in approximating Bayesian prediction performance. We derive the theoretical relationship between DBM and EXP, and show that EXP gains computational efficiency by foregoing the representation of inferential uncertainty (as does the delta rule), but that it nevertheless achieves near-Bayesian performance due to its ability to incorporate a "persistent prior" influence unique to DBM and absent from the other algorithms. Furthermore, we show that EXP is comparable to DBM but better than all other models in reproducing human behavior in a visual search task, suggesting that human learning and prediction also incorporates an element of persistent prior. More broadly, our work demonstrates that when observations are information-poor, detecting changes or modulating the learning rate is both difficult and thus unnecessary for making Bayes-optimal predictions.

The authors introduce an approximate model of the Bayesian dynamic belief model (DBM)  that is based on exponential filtering and can produce similar results as the DBM but at lower computational costs and increased neural plausibility. I like the idea of defining more plausible solutions to the puzzle of volatililty in human behavior. Yet this paper somehow left me wanting more.   Here are some suggestions on how to improve it:  - as an author I always dislike reviews that say the present work isn't novel enough myself, but this paper really felt a little like that; I don't think that this is necessarily the authors fault per se and definitely doesn't justify rejecting the paper. However, I think the intro could motivate the models a little more. I think it would make sense to mention some cases where humans are volatile, how DBM captures this behavior and why more plausible implementations based on approximations are useful (for example, mentioning computational and neural constraints).   - some minor comments: sometimes the quotation marks are inverted, sometimes not. some of the equations aren't numbered.  - The experiment is a little opaque to me. I don't really understand what was done there. Is this the data from the Yu & Huang paper? How many participants were there? What was their task and how did they get rewarded? I'm not fully sure what Figure 1c means. I guess it shows that the predictions of DBM and Exp align with the data in terms of which patch is likely on average. Is this the only thing the model can capture here? Are there any inter-individual differences that the model can pick up on? Why is the prediction only about the first fixation? Couldn't this be captured already by accounting for some level of stickiness based on what was previously right? How does this relate to volatility of participants' belief if it will be the same option again? I think this part needs more explanation  - In general, I was really excited about the paper when I saw the title, but then it is never really explained how this actually demystifies volatile learning. Doesn't the DBM model already demystify volatility and the current model is a more neurally plausible approximation to that model? My concern about this paper is that it might be a little too niche for the general NIPS community as it is about a more plausible approximation to a model that has been proposed before, so people would essentially have to know the prior model, too. I don't think that's a bad idea and incremental work should be encouraged of course, but then I would really try and emphasize how this goes beyond DBM, perhaps even thinking about a novel prediction it could generate.  - equivalent to a particular form of leaky-integrating neuronal dynamics [19]. -> I think the general NIPS audience would need another sentence here to explain why this is important.  - I wonder how exponential filtering relates to other filtering approaches such as Kalman filtering or Harrison Ritz' control theoretic model of learning in changing envrionments.


SNIPER: Efficient Multi-Scale Training
We present SNIPER, an algorithm for performing efficient multi-scale training in instance level visual recognition tasks. Instead of processing every pixel in an image pyramid, SNIPER processes context regions around ground-truth instances (referred to as chips) at the appropriate scale. For background sampling, these context-regions are generated using proposals extracted from a region proposal network trained with a short learning schedule. Hence, the number of chips generated per image during training adaptively changes based on the scene complexity. SNIPER only processes 30% more pixels compared to the commonly used single scale training at 800x1333 pixels on the COCO dataset. But, it also observes samples from extreme resolutions of the image pyramid, like 1400x2000 pixels. As SNIPER operates on resampled low resolution chips (512x512 pixels), it can have a batch size as large as 20 on a single GPU even with a ResNet-101 backbone. Therefore it can benefit from batch-normalization during training without the need for synchronizing batch-normalization statistics across GPUs. SNIPER brings training of instance level recognition tasks like object detection closer to the protocol for image classification and suggests that the commonly accepted guideline that it is important to train on high resolution images for instance level visual recognition tasks might not be correct. Our implementation based on Faster-RCNN with a ResNet-101 backbone obtains an mAP of 47.6% on the COCO dataset for bounding box detection and can process 5 images per second during inference with a single GPU. Code is available at https://github.com/mahyarnajibi/SNIPER/.

#1. Summary This paper introduces practical techniques, called SNIPER, for speed-up training of object detection networks based on Region Proposal Networks (RPNs). The idea is simple: Instead of considering numerous object candidate boxes in the entire input image, SNIPER samples sub-images called “chips” and train detection models to localize objects in such chips. Specifically, positive chips are generated to contain ground-truth object boxes of a certain scale range, and negative ones are sub-images that has object proposals likely but not actual ground-truth. Since the scale of chips are commonly normalized to a small scale, object detectors can be scale invariant (as in SNIP [24]) and its training procedure becomes more efficient and faster than before.  #2. Quality 1) Experimental evaluation is weak, and the advantages of SNIPER claimed in the manuscript are not empirically justified. First, SNIPER is compared only with SNIP, its previous version, and current state of the arts are not listed in the tables. Moreover, even though the speed-up of training is the most important and unique contribution of SNIPER, there is no quantitative analysis on that aspect at all.  2) Its originality is quite limited. Please see #3 below for more details. 3) The quality of presentation is not good enough. Please see #4 below for more details. >> Thus, overall the quality of this submission seems below the standard of NIPS.   #3. Originality The originality of SNIPER seems quite limited since it is an extension of SNIP [24]. Specifically, it can be regarded as a set of practices that are designed to speed-up of SNIP, which is slower than conventional pipelines for training object detectors due to its extensive use of image pyramid for scale-invariant learning. That is, the component improving detection accuracy by scale-invariant learning is the contribution of SNIP, not of SNIPER.   #4. Clarity The paper is readable, but sometimes difficult to follow due to the following issues: 1) The algorithm for drawing chips is not presented. The manuscript simply enumerates a few conditions to be a positive/negative chip, and no detail of the procedure is introduced. 2) Some techniques taken by SNIPER are not well justified. For example, it is unknown what is the advantages of taking the strategy introduced in line 164~165, why a ground-truth box that is partially overlapped with a chip is not ignored, and so on. 3) Often previous approaches are not cited, and their abbreviations are not given as well. For example, OHEM in line 169, SNIP in line 171, synchronized batch normalization in line 183, training on 128 GPUs in line 184, OpenImages v4 in line 195, and so on. >> Due to the above issues, this submission looks not ready for publication. Also, they damage reproducibility of SNIPER.    #5. Significance 1) Strength: The proposed techniques can improve the efficiency of training deep object detection networks significantly.  2) Weakness: The main idea of learning detectors with carefully generated chips is good and reasonable, but is implemented by a set of simple practical techniques. 3) Weakness: This is an extension of SNIP [24], and focuses mostly on speed-up. Thus its novelty is significantly limited.


A Bridging Framework for Model Optimization and Deep Propagation
Optimizing task-related mathematical model is one of the most fundamental methodologies in statistic and learning areas. However, generally designed schematic iterations may hard to investigate complex data distributions in real-world applications. Recently, training deep propagations (i.e., networks) has gained promising performance in some particular tasks. Unfortunately, existing networks are often built in heuristic manners, thus lack of principled interpretations and solid theoretical supports. In this work, we provide a new paradigm, named Propagation and Optimization based Deep Model (PODM), to bridge the gaps between these different mechanisms (i.e., model optimization and deep propagation). On the one hand, we utilize PODM as a deeply trained solver for model optimization. Different from these existing network based iterations, which often lack theoretical investigations, we provide strict convergence analysis for PODM in the challenging nonconvex and nonsmooth scenarios. On the other hand, by relaxing the model constraints and performing end-to-end training, we also develop a PODM based strategy to integrate domain knowledge (formulated as models) and real data distributions (learned by networks), resulting in a generic ensemble framework for challenging real-world applications. Extensive experiments verify our theoretical results and demonstrate the superiority of PODM against these state-of-the-art approaches.

Paper summary:    The paper proposed a learning based hybrid proximal gradient method for composite minimization problems. The iteration is divided into two modules: the learning module does data fidelity minimization with certain network-based priors; consequently the optimization module generates strict convergence propagations by applying proximal gradient feedback on the output of the learning module. The generated iterates were shown to be a Cauchy sequence converging to the critical points of the original objective. The method was applied to image restoration tasks with performance evaluated.      Comments:      The core idea is to develop a learning based optimization module to incorporate domain knowledge into conventional proximal gradient descent procedure. One issue with such a hybrid scheme is that by definition the output of the learning module is acceptable to the proximal gradient module only when the bounding condition in Eq.(9) is met. This then raises a serious concern that the learning module could have very limited effect on the iteration procedure if its output is frequently rejected by the proximal gradient module, i.e., the bound (9) is frequently violated. If that is the case, then the proposed method is no more than a proximal gradient method. So it will be helpful to provide more insightful explanations on the condition (9), e.g., to discuss how hard is it to be fulfilled.    Another main concern is the readability of paper. The model description part in Section 3 is rather abstract and hard to follow in general. To gain better intuition of method, the reviewer suggests providing one illustrating example or two such as those appeared in Section 5 to aid model elaboration.   The convergence analysis falls on the weak side: the procedure was only shown to converge asymptotically to critical points, using fairly standard proof skills in proximal gradient analysis. The rate of convergence remains largely unknown, even for convex cases.   Strong points:  + The attempt to bridge model optimization and deep propagation is somewhat interesting and novel.  + Numerical study is extensive.  Weak points:   - The motivation and significance of this work are not fully justified in text and mathematical analysis.  - Some important technical details about the proposed algorithms are missing (see the comments above). - The writing quality needs to be significantly improved.   === Updated review after author response ===  I further checked the technical proofs in more details and spot a seemingly fatal flaw in the proof of Proposition 1 (Line 46 - 47 of the supplement): i$x_G^l$ is claimed as a global minimizer of (10). However, as far as I can tell, $x_G^l$ is no more than a first-order stable point of (10). Since the primal interest of this work is on nonconvex problems, it is obvious false to claim that a stable point is a global minimizer.   In the meanwhile, after reading the author response, I found most of my major concerns were not clarified in a satisfying way. Particularly, the following issues are still there:  1. The problem statement is quite confusing: the original optimization problem is defined in Eq. (1) but the algorithms are designed for solving (2). The connection between these two formulations is not well explained. In particular, the inner product error term in Eq. (2) does not make sense to me.   2. Unless the authors can show that the bounding condition (9) is eventually always true in iteration, the proposed algorithm essentially reduces to a standard PG descent procedure.   3. The convergence analysis turns out to be weak and of limited novelty: it mimics largely the conventional PG analysis without introducing any conceptually new proof techniques and/or results.   All in all, I still have serious concerns about this submission and vote strongly for reject.


Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically BalancedS
We study the implicit regularization imposed by gradient descent for learning multi-layer homogeneous functions including feed-forward fully connected and convolutional deep neural networks with linear, ReLU or Leaky ReLU activation. We rigorously prove that gradient flow (i.e. gradient descent with infinitesimal step size) effectively enforces the differences between squared norms across different layers to remain invariant without any explicit regularization. This result implies that if the weights are initially small, gradient flow automatically balances the magnitudes of all layers. Using a discretization argument, we analyze gradient descent with positive step size for the non-convex low-rank asymmetric matrix factorization problem without any regularization. Inspired by our findings for gradient flow, we prove that gradient descent with step sizes ⌘ t " O´t´p 1 2` q( 0 † § 1 2 ) automatically balances two low-rank factors and converges to a bounded global optimum. Furthermore, for rank-1 asymmetric matrix factorization we give a finer analysis showing gradient descent with constant step size converges to the global minimum at a globally linear rate. We believe that the idea of examining the invariance imposed by first order algorithms in learning homogeneous models could serve as a fundamental building block for studying optimization for learning deep models.

Edit after author feedback, I do not wish to change my score: - To me balanced positive quantities is not only about their difference. They should have similar order of magnitude, the difference between 1e-13 and 1 is pretty small but they are clearly unbalanced. - The author put a lot of emphasis on boundedness of iterates suggesting connections between their result and boundedness. - The authors do claim more than small difference between consecutive layer weight norm: lines 160 to 163. - I still think that writting down that a polynomial is not smooth in a math paper is not professional. - The author do abuse O notation: it involves a "forall" quantifier on a logical variable: en.wikipedia.org/wiki/Big_O_notation The same goes for "Theta" and "poly" notation. None of the statements of the paper involving these notations has this feature: the variables epsilon, d, d1 and d2 are fixed and are never quantified with a "forall" quantifier. The fact that  a notation is standard does not mean that it cannot be misused.    The authors consider deep learning models with a specific class of activation functions which ensures that the model remains homogeneous: multiplying the weight of a layer by a positive scalar and dividing the weights of another layer by the same amount does not change the prediction of the network. This property is satisfied for relu types activations. A consequence is that the empirical risk is not coercive, a common assumption in convergence analysis of first order methods. In order to circumvent this issue, the authors claim that gradient descent algorithm has a regularization effect, namely that it tends to ballance the magnitude of the weights of each layers such that they remain roughly of the same order. The main results are as follows: - The authors prove that difference of norm squared of consecutive losses is a constant over continuous (sub)-gradient trajectories for relu networks. - The authors use this intuition to provide results for matrix factorization without constraint: convergence to solution for decaying step size and convergence to a solution for constant step size in the rank 1 setting.    Main comments: The abstract and introduction are very motivating and attractive. The idea of looking at invariances is nice and the balancing effect is a neat concept. Unfortunately, the content of the paper does not really reflect what the author claim in the first place: - The continuous time result is not really balancing anything and the authors clearly over interpret what they obtained. - 6 / 8 of the paper is dedicated to continuous subgradient trajectories in deep learning but 1 page of technical proof is dedicated to it (counting duplication) and 9 pages are dedicated to matrix factorization.  While the arguments look correct to me, it looks like the main content of the paper is about gradient descent for unconstrained matrix factorization and the connection with deep learning, despite being a nice observation looks a bit like hand waving. Some elements of the mathematical presentation could be improved.     Details:  $$ About "auto-balancing" $$ The abstract says "This result implies that if the weights are initially small, gradient flow automatically balances the magnitudes of all layers" This is wrong. Nothing in Theorems 2.1 and 2.3 and Corollary 2.1 says that the weights will adjust and balance. Nothing even says that they should be bounded. The difference remains constant, but the weights could well diverge. The paragraph after corollary 2.1 affirms without any supporting arguments that the proposed result explains convergence. In particular, line 161-162 is wrong even in the simple case of matrix factorization. The fact that two numbers a and b satisfy a-b small does not mean that a or b are small. It seems to me that Theorem 2.1 is actually a special case of Theorem 2.3. Why giving a detailed proof for both of them then?    $$ Why Gaussian random initialization? $$ Theorem 3.1 and 3.1 are stated with high probability over the random Gaussian initiailization. This is by no mean sufficient. Is 3/4 a high probability? Do the authors mean that it can be arbitrarily close to 1? How to tune the parameters then? Without a proper quantification, the result is of very little value. This can also be seen in the proof of Lemma 3.1, line 453, the proper way to present it would be to say which event is considered, to condition on that event and to estimate the probability of this event to be observed. Furthermore, I do not understand why the authors need Gaussian initialization. The main proof mechanism is that if initilization is close enough to the origin, then iterates remain bounded. This has nothing to do with Gaussianity. The authors could choose uniformly in a compact set, a hypercube is easy to sample from for example. The proof arguments would follow the same line but hold almost surely over the random initialization. The authors would avoid the non quantitative high probability statement and the incorect use of epsilon / poly(d).   $$ Confusion about smoothness $$ There is a confusion throughout the paper about the notion of smoothness: - Line 35: f is not smooth. - Line 105: the objective is not smooth - Line 106: balancing implies smoothness - Line 162: the objective may have smoothness - Line 402: we prove that f is locally smooth  This is completely misleading and mixes two important notions: - Smoothness is the property of being continuously differentiable, eventually several times, the more times, the more smooth (see wikipedia "smoothness" math page). - Optimization and ML people introduced notions of \beta-smoothness, which is just Lipschitz continuity of the gradient.  The two notions are complementary. From an analysis point of view, being non smooth is the contrary of being smooth: not differentiable or extended valued for example. However being non-smooth is not equivalent to "not being beta-smooth" or more precisely "not being beta-smooth for any beta". To my understanding the appropriate notion of "not being beta-smooth for any beta" is that of stiffness (see for example Convex Analysis and Minimization Algorithms I: Fundamentals, Jean-Baptiste Hiriart-Urruty, Claude Lemarechal page 375), that is smooth but with a modulus so big that it looks like nonsmooth numerically. This notion is pretty common in numerical analysis (see "Stiff equation" on wikipedia).  The discussion in the main text is absolutely missleading: f is not smooth? f is a polynomial, it is infinitely smooth but it does not have a global smoothness modulus. Balancing implies smoothness? Smoothness is already here, twice continuous differentiability implies beta-smoothness on any bounded set. We  prove local smoothness for f? As a polynomial, f is locally smooth, this does not require a proof, the authors actually compute a smoothness modulus.     $$ Is the statement 5 = O(4) true or false? $$ In Theorem 3.1, the authors use N(0, \epsilon / poly(d)). Would 10 be $\espilon / poly(d)$ for example? Similarly in 266, $d_1 = \theta(d_2)$, would $1000$ be $\theta(500)$? or not? Asymptotic notations are asymptotic because they describe limiting behaviour which means stabilisation when some parameter increases or decreases. In the first statement, epsilon and d are fixed, similarly, d_1 and d_2 are fixed in the second statement. In addition to be mathematicaly meaningless, this provides no clue about how to choose variance in Theorem 3.1.    $$ Minor remarks $$ - Notations the notation [.] is both used for the elements of a vector and for a set of integers and this sometimes makes reading not so easy. - Line 139, may be the authors could mention that the only homogeneous functions in one variable are more or less Relu like functions (linear when restricted to R+ and R-). - Equation (5), why should a solution of the differential inclusion exist, why this problem is well posed? - Line 197, it would be nice to have a more precise reference to a particular result and explain why this chain rule should hold for the specific situation considered in this work. - Equation after line 206, an equality is missing. - More explaination would be welcome after line 207. - Line 291, connections between optimization and ODE have been known for a while and investigated extensively in the early 2000s by the mathematical optimization community, in particular Hedy Attouch and his team. - Line 381, typo, theorem 2.3 - Line 435, argument is missing for boundedness of V - Equation (16), there is a typo. - I did not have time to check the rank 1 case proof in the appendix.


Communication Compression for Decentralized Training
Optimizing distributed learning systems is an art of balancing between computation and communication. There have been two lines of research that try to deal with slower networks: communication compression for low bandwidth networks, and decentralization for high latency networks. In this paper, We explore a natural question: can the combination of both techniques lead to a system that is robust to both bandwidth and latency?
Although the system implication of such combination is trivial, the underlying theoretical principle and algorithm design is challenging: unlike centralized algorithms, simply compressing exchanged information, even in an unbiased stochastic way, within the decentralized network would accumulate the error and fail to converge. In this paper, we develop a framework of compressed, decentralized training and propose two different strategies, which we call extrapolation compression and difference compression. We analyze both algorithms and prove both converge at the rate of O(1/ √ nT ) where n is the number of workers and T is the number of iterations, matching the convergence rate for full precision, centralized training. We validate our algorithms and find that our proposed algorithm outperforms the best of merely decentralized and merely quantized algorithm significantly for networks with both high latency and low bandwidth.

The paper presents two approaches to combining decentralized learning with model compression. Both method use gossip communication over a given network of nodes and model averaging. Before sharing the models with other nodes, they are compressed by either by using an unbiased compression on the difference of the model to the last version (the algorithm is called DCD-PSGD), or by an unbiased compression of a linear extrapolation of the model using the last version (ECD-PSGD). For both approaches, their convergence rate is theoretically analyzed. The combination of decentralized learning and model compression allows to handle networks with both low bandwidth and high latency. This claim is substantiated by an empirical evaluation.   Quality: The idea of using compression in decentralized learning seems natural but interesting and the results appear to be useful. The technical content is well-presented and sound, the proofs appear to be correct. The claims of the paper are supported by both the theoretical analysis and the experiments. There is one minor issue I have. In Section 4.1 the paper claims a linear speedup of the DCD-PSGD algorithm (and the same for ECD-PSGD in Section 4.2). This definitely holds as long as the computational cost of calculating the average is in O(1), which is true as long as the network has an upper bounded node degree. However, for a fully connected network, the cost for calculating the average at each node would be in O(n), which is a constant overhead. This should reduce the speedup to sublinear. If this is correct, I would ask the authors to add an assumption, like an upper bounded node degree of the network, where the bound does not depend on the network size.   Clarity: The paper is very well-written and clearly structured. The theoretical parts are comprehensible. The paper is overall a pleasure to read. There are some minor issues with English style and grammar, so I would ask the authors to have the paper proof-read. There are also some minor points and typos: - In Section 4.1 "Consistence with D-PSGD": I guess, ECD-PSGD should be DCD-PSGD, right?  - In Section 4.2, Eq. (3) there is a t missing before x_t^(j) - In Section 5: it would be helpful to know which exact compression operator was used - In Section 5.4: blackucing should be reducing - In Appendix A.2, proof of Lemma 7: in the equation after "Therefore it yields", I don't see a difference between the lines 3 and 4. I guess one is a copy artifact.  I would also like to ask the authors to check the formatting of the paper, as it seems to be a bit non-standard: the abstract is in italic, the placement of Figure 1 and the Definitions and Notations subsection is unusual, and there are weird line spaces (e.g., 226 to 227).  Originality: Both decentralized learning by model averaging and communication compression have been studied before, as the authors admit, but their combination is relatively new (it was discussed in an arxiv paper of Konecny et al. "Federated Learning: Strategies for Improving Communication Efficiency" that is not yet published, but there it is only studied empirically). Moreover, they are to the best of my knowledge the first to theoretically analyse this approach. Thus, the main contribution of the paper are two novel and theoretically sound algorithms for decentralized learning with compression.   Significance: Even though not groundbreaking, I really liked the paper. The algorithms it proposes seem to be useful for decentralized machine learning in many real-world application. The algorithms are proven to be theoretically sound and the empirical evaluation indicates the usefulness of the approach.   The author's response has addressed my concern about the assumption of O(1) runtime complexity for calculating the average.


Speaker-Follower Models for Vision-and-Language Navigation
Navigation guided by natural language instructions presents a challenging reasoning problem for instruction followers. Natural language instructions typically identify only a few high-level decisions and landmarks rather than complete lowlevel motor behaviors; much of the missing information must be inferred based on perceptual context. In machine learning settings, this is doubly challenging: it is difficult to collect enough annotated data to enable learning of this reasoning process from scratch, and also difficult to implement the reasoning process using generic sequence models. Here we describe an approach to vision-and-language navigation that addresses both these issues with an embedded speaker model. We use this speaker model to (1) synthesize new instructions for data augmentation and to (2) implement pragmatic reasoning, which evaluates how well candidate action sequences explain an instruction. Both steps are supported by a panoramic action space that reflects the granularity of human-generated instructions. Experiments show that all three components of this approach-speaker-driven data augmentation, pragmatic reasoning and panoramic action space-dramatically improve the performance of a baseline instruction follower, more than doubling the success rate over the best existing approach on a standard benchmark.

This paper builds upon the indoor vision and language-grounded navigation task and sequence-to-sequence model described in (Anderson et al, 2017), by introducing three improvements: 1) An encoder-decoder-like architecture, dubbed “speaker-follower” model, that not only decodes natural language instructions into a sequence of navigation actions using seq2seq, but also decodes a sequence of navigation actions and of image features into a sequence of natural language instructions using a symmetric seq2seq. That speaker model can then be used for scoring candidate routes (i.e., candidate sequences of images and actions) w.r.t. the likelihood of the natural language instruction under the speaker model. This enables a form of planning for the seq2seq-based agent. 2) That same speaker architecture can be used to generate new natural language descriptions for existing or random-walk generated trajectories through the environment, and thus augment data for training the follower. 3) Instead of considering 60deg field-of-view oriented image crops of the panoramic images as observations for the agent, entailing repetitive rotations to align the agent with the target destination, the agent is presented directly with a 360deg view, and uses an attention mechanism to predict where to look at, then compute the probability of going there in the image. The image and motion are decomposed into 12 yaw and 3 pitch angles.  The authors achieve state-of-the-art performance on the task and do a good ablation analysis of the impacts of their 3 improvements, although I would have liked to see navigation attention maps in the appendix as well. Improvement #3 (panoramic action space) is a significant simplification of the task; luckily the NE and SR metrics are unaffected.  It is worth noting that the idea of using language prediction for navigation in a grounded 3D environment has already been introduced by Hermann et al, 2017: "Grounded language learning in a simulated 3D world”.  Minor remark: The abstract is difficult to understand without reading the paper (both the “pragmatic reasoning” “panoramic action space” could be reformulated as model-based planning and visual attention-based navigation in panoramic images).  In summary: Quality: overall well presented paper that contributes to the development of language-grounded visual navigation agents Clarity: good but reformulate the abstract Originality: rather incremental work, missing reference on language generation for language-grounded agent Significance: good, state-of-the-art results on the Matterport dataset  Update: I have read the author's rebuttal and updated my score accordingly from 6 to 7.


How To Make the Gradients Small Stochastically: Even Faster Convex and Nonconvex SGD *
Stochastic gradient descent (SGD) gives an optimal convergence rate when minimizing convex stochastic objectives f (x). However, in terms of making the gradients small, the original SGD does not give an optimal rate, even when f (x) is convex. If f (x) is convex, to find a point with gradient norm ε, we design an algorithm SGD3 with a near-optimal rate O(ε −2 ), improving the best known rate O(ε −8/3 ) of [17]. If f (x) is nonconvex, to find its ε-approximate local minimum, we design an algorithm SGD5 with rate O(ε −3.5 ), where previously SGD variants only , 14, 30]. This is no slower than the best known stochastic version of Newton's method in all parameter regimes [27].

This work studies convergence rates of the gradients for convex composite objectives by combining Nesterov’s tricks used for gradient descent with SGD. The authors provide three approaches which differ from each other only slightly and they provide the convergence rates for all the proposed approaches. My comments on this work are as follow:   1. It is indeed important to study convergence rates of gradients especially for non-convex problems. The authors motivate the readers by mentioning this but they assume convexity in their problem set-up. It would have been fine if they were comparing other convergence rates for convex problems as well but I am not sure the existing results they discussed are defined for convex. For instance the work in citation 11 discusses the rates for nonconvex functions. Also, I could not locate the rate \epsilon^{-8/3} in citation 11. I’ll appreciate if the authors clearly mention which result and what assumptions they used from citation 11 in their rebuttal. Same concern applies to the results in citation 15.  2. My general objection about the paper is its representation and organization. I found it really hard to follow the main motivation from a stack of lemmas, corollaries and theorems. I’d suggest authors to narrow their focus to maybe one approach but discuss it in detail. Abstract should also be made more clear. It is not obvious that the manuscript only focuses on convex problems just by reading the abstract. A section of a discussion or a conclusion would also make the manuscript more readable.  3. What is the relationship between \epsilon and \delta between lines 41 and 42?  4. I don’t really understand what “replacing the use of inequality” means in lines 50-51.  5. I am not sure the statement in line 113 about “online” is correct. I understand the term “online” could be used for different things in machine learning but I think it is important to clarify what authors really mean by online setting in this work. Do they mean batch training which requires the entire training data or training with data which appears in a sequential order.  Overall, although the work looks interesting in terms of some theoretical convergence results, I think there is still some room to make the manuscript more convincing by working on the representation.


Sparse Attentive Backtracking: Temporal Credit Assignment Through Reminding
Learning long-term dependencies in extended temporal sequences requires credit assignment to events far back in the past. The most common method for training recurrent neural networks, back-propagation through time (BPTT), requires credit information to be propagated backwards through every single step of the forward computation, potentially over thousands or millions of time steps. This becomes computationally expensive or even infeasible when used with long sequences. Importantly, biological brains are unlikely to perform such detailed reverse replay over very long sequences of internal states (consider days, months, or years.) However, humans are often reminded of past memories or mental states which are associated with the current mental state. We consider the hypothesis that such memory associations between past and present could be used for credit assignment through arbitrarily long sequences, propagating the credit assigned to the current state to the associated past state. Based on this principle, we study a novel algorithm which only back-propagates through a few of these temporal skip connections, realized by a learned attention mechanism that associates current states with relevant past states. We demonstrate in experiments that our method matches or outperforms regular BPTT and truncated BPTT in tasks involving particularly longterm dependencies, but without requiring the biologically implausible backward replay through the whole history of states. Additionally, we demonstrate that the proposed method transfers to longer sequences significantly better than LSTMs trained with BPTT and LSTMs trained with full self-attention.

The authors augment an RNN with skip connections in time, that are sparsely gated by learnable attention. This allows to reap the benefits of full BPTT while effectively using only truncated BPTT.  While other forms of attention-gated skip connections in time have been suggested before, to which the authors compare, here the authors looked at sparse (still differentiable) retrieval where only a few top memory entries are selected, enabling the benefits of backpropagating over only a few selected earlier states.  Overall, I think this work is very significant, both for enabling faster implementations of BPTT when considering long time horizons, but also for suggesting future directions for how the brain might perform credit assignment and for pointing out further brain strategies / biases to employ in machine learning. With some clarifications / changes as below, I recommend the acceptance of this article for NIPS.  1. In lines 58-60, the authors say that BPTT would require "playing back these events". However, it is not so simple. The error must be backpropagated using the same recurrent weights, and combined with the activities and derivatives at previous time points to modify the weights. The authors should highlight this, else BPTT in the brain seems a simple matter of replaying past memories which is still bio-plausible, even though requires long time scales, whereas BPTT is far less bio-plausible, from perspectives of architecture and electro-physiology.  2. In lines 176 to 181, the authors show how a hard attention mechanism is implemented retrieving only the k_{top} memories. The authors should clarify how this memory access is differentiable, as the usage of soft-attention by other memory-augmented implementations was partly to enable differentiable access allowing backprop to pass through, and would allow the learning of what to attend to in the past given current hidden state \hat{h}. It is not clear here how the authors maintain differentiability while keeping hard attention.  Minor: l 156: "computationsl" l 223" "pixel-by-pixle"


Differentiable MPC for End-to-end Planning and Control
We present foundations for using Model Predictive Control (MPC) as a differentiable policy class for reinforcement learning in continuous state and action spaces. This provides one way of leveraging and combining the advantages of model-free and model-based approaches. Specifically, we differentiate through MPC by using the KKT conditions of the convex approximation at a fixed point of the controller. Using this strategy, we are able to learn the cost and dynamics of a controller via end-to-end learning. Our experiments focus on imitation learning in the pendulum and cartpole domains, where we learn the cost and dynamics terms of an MPC policy class. We show that our MPC policies are significantly more data-efficient than a generic neural network and that our method is superior to traditional system identification in a setting where the expert is unrealizable.

\documentclass[12pt,a4paper]{article} \usepackage[utf8]{inputenc} \usepackage{amsmath} \usepackage{amsfonts} \usepackage{amssymb}  \begin{document}  Consider rendering this review in LaTeX for a better experience.   \section{Paper summary}  This paper builds upon the recently proposed approach~\cite{amos2017optnet} to differentiating through quadratic programs (QP). More precisely, if $x^\star$ is the solution of a QP, the algorithm developed in~\cite{amos2017optnet} provides an efficient way to find derivatives of a scalar loss function $l(x^\star)$ with respect to the parameters of the QP, i.e., the matrices and vectors in the definition of the QP.  Since the linear quadratic regulator (LQR) problem is but a large QP, one can apply the methodology from~\cite{amos2017optnet}. Due to the chain structure of the computation graph for the LQR, the problem of finding derivatives of a loss function defined over optimal trajectories can be efficiently solved using Riccati recursion, which is summarized in Algorithm~1.  A non-trivial insight of the paper is to extend this approach to non-linear trajectory optimization problems. Popular sequential quadratic programming (SQP) solvers (including iLQR) proceed by solving a sequence of local QP approximations of the original non-linear trajectory optimization problem. Consequently, once converged, they output the `optimal' quadratic approximation of the original problem as a by-product. By applying Algorithm~1 to this surrogate QP, one can find the exact derivatives of a loss function defined on an optimal trajectory.  Another contribution of the paper is to incorporate box constraints on the controls by considering box-constrained QPs. The full procedure of differentiating through non-linear trajectory optimization is described in Algorithm~2.  The main advantage of the proposed algorithm is in providing constant-time backward pass for evaluating the derivative. Prior works would unroll all iterations of the SQP solver, thus the backward pass would scale linearly with the number of SQP iterations. Figure~1 shows the dramatic effect of the simplification that the proposed algorithm provides.  As a proof of concept, the authors apply Algorithm~1 to LQR system identification problem with time-invariant matrices $A$ and $B$. The unusual feature of the approach taken in this paper is that system identification happens as a consequence of minimizing quadratic imitation loss between actions of the optimal controller and the one with adjustable matrices $A$ and $B$. This experiment serves mainly to confirm that the equations derived in the paper are correct because, of course, this is not the most efficient way to do system identification.  Another set of simulated toy tasks shows that the algorithm can recover the system dynamics as well as the reward function from observed sequences of optimal controls $u_{1:T}(x; \theta)$ resulting from an initial state $x$.   \section{Paper review} Differentiation through LQR presented in this paper is a sound approach. However, I think a bit more care is needed in the non-linear trajectory optimization case. This point is explained in the following subsections.  \subsection{Title is misleading} The title reads ``differential MPC'', but the presented algorithm is actually ``differentiable non-linear trajectory optimization''. The controllers considered in the paper are of the form $u^\star_{1:T} = \pi(x_1; f_\theta, c_\theta)$ where $f_\theta(x,u)$ denotes the model of the dynamics and $c_\theta(x,u)$ denotes the model of the cost function. Here, $\pi$ is a function performing non-linear trajectory optimization. Derivatives that the authors propose are taken with respect to the optimal trajectory $\tau_{1:T} = (x^\star, u^\star)_{1:T}$.  On the other hand, an MPC controller has the form $u^\star_1 = \pi(x_1; f_\theta,c_\theta)$. So, only the first action returned by the trajectory optimizer gets executed. Then, the next state of the system is $x' = f(x, \pi(x; f_\theta,c_\theta))$. Differentiating a function of a trajectory generated in such a way with respect to $\theta$ is much more involved. But this is what one would expect to be called ``differentiable MPC''.  \subsection{LQR with box constraints} In Line 170, the authors write ``we use an LQR solver that zeros the appropriate controls''. What exactly is meant by zeroing appropriate controls? As far as I can tell, Riccati recursion does not work if one has box constraints. Instead, one has to solve a box-constrained QP at every time step  of a trajectory, starting at the end. But that sounds like a lot of computation. Is that how it is done? Or do you compute an unconstrained LQR and then clamp the controls?  \subsection{Local minima in non-linear optimization} How do you deal with the problem that the SQP solver delivers only a local optimum? The solution to which the solver converges in the forward run is usually highly dependent on the initial trajectory. That means the derivative computed according to the proposed method depends on the initialization of the SQP solver.  \subsection{Questions about experiments} In Line 234, the authors write ``we use LQR experts''. Do you mean `iLQR experts'? Pendulum and cartpole are nonlinear problems, so I assume `iLQR' is meant here.  Could you elaborate on the cost functions in your experiments and whether you use box constraints on the controls, please? More concretely, I am wondering if the pendulum is underpowered in your experiments.  What kind of model are you learning? Is it time-varying or time-invariant?  A partial explanation of why the experiments where both the dynamics and the cost function are learned are not as conclusive may lie in the non-uniqueness of the solution: there may be more than one combination of dynamics and cost that yield the same controls. As an example, consider 1-step LQR with 1D state and controls. The total cost is \begin{equation}   J = Qx_1^2 + Ru_1^2 + Qx_2^2. \end{equation} Substitute $x_2 = ax_1 + bu_1$ to get \begin{equation}   J = (Q + Qa^2)x_1^2 + (R + Qb^2) u_1^2 + 2 Q ab x_1 u_1. \end{equation} Optimizing it with respect to $u_1$, one finds \begin{equation}   u_1 = \frac{Qab}{R+Qb^2} x_1. \end{equation} From here, it is clear that if we allow both $\{a,b\}$ and $\{Q,R\}$ to vary, there is more than one combination of them that yields the same gain.   \subsection{Minor remarks} 1) Why are plots cut at $1\mathrm{e}3$ iterations? 2) Why is the car going left in the beginning in Figure 3? 3) Why is SysID cost oscillating in Figure 4? 4) Somwhere in the text, you call derivatives `analytic'. I think calling them `exact' would be more appropriate, since there is no closed-form formula, only an algorithm for computing them.   \section{Update after rebuttal} I would like to thank the authors for addressing my comments. In my opinion, the paper provides a solid contribution and I vote for accepting it.  \begin{thebibliography}{9}  \bibitem{amos2017optnet} Amos, B., \& Kolter, J. Z. (2017). Optnet: Differentiable optimization as a layer in neural networks. \textit{arXiv preprint arXiv:1703.00443.}   \end{thebibliography}   \end{document}


Bilinear Attention Networks
Attention networks in multimodal learning provide an efficient way to utilize given visual information selectively. However, the computational cost to learn attention distributions for every pair of multimodal input channels is prohibitively expensive. To solve this problem, co-attention builds two separate attention distributions for each modality neglecting the interaction between multimodal inputs. In this paper, we propose bilinear attention networks (BAN) that find bilinear attention distributions to utilize given vision-language information seamlessly. BAN considers bilinear interactions among two groups of input channels, while low-rank bilinear pooling extracts the joint representations for each pair of channels. Furthermore, we propose a variant of multimodal residual networks to exploit eight-attention maps of the BAN efficiently. We quantitatively and qualitatively evaluate our model on visual question answering (VQA 2.0) and Flickr30k Entities datasets, showing that BAN significantly outperforms previous methods and achieves new state-of-the-arts on both datasets.

The main contribution of this paper is a multimodal attention that links previous work of low-rank pooling, with Co-Attention methodologies, i.e., use of affinity matrix between modalities. Being competitive, the paper combines of many of the recent advancement, such as Down/Up attention, glimpses and counter model. Down/Up methodology with Co-Attention is new, and beautifully illustrated, showing an exact interaction between question elements and detected-object regions. The final model achieve SOTA results in for a very competitive task.  Clarity/Quality: The paper is well-written. contributions clearly laid out.  Originality: The paper inspired and closely follows work of Lu et. al. [18] parallel model combines with low-rank bilinear pool, such as MCB, MLB, MFH. Significance: The proposed approach demonstrates clear improvements for VQA task. Analysis of different attention methods proves the proposed attention is better than previous work.  Concerns: •Related work of the paper is not accurate. L123-124 claim that Co-Attention neglected interactions between modalities. The parallel version of [18] or [20] defiantly capture the interactions between the modalities via similarity matrix. The parallel version was ignored in Quantitative analysis. I think you should elaborate about the exact differences between your technique, and parallel-like techniques.   •Related work is also not comprehensive enough in my opinion. As attention paper, the discussion is limited only to very similar approaches. For instance, “Structured Attentions for Visual Question Answering” from ICCV17, solves multimodal attention as structure prediction, or ‘’High-Order Attention Models for Visual Question Answering`` from NIPS17, which also discuss Co-Attention, and suggests a network with combined unary and pairwise information (i.e., multiply interactions), which surpass [18], and was not mentioned.  •Fig 2. How exactly have you tested the different attentions? Did you replace only the attention unit? Did you tune the network with different architecture?  • why this model is avoiding overfitting, or able to use more glimpses?  Conclude:  I like the paper, though it feels a bit too concentrated on the competitive aspects, i.e. accuracy score. I am missing a more comprehensive justification and comparison to similar co-attention techniques. The attention unit is novel, but incremental from common co-attention practice. Nevertheless, the overall quality of the submission is good, with SOTA performance and a complete analysis for many design choices, therefore I’m in favor for acceptance.  After author response: I thank the authors for their response. I never intended to misjudge the significance of BAN results vs previous work, I was more confused about the important ideas of BAN versus other approaches that use bilinear attention. While the difference from  iteratively approaches for co-attention is obvious, I'm still confused about the difference from previous work that use bilinear attention. The paper only mention a very early approach by Xu and Saenko. The proposed low rank bilinear operation is not new, and was already suggested in the papers I mentioned.  Arguing that bilinear approaches are weak against iterative approaches does not align with the paper main argument. To progress the research in this direction,  I hope you will chose to clarify the importance of your approach for attention in the revised version, and looking forward to see the paper published at NIPS.