Differentially Private Contextual Linear Bandits
We study the contextual linear bandit problem, a version of the standard stochastic multi-armed bandit (MAB) problem where a learner sequentially selects actions to maximize a reward which depends also on a user provided per-round context. Though the context is chosen arbitrarily or adversarially, the reward is assumed to be a stochastic function of a feature vector that encodes the context and selected action. Our goal is to devise private learners for the contextual linear bandit problem. We first show that using the standard definition of differential privacy results in linear regret. So instead, we adopt the notion of joint differential privacy, where we assume that the action chosen on day t is only revealed to user t and thus needn't be kept private that day, only on following days. We give a general scheme converting the classic linear-UCB algorithm into a joint differentially private algorithm using the tree-based algorithm [10,18]. We then apply either Gaussian noise or Wishart noise to achieve joint-differentially private algorithms and bound the resulting algorithms' regrets. In addition, we give the first lower bound on the additional regret any private algorithms for the MAB problem must incur.

Summary:  In this paper the authors study how to achieve differential privacy in the contextual linear bandits setting. The authors first show that the standard notion of privacy in this setting leads to the linear regret and then they adopt the more relaxed notion of joint differential privacy. They propose the Linear UCB with changing perturbations algorithm as a general framework for achieving this joint differential privacy while achieving a reasonable regret guarantees. The authors provide general regret bounds for their proposed algorithm under some assumptions in section 3. Later in section 4, they explain two different ways for perturbing the estimations in order to achieve joint differential privacy and they combine them with the results in section 3 to provide regret bounds for them. Finally, in section 5 of the paper, the authors provide some lower bounds and show that any algorithm that achieves differential privacy is required to incur an extra regret term.   Evaluation:  The paper is very well-written and well-motivated. This work draws an interesting connection between the contextual linear bandit literature and the differential privacy literature. In order to achieve algorithms that can provide privacy and low-regret simultaneously, the authors: (1) derive general regret bounds for their proposed algorithm (which can be found in Proposition 4) and (2) show that with reasonable perturbations the privacy can also be achieved. From the technical analysis, (1) is an easy extension of analysis in [1] and for (2) authors use the result in [2] or the concentration bounds on \chi^2 distribution. Therefore, the technical analysis does not seem to be very different that what existed in the literature.   On the other hand, the derivation of lower bounds seem to be pretty novel. Also, I like the fact the proposed algorithm is very general and authors show two different ways to make it work. Finally, I want to emphasize the clarity of this work.  Gathering all these points together, I recommend this paper to be accepted to the conference.  One slight comment: the citations [28] and [29] in the paper are the same.  [1] Abbasi-Yadkori, Y., Pál, D., & Szepesvári, C. (2011). Improved algorithms for linear stochastic bandits. In Advances in Neural Information Processing Systems (pp. 2312-2320). [2] Sheffet, O. (2015). Private approximations of the 2nd-moment matrix using existing techniques in linear regression. arXiv preprint arXiv:1507.00056.


Out of the Box: Reasoning with Graph Convolution Nets for Factual Visual Question Answering
Accurately answering a question about a given image requires combining observations with general knowledge. While this is effortless for humans, reasoning with general knowledge remains an algorithmic challenge. To advance research in this direction a novel 'fact-based' visual question answering (FVQA) task has been introduced recently along with a large set of curated facts which link two entities, i.e., two possible answers, via a relation. Given a question-image pair, deep network techniques have been employed to successively reduce the large set of facts until one of the two entities of the final remaining fact is predicted as the answer. We observe that a successive process which considers one fact at a time to form a local decision is sub-optimal. Instead, we develop an entity graph and use a graph convolutional network to 'reason' about the correct answer by jointly considering all entities. We show on the challenging FVQA dataset that this leads to an improvement in accuracy of around 7% compared to the state of the art.

This paper proposes a neural architecture for Fact Based VQA (introduced in [1]), an extension of VQA requiring leveraging of facts from an additional knowledge base. Given an image and a question about it, the model further retrieves the top-100 facts from the KB based on cosine similarity between an averaged Glove embedding representation between question and fact. Further, the facts are embedded as entities in a graph, and each entity is processed via a Graph Convolutional Network and fed to a multilayer perceptron trained to be verified as the correct answer. Results, comparisons against baselines, and ablations are presented on the FVQA dataset.  Strengths  – This is a well written and interesting work that approaches an important problem and presents a simple and effective model for the same – The approach significantly outperforms the previous state of the art on Fact based VQA, and the ablations studied and error contribution analysis validate the modeling choices effectively. In general, the experimental results are quite thorough and promising, and the qualitative results are compelling and provide good insights. – The paper provides sufficient experimental detail and the results should be possible to reproduce  Weaknesses/Questions  – In the error decomposition, the fact-retrieval error can be controlled for by setting a higher threshold at the cost of more computation. Have the authors experimented with retrieving a larger set of facts, or were diminishing returns observed after the threshold of 100? – Some additional analysis of the typical structure of the induced graph would be interesting to see – how sparse is the adjacency matrix on average? L95 states: ‘By stacking multiple layers, we are able to gather information from nodes further away’ – some analysis/empirical evidence of whether the model can pick up on longer range relationships in the graph, would also be interesting to include and would strengthen the paper.  [1] Wang, Peng, Qi Wu, Chunhua Shen, Anthony Dick, and Anton van Den Hengel. 2017. “FVQA: Fact-Based Visual Question Answering.” IEEE TPAMI  ----  I have read through the author response and reviews, and am happy to keep my accept rating. The paper is well written and presents a novel formulation and strong results for the FVQA task. The analysis and experimentation is thorough, and in my opinion the authors have provided a strong rebuttal to most concerns raised in the reviews.


Representation Balancing MDPs for Off-Policy Policy Evaluation
We study the problem of off-policy policy evaluation (OPPE) in RL. In contrast to prior work, we consider how to estimate both the individual policy value and average policy value accurately. We draw inspiration from recent work in causal reasoning, and propose a new finite sample generalization error bound for value estimates from MDP models. Using this upper bound as an objective, we develop a learning algorithm of an MDP model with a balanced representation, and show that our approach can yield substantially lower MSE in common synthetic benchmarks and a HIV treatment simulation domain.

The paper studies how to fit a model for the purpose of off-policy policy evaluation (OPPE), that is, estimating the value function of a policy using data from a different policy. Instead of using a simple loss function under the data distribution, the proposed method addresses the distribution shift issue by incorporating data distribution under target policy, approximated by importance sampling. Doing so alone, however, results in data sparsity, so the ultimate objective also includes the error of the behavior policy to stabilize and regularize learning. Finite sample generalization error bounds are proved for the proposed approaches, and empirical evaluation compares the approaches to other OPE methods for value function and expected return estimations.   I like the research question of how to estimate a model for OPE that takes into account the distribution shift issue. The ideas in the paper make a lot of intuitive sense. The paper is well written and pleasant to read, and the experiments show some promising results.  Now let me turn to weaknesses. The first major weakness is that, the theory part does not really support the proposed method well (despite the amount of maths involved). One thing I want to highlight here is that, as far as I understand, the introduction of MSE_mu in the objective is not justified at all by the theorems. Namely, if we get rid of MSE_mu we should be able to obtain *better* upper bounds; see Bullet 4. Neither is it justified empirically.  It should be noted that some of the theoretical caveats are addressed by the positive empirical results. That said, the experiment setup on the Sepsis dataset, where the most promising results are shown, is very questionable; see bullet 6 below. Put it bluntly, I would be more comfortable accepting the paper if this section is simply dropped (unless the methodology can be rigorously justified).   Overall this is a borderline paper to me, and I temporarily lean very slightly towards rejection. That said, I really like the research topic and find the ideas of the paper quite nice, so I am ready to be convinced by the authors and the other reviewers (esp. on the Sepsis data issue) and change my mind.  Detailed comments:  1. Why is E_{s0}[(...)^2] the objective to consider at all? In particular, why do we only consider the initial state distribution of s0, and ignore the later time steps? Details follow:  (a) s0 can be deterministic w.l.o.g (just add a fake initial state that transitions to the true initial distribution), in which case this objective is no different from the average value objective (APV, line 43). So for the objective to make a difference, we are at the mercy of the environment---that its initial state distribution is well spread somehow. When is this the case?  (b) Which objective is right really depends on what you want to do with the estimated value function. In the experiment section, the best use case of the proposed method is still average value estimation. If this is the ultimate use case, *for the purpose of theoretical development*, you should start with the average value objective, and then explain why using the proposed objective as a *surrogate* actually leads to better average value estimations.   (c) Now there is definitely more we can do with a value function other than turning it into a scalar return---for example, policy improvement. But if policy improvement is the goal, we need to consider distributions of states beyond the initial step (see e.g., the CPI paper by Kakade and Langford).   2. Eq.4 is said to be an upper bound on the OPPE error. Isn't OPPE error simply the first term? Calling Eq.4 an upper bound is mathematically correct but perhaps conceptually misleading: for example, we say that hinge-loss is an upper bound on 0-1 loss, but seldom say that the L-2 regularized objective is an upper bound on the empirical risk (note it could upper bound population risk as in SRM and that's legit, but that's not what's going on in this paper). Overall I feel that the introduction of the MSE_mu is more heuristic than it sounds to be.  The looseness of Eq.4 also factors into Thm 2. (The first step in the equation between Line 225 and 226 performs this relaxation.) This brings to my next point...  3. One thing that is missing is the comparison to baseline approaches in theory. What guarantees can we give for simply fitting the model on data distribution? Is the guarantee in the current paper better than that of the baseline approach?   4. As far as I understand, the current analyses do not justify the introduction of MSE_mu, since Thm 2 relaxes MSE_pi to MSE_pi + MSE_mu and bounds them separately. If we just drop MSE_mu, we should get better upper bounds? Please comment on this in the rebuttal.  5. The empirical evaluation is thorough and shows promising results. However, given the caveat in bullet 3, I think it's very important to perform an ablation study and see how the method performs without the MSE_mu part.  6. For the Sepsis data, I am not sure why the artificial split of data is valid. For example, if the actions made during data collection depend on the patient's characteristics, such a split of data could give us two subpopulations that are very different in distribution, breaking the basic assumptions of off-policy evaluation. Unfortunately, I cannot find any description of the behavior policy in the paper.   More generally speaking, what this section does is to "mimic" an A/B test using observational data. There is extensive literature related to this from Stats/Econ (e.g., instrumental variables), and this usually needs to be done with extreme care. While I am not an expert on this, I feel that the paper's treatment here is surprisingly casual. Also note that it wouldn't be much of a problem if someone does this in the algorithm: if an algorithm does something wrong it will perform poorly, and the poor performance will be observed correctly. The problem is more serious here because the methodology is used to form the groundtruth, hence any mistake would affect the validity of the conclusions we draw from the results.    Minor - "Individual" vs "average" policy value: the terminologies here are somewhat unconventional. Perhaps consider "value function" vs "expected return" of a policy.  - References need a very careful pass: [5] and [9] should use conference versions instead of arXiv, author name in [8], [20] has "\phi" in it, etc. - Line 197, space before the comma.  ================================== After rebuttal: Thanks for the rebuttal. I am happy to learn that the authors agree with quite a few of my concerns and have proposed concrete revisions to address them. Despite that the revisions seem quite substantial, I am reasonably confident that the authors can implement them faithfully for camera-ready, so I am raising my score accordingly. Re the initial state issue, the authors' argument is in line with my 1(b) in review, so you might want to consider the alternative presentation I mentioned there.


Content preserving text generation with attribute controls
In this work, we address the problem of modifying textual attributes of sentences. Given an input sentence and a set of attribute labels, we attempt to generate sentences that are compatible with the conditioning information. To ensure that the model generates content compatible sentences, we introduce a reconstruction loss which interpolates between auto-encoding and back-translation loss components. We propose an adversarial loss to enforce generated samples to be attribute compatible and realistic. Through quantitative, qualitative and human evaluations we demonstrate that our model is capable of generating fluent sentences that better reflect the conditioning information compared to prior methods. We further demonstrate that the model is capable of simultaneously controlling multiple attributes.

The authors aim to achieve style transfer in text governed by controllable attributes. For this, authors propose a model which builds on prior works on unsupervised machine translation and style transfer. The proposed model uses reconstruction and back-translation losses. Authors propose to add denoising via dropping inputs and using an interpolated hidden representation. An additional adversarial loss is added to ensure the generated distribution of (hx,l) matches the input data distribution. An undesirable solution for D can be to simply use hx and hy, and ignore the attributes. To deal with this, (hx,l') is added as additional negative sample. Authors test their proposed model and training methods on  multiple datasets - sentiment change, Shakespearean English style transfer, active/passive/tense/etc. on Book Corpus dataset. Human evaluation is carried out in some cases.  An ablative study is also carried out to highlight the importance of different terms in the loss function.  While the proposed model and experiments seem interesting, there seems to be limited novelty in the used model compared to prior works.    Description of model and loss functions is not clear and seems incomplete at times. For example, in section 3.3, it seems that authors want to use adversarial loss to distinguish b/w 'real' and 'generated/fake' sentence-attribute tuples - shouldn't then the equation 4 use max over D and min over G?  Additionally, is there anything preventing decoder output states hx to encode information about attributes l without actually using it? If hx does include information about l, then isn't the task of D much easier? In such a situation, role of D is not clear. I would suggest authors to add more discussions and explanations for their model choices.  Most of the experiments seem convincing - the proposed method performs pretty well on a variety of datasets. 4.6 describes an important set of experiments wherein multiple attributes are changed - however this part misses on many details such as distribution of attribute values and number of sentences in data, and performance wrt content preservation. Discussions and analysis about the types of errors the model makes on the three datasets could be added for better understanding of the working of the model.   I am interested in understanding more about instructions provided for human evaluation, especially with respect to content preservation. Authors point out that annotators were asked 'whether the original and generated sentences are related by the desired property' - what is 'property' here? Is it same as attribute? In that case, this seems a bit ill-defined since 'related' is a very loose relation - can two wildly different sentences with negative sentiments can be said to be related wrt the sentiment attribute? How was the meaning of the term 'related' conveyed to the annotators? Authors could have provided more details or a screenshot of evaluation framework in the supplementary material.  Update: I have considered the authors' response, and have increased my score to 7.


Neural Nearest Neighbors Networks
Non-local methods exploiting the self-similarity of natural signals have been well studied, for example in image analysis and restoration. Existing approaches, however, rely on k-nearest neighbors (KNN) matching in a fixed feature space. The main hurdle in optimizing this feature space w. r. t. application performance is the non-differentiability of the KNN selection rule. To overcome this, we propose a continuous deterministic relaxation of KNN selection that maintains differentiability w. r. t. pairwise distances, but retains the original KNN as the limit of a temperature parameter approaching zero. To exploit our relaxation, we propose the neural nearest neighbors block (N 3 block), a novel non-local processing layer that leverages the principle of self-similarity and can be used as building block in modern neural network architectures. 1 We show its effectiveness for the set reasoning task of correspondence classification as well as for image restoration, including image denoising and single image super-resolution, where we outperform strong convolutional neural network (CNN) baselines and recent non-local models that rely on KNN selection in hand-chosen features spaces.

Update: I am somewhat convinced by the rebuttal. I will increase my rating, although I think that the quantitative improvements are quite marginal.  Summary: Authors propose a neural network layer based on attention-like mechanism (a "non-local method") and apply it for the problem of image restoration. The main idea is to substitute "hard" k-NN selection with a continuous approximation: which is essentially a weighted average based on the pairwise distances between the predicted embeddings (very similar to mean-shift update rule).  Although the paper is clearly written, the significance of the technical contributions is doubtful (see weaknesses), thus the overall score is marginally below the acceptance threshold.  Strengths: + The paper is clearly written. + There is a (small) ablation study. +/- I like the connection with latent variable models learning (L114), the high variance of the gradient estimators is a big problem. However, I would expect a deeper analysis of this issue, and in particular how does the "continuous deterministic relaxation" help this.   Weaknesses: - In general, very similar approaches have been widely used before (matching networks, memory networks, non-local neural networks to name a few), and the difference suggested by the authors (L71, label probabilities vs features), does not seem very significant. Effectively, instead of aggregating the labels of the neighbours authors aggregate the features of the neighbours.  - Authors clearly state that the "N3 block" can be used as a generic non-local module, yet the only problem they consider is image restoration.  - In Table 2, the results for k=1 are already better than the baselines. This actually hints that the performance gain might be for a reason different from applying Eq.10. - It is not entirely clear if the equivalence with hard k-NN for t -> 0 is really of practical importance: there are no details in the paper of what kind of values the network produces for the temperature.


Adapted Deep Embeddings: A Synthesis of Methods for k-Shot Inductive Transfer Learning
The focus in machine learning has branched beyond training classifiers on a single task to investigating how previously acquired knowledge in a source domain can be leveraged to facilitate learning in a related target domain, known as inductive transfer learning. Three active lines of research have independently explored transfer learning using neural networks. In weight transfer, a model trained on the source domain is used as an initialization point for a network to be trained on the target domain. In deep metric learning, the source domain is used to construct an embedding that captures class structure in both the source and target domains. In few-shot learning, the focus is on generalizing well in the target domain based on a limited number of labeled examples. We compare state-of-the-art methods from these three paradigms and also explore hybrid adapted-embedding methods that use limited target-domain data to fine tune embeddings constructed from sourcedomain data. We conduct a systematic comparison of methods in a variety of domains, varying the number of labeled instances available in the target domain (k), as well as the number of target-domain classes. We reach three principal conclusions: (1) Deep embeddings are far superior, compared to weight transfer, as a starting point for inter-domain transfer or model re-use (2) Our hybrid methods robustly outperform every few-shot learning and every deep metric learning method previously proposed, with a mean error reduction of 34% over state-of-the-art. (3) Among loss functions for discovering embeddings, the histogram loss (Ustinova & Lempitsky, 2016) is most robust. We hope our results will motivate a unification of research in weight transfer, deep metric learning, and few-shot learning.

The paper proposing to unite three distinct areas of transfer learning, namely, weight adaptation, few shot learning and metric learning. Weight adaptation is typically defined as fine-tuning a pre-trained network on a different but related problem. Few shot learning is defined as given a large labeled dataset, how can we further adapt when given a small set of examples for related but novel set of classes. Metric learning is defined as learning a metric space and typically tested in a few-shot retrieval setup in the literature. Hence, all these methods can be seen as handling different special case of same problem (called k-ITL in the paper). The paper experimentally shows that a trivial combination of two state-of-the-art method actually out perform all other methods for all three tasks. Hence, the main takeaway of the paper is necessity to consider these tasks in the same context and unite the literature.  STRENGTHS The main claim of the paper is significant, clear, sensible and surprisingly was hidden in plain sight for so long. Clearly, metric learning and few-shot learning are closely related problems and they have been studied by different communities using similar techniques. Considering the way that metric learning is typically evaluated (Recall@R for novel classes), it is reasonable to apply metric learning to few shot learning problems. Somewhat surprisingly this combination results in state-of-the art method which outperforms existing few-shot learning algorithms. Although the algorithmic contribution is straightforward, the work will be significant to both metric learning and few-shot learning communities.  The paper is written very clearly and it is easy to understand. Authors perform a thorough experimental study considering almost all factors. Although some factors like class imbalance in train and adaptation left unexplored, authors are upfront about them.   WEAKNESSES I personally think the paper does not do justice to weight adaptation. The proposed setup is only valid for classification using same kind of data (modality, appearance etc.) between training and adaptation; however, weight adaptation is a simple method which be used for any problem regardless of change of appearance statistics or even changing the type of the modality. In practice, weight adaptation is proven to be useful in all these cases. For example, when transferred from ImageNet to audio spectrograms or transferred from classification to object detection, weight transfer stays useful. The paper studies a very specific case of transfer learning (k-ITL) and it shows that weight adaptation is not useful in this specific setting. Whether the weight adaptation useful in remaining settings or not still remains as an open question. The authors should clarify that there are more problems in transfer learning beyond k-ITL and they only consider this specific case.  As a similar issue, the experimental setup only consider class distribution as a possible difference between training and adaptation. Clearly, low-level appearance statistics does not change in this case. It would be an interesting study to consider this since weight adaptation is shown to be handling it well. One simple experiment would be performing an experiment with training on tinyImageNet and adapting on CIFAR-10/100. This experiment would increase the robustness of the claims of the paper. Although it is little beyond of scope of k-ITL setup, I think it is necessary to be able to make the strong claim in L275-280.  MINOR ISSUES: - Averaging the accuracies over different tasks in Figure 6 does not seem right to me since going from 90% to 95% accuracy and going from 10% to 15% should ideally be valued differently. Authors should try to give the same plot for each dataset separately in addition to combining datasets.  - It is not direcly obvious that the penultimate layer will discard the inter-class relationship by orthogonalizing each class as stated in L281-L288. This statement is dependent on the type of the loss is used. For example, correctness of the statement is clear to me for L2 loss but not clear for cross-entropy. Either authors should provide some additional proof or justification that it is correct for all loss functions or should give more specifications.  In summary, paper is stating an important but hidden in plain-sight fact that few-shot learning and metric learning is closely related and metric learning methods can very well be used for few-shot learning. This unified approach results in a straightforward algorithm which improves the state-of-the-art significantly. Although there are minor issues with the paper, it is an important contribution to both metric learning and few-shot learning communities.  UPDATE: I have read the authors response and here are my comments on them: 1.a) Example citation for transfer from image to audio: Amiriparian et al., Snore Sound Classification using Image-Based Deep Spectrum Features. 1.b) Transfer beyond k-ITL: I also agree the proposed method should be able to handle this as well but unless we see an experiment on this, it is mostly a speculation. I think the other answers are satisfactory. I am keeping my score as 8.


Diverse Ensemble Evolution: Curriculum Data-Model Marriage
We study a new method "Diverse Ensemble Evolution (DivE 2 )" to train an ensemble of machine learning models that assigns data to models at each training epoch based on each model's current expertise and an intra-and inter-model diversity reward. DivE 2 schedules, over the course of training epochs, the relative importance of these characteristics; it starts by selecting easy samples for each model, and then gradually adjusts towards the models having specialized and complementary expertise on subsets of the training data, thereby encouraging high accuracy of the ensemble. We utilize an intra-model diversity term on data assigned to each model, and an inter-model diversity term on data assigned to pairs of models, to penalize both within-model and cross-model redundancy. We formulate the data-model marriage problem as a generalized bipartite matching, represented as submodular maximization subject to two matroid constraints. DivE 2 solves a sequence of continuous-combinatorial optimizations with slowly varying objectives and constraints. The combinatorial part handles the data-model marriage while the continuous part updates model parameters based on the assignments. In experiments, DivE 2 outperforms other ensemble training methods under a variety of model aggregation techniques, while also maintaining competitive efficiency.

This paper proposes a new technique for training ensembles of predictors for supervised-learning tasks. Their main insight is to train individual members of the ensemble in a manner such that they specialize on different parts of the dataset reducing redundancy amongst members and better utilizing the capacity of the individual members. The hope is that ensembles formed out of such predictors will perform better than traditional ensembling techniques.   The proposed technique explicitly enforces diversity in two ways: 1. inter-model diversity which makes individual models (predictors) different from each other and 2. intra-model diversity which makes predictors choose data points which are not all similar to each other so that they don't specialize in a very narrow region of the data distribution. This is posed as a bipartite graph matching problem which aims to find a matching between samples and models by selecting edges such that the smallest sum of edge costs is chosen (this is inverted to a maximization problem by subtracting from the highest constant cost one can have on the edges.) To avoid degenerate assignments another matching constraint is introduced which restricts the size of samples selected by each model as well. This will ensure that no model starves of samples. The two constraints are posed as matroid constraints on a utility which consists of three terms: 1. The sum of edge costs to be minimized (posed as a maximization) 2. a inter-model diversity term and 3. a intra-model diversity term. The last two terms are chosen from the family of submodular functions which are well-known to naturally model diversity and diminishing return properties.   Since the overall objective function results in a combinatorial (submodular) and a continuous optimization problem the proposed solution method alternates between the two. First the discrete optimization is solved to find a better matching and then the continuous optimization is run to train the models on the new matches.   A curriculum is designed for this process so that initially there is more weight on diversity terms and then reducing them slowly.   Experiments are presented on CIFAR10, CIFAR100, Fashion and STL10 datasets. Bagging and random initialization of DNNs are used as the training baselines. A number of inference schemes are demonstrated with the best performing ones being the Top-K NN-LossPredict which trains a NN to predict for a given sample which m models are the best ones.   On all four datasets DivE2 performs the best using mostly Top-K NN-LP as the inference method. On CIFAR-100 DivE2 + all average inference works the best.  Comments:  - The paper is generally well-written and easy to understand. Thanks!!  - The proposed method is interesting, novel and tackles a relevant problem.  - Major concerns:   1. Missing baselines: In addition to the bagging and random initialization training methods the natural other ones to try which try to incorporate diversity are [38] "Why M-heads are better than one..." and [not currently cited] "Stochastic MCL for Training Diverse Deep Ensembles, Lee et al.", NIPS 2016 and since the authors are not tackling structured prediction tasks but multi-class classification tasks: "Contextual Sequence Prediction with Application to Control Library Optimization" by Dey at al., RSS 2012 which also provides the theoretical groundings for [10].  2. Datasets and tasks: The authors are currently focussed on small multi-class datasets. It will be much stronger statement to show results on larger datasets like ImageNet and/or MS COCO or other non-vision large datasets for multi-class tasks. (Note that most of the related work in this area have demonstrated on structured prediction tasks but I understand that the scope of this paper focused on multi-class prediction problems.)  I am happy to be convinced why any/all of the above baselines are not suitable candidates for comparison to the proposed method.   3. List of relevant work which should be discussed and cited:   "Stochastic MCL for Training Diverse Deep Ensembles", Lee et al., NIPS 2016  "Contextual Sequence Prediction with Application to Control Library Optimization" by Dey at al., RSS 2012  "Learning Policies for Contextual Submodular Prediction", Ross et al. ICML 2013.   "Predicting Multiple Structured Visual Interpretations", Dey et al., ICCV 2015   "Confident Multiple Choice Learning", Lee et al., ICML 2017  - Minor concerns:  1. From the results presented in Table 2 it seems that Top-k NN-LP should always be used as the inference technique even with other training methods like BAG or RND. Is this assessment fair?   2. On CIFAR-100 all-average is performing the best although Top-K NN-LP is not that far behind. On a larger dataset like ImageNet or MS Coco will all average still do the best or do the authors think that Top-K NN-LP will prevail?


PAC-Bayes Tree: weighted subtrees with guarantees
We present a weighted-majority classification approach over subtrees of a fixed tree, which provably achieves excess-risk of the same order as the best tree-pruning. Furthermore, the computational efficiency of pruning is maintained at both training and testing time despite having to aggregate over an exponential number of subtrees. We believe this is the first subtree aggregation approach with such guarantees. The guarantees are obtained via a simple combination of insights from PAC-Bayes theory, which we believe should be of independent interest, as it generically implies consistency for weighted-voting classifiers w.r.t. Bayes -while, in contrast, usual PAC-bayes approaches only establish consistency of Gibbs classifiers.

**Summary and main remarks**          The manuscript investigates generalization bounds for classification trees, using a PAC-Bayesian approach. Such generalization bounds exist in the literature for two competing strategies to build classification trees: given a fully developed tree, (i) pruning and (ii) weighted majority vote on all subtrees. Pruning is by far the most popular strategy, since it enjoys computational and statistical guarantees. The main claim from the manuscript is that the authors devise a weighted majority vote strategy which has similar generalization bounds and a computational complexity bounded by the total nomber of nodes of the whole tree. The workhorse of this approach is the PAC-Bayesian theory, which allows to derive oracle generalization bounds for randomized classifiers sampled from a Gibbs potential. Up to my knowledge, the use of PAC-Bayesian tools to analyze classification trees is original.          Existing PAC-Bayesian bounds hold for randomized classifiers sampled from the Gibbs potential, and the passing to a Q-weighted majority vote (where Q is a distribution on the set of classifiers -- in the manuscript, this set is assumed finite) is not trivial, and is identified as an issue in earlier papers (e.g., Lacasse, Laviolette, Marchand, Germain and Usunier, 2007). The authors adapt an existing PAC-Bayesian bound from Thiemann, Igel, Wintenberger and Seldin (beware, the reference details are wrong), and the cornerstone of their analysis is stated in Proposition 2 (page 4). The crux is to twist the loss function to take into account the different nature of the classifier (randomized vs. aggregate / expectation). Proposition 2 is then a straightforward consequence of the previous bound from Thiemann et al. This is a simple yet elegant argument.          The authors then adapt their version of a PAC-Bayesian bound for Q-weighted majority vote to the case of trees and aggregates of subtrees: Theorem 1 (page 5). This oracle bound which is presented states that with arbitrarily high probability, the excess risk of the Q-weighted majority vote is upper bounded by the best (= smallest) excess risk of the subtrees, up to remainder terms of magnitude $\mathcal{O}(D_0 |\pi(T)|/(n\lambda) + \sqrt{n})$ *and* up to a multiplicative constant which is $L/(1-\lambda/2)$. This Theorem 1 bears my main concerns about the manuscript.          (i) why not optimize with respect to $\lambda$? I agree with the comment on lines 129--131 yet the bound would be sharper and easier to read. I am a bit confused about the fact that $\lambda$ is not taken care of in the theory even though its optimization seems straightforward, and automatically selected via cross-validation in the numerical experiments. Have I missed something there?         (ii) the leading constant is never 1 as soon as $L\geq 2$, thus the oracle inequality is not sharp. This clearly undermines the conclusion as the excess risk is not comparable to the risk of the oracle (the best subtree, in the sense of the smallest excess risk). I reckon that obtaining a sharp inequality might not be possible with the scheme of proofs used by the authors yet I would find interesting to see comments on this.         (iii) at first glance, the rate seemed to be 1/n but it is actually 1/\sqrt{n}, which is much more coherent given the assumptions made in the manuscript. I feel the authors should more clearly state this fact, to avoid ambiguity and possible confusion.          The manuscript then moves on to the (favorable) situation where the excess risk of the oracle is known to converge to 0. This is done at the price of a contraction bound on the regression functions $\mathbb{E \mathbf{1}_{Y=\ell}$ with an exponent parameter $\alpha$. I am a bit puzzled by this assumption as the regression functions take their values in $(1,L)$ and that the diameter of the input space is 1. So I feel that this assumption might be quite stringent and implies a rather strong constraint on the unknown underlying distribution of $(X,Y)$. An additional comment might be useful there.         In this setting, the rate of convergence attained by the oracle is the classical nonparametric rate $n^{-\alpha/(2\alpha+d)}$ where $d$ stands for the intrinsic dimension rather than the dimension $D$ of the input space. While very classical in nonparametric statistics, I fear that $d$ is, in practice, quite large for most classification trees, and the rate becomes arbitrarily slow. Nevertheless, this remark is a bit beyond the scope of the manuscript since the key message here is that whenever the oracle is consistent (its excess risk converges to 0), the Q-weighted majority vote inherits the rate of convergence of the oracle. The rate does not really matter to draw that conclusion. This is a very classical conclusion in aggregation literature yet it is the first time I see this for classification trees.          **Overall assessment**          The manuscript is well-written and carries interesting results, using ideas which are not novel but never combined so far. The scope of the work is of interest to the NIPS community. However I have some concerns about the main results.          **Specific remarks**          - In the references: improper capitalization of words such as PAC, Bayes / Bayesian. Reference 14 is incorrect (remove "et al."). Reference 9 is incorrect (it has been long published, at ALT 2017).         - LaTeX tip: in math mode, use \ell instead of l, this us much more pleasant to the eye!          [Rebuttal acknowledged. I am satisfied with the answers provided by the authors and I upgrade my score to 7.]


A Stein variational Newton method
Stein variational gradient descent (SVGD) was recently proposed as a general purpose nonparametric variational inference algorithm [Liu & Wang, NIPS 2016]: it minimizes the Kullback-Leibler divergence between the target distribution and its approximation by implementing a form of functional gradient descent on a reproducing kernel Hilbert space. In this paper, we accelerate and generalize the SVGD algorithm by including second-order information, thereby approximating a Newton-like iteration in function space. We also show how second-order information can lead to more effective choices of kernel. We observe significant computational gains over the original SVGD algorithm in multiple test cases.

The authors propose an version of Stein variational gradient descent (SVGD) that uses approximate second-order information, both to improve the optimization (by analogy with other quasi-Newton optimization methods) and to create a geometry-aware kernel (by analogy with pre-conditioning).  The use of second order information improves convergence considerably over standard SVGD, with the greatest benefit provided by both the geometry-aware kernels and second-order optimization steps.  Stein methods for approximate inference are an exciting topic these days, the improvements are both practically feasible and significant, and the paper is very well-written.  I will argue that the paper should be accepted and really only have minor comments and suggestions. - A more appropriate title is “A Stein variational Quasi-Newton method”, since the authors are not actually able to take a full Newton step, even on small problems, due to the fact that the transforms, and hence push-forward density, are not available in closed form (line 128). - In general, I might replace the phrase “inexact Newton” with “quasi-Newton” for continuity with other optimiziation literature (e.g. line 130). - “There exist infinitely many such maps” (line 62) is repeated from the introduction. - At line 80, you can define equation (6) without Frechet differentiability; it is equation (7) that requires it (specifically, that \nabla J_{p_l} does not depend on V).  So for clarity perhaps it would be better to move the assumption of Frechet differentiability to line 81. - I think it might be better to spell out the definition of N(x) at line 131.  I gathered that it is the outer product of the gradient of log p(x) at x, but it would be better to say that explicitly since this is an important part of the algorithm.  Calling it equivalent to the Fisher information matrix is confusing -- the Fisher information matrix is usually as the covariance of the scores, whereas I think the authors don’t intend any expectation to be calculated.  Furthermore, a correctly specified Fisher information matrix could be calculated either from the scores or from the Hessian, and the authors want to emphasize that it is the former, not the latter. - At line 150, the authors define a matrix, not a metric, so it would be better to call it that.  This matrix is later used to /form/ a metric. - At line 236, is it supposed to be F(x), not F(u)?  In line 238 F is applied to the Brownian motion, not the solution to the SDE. - Line 254, for comparison I’d be interested to know how long HMC took.


Fast deep reinforcement learning using online adjustments from the past
We propose Ephemeral Value Adjusments (EVA): a means of allowing deep reinforcement learning agents to rapidly adapt to experience in their replay buffer. EVA shifts the value predicted by a neural network with an estimate of the value function found by planning over experience tuples from the replay buffer near the current state. EVA combines a number of recent ideas around combining episodic memory-like structures into reinforcement learning agents: slot-based storage, content-based retrieval, and memory-based planning. We show that EVA is performant on a demonstration task and Atari games.

Summary: This paper proposes a method that can help an RL agent to rapidly adapt to experience in the replay buffer. The method is the combination of slow and general component (i.e. model-free ) and episodic memories which are fast and adaptive. An interesting part of this proposed approach is that it slightly changes the replay buffer by adding trajectory information but get a good boost in the perfoamnce.  In addition, the extensive number of experiments have been conducted in order to verify the claim of this paper.   Comments and Questions - This paper, in general, is well-written (especially the related works, they actually talk about the relation and difference with previous works) except for the followings:     --  Section 3.1 and 3.2 do not have smooth flow. The message of these parts wasn't clear to me at first. it took me multiple iterations to find out what are the main take away from these parts. - Some variable or concepts are used without prior definition, the paper should be self-contained:  'e' in figure 1. Is it the same as 'e' in Algorithm 1? if yes, why different formats are used? if not, define.  Is it a scalar? vector?  - Define the counter-factual action. Again the paper should self-contained. - Line 62: 'The adjustment to the value function is not stored after it is... 'is very confusing to me.  Can you clarify? - Line 104-108: 'Since trajectory information is also stored, the 50 subsequent steps are also retrieved for each neighbour.' I am not sure understood this one, can you explain?  - Why in algorithm 1, 'h_t' is appended to the replay buffer but it is not sth that was mentioned in the text, eqautions, or figure. Can you explain?  In general, I like the idea of this paper and I wish the text was more clear (refer to above comments) Minor: -Line 123: 'W Idealy' is a typo? - In Algorithm 1: ' ...via the TCP algorithm', referring to the TCP equation(s) would be a better idea


On Fast Leverage Score Sampling and Optimal Learning
Leverage score sampling provides an appealing way to perform approximate computations for large matrices. Indeed, it allows to derive faithful approximations with a complexity adapted to the problem at hand. Yet, performing leverage scores sampling is a challenge in its own right requiring further approximations. In this paper, we study the problem of leverage score sampling for positive definite matrices defined by a kernel. Our contribution is twofold. First we provide a novel algorithm for leverage score sampling and second, we exploit the proposed method in statistical learning by deriving a novel solver for kernel ridge regression. Our main technical contribution is showing that the proposed algorithms are currently the most efficient and accurate for these problems.

After Author Response:  Thanks to the authors for their various clarifications.  After further thought, however, I have updated my review to a 5.  While this works seems to have very high potential, the current draft has several important shortcomings: (1) It is not yet polished enough for broad consumption.  Significant revisions would be necessary to increase clarity.  (2) The empirical section could be significantly improved; in particular, the experiments should be used to more directly validate the claims from the theorems, and to disentangle the issues related to faster optimization vs. improved final model --- this discussion should also be much more prominent. ===================================== Summary: This paper presents a novel algorithm for estimating leverage scores for kernel matrices.  The running time of this algorithm improves upon state of the art.  This algorithm is based on choosing samples of ever increasing size with which to approximate the leverage scores.  These approximate leverage scores are then used to choose the Nystrom centers in the FALKON algorithm [1] (this method is named FALKON-LSG). The paper proves that given this choice of centers, FALKON-LSG achieves optimal learning rates (like FALKON does) but with improved runtime.  It validates the quality of the estimated leverage scores, the runtime of their algorithm, and the accuracy of FALKON-LSG, empirically.  The experiments show that FALKON-LSG converges faster than the original FALKON, though the two algorithms generally converge to very similar final test accuracies.  I will now summarize the quality, clarity, originality, and significance of the work: 1) Quality: The general quality of the work seems high to me (though I have not checked the proofs).  My primary concerns over the quality of the work are: -- There appear to be a fair number of careless mistakes throughout the paper, both in terms of the math and the text (these are listed at the end of the review).  This makes me concerned that perhaps that proofs also have careless mistakes scattered throughout, which would suggest that the work isn't yet polished enough for publication. -- One of the primary theoretical claims of the paper is that this new algorithm for leverage score sampling allows for achieving the same generalization performance as FALKON and exact kernel ridge regression, but with improved running time.  In particular, the claim appears to be that only O(d_eff) Nystrom centers need to be sampled (using approximate leverage scores) to achieve the same generalization performance as sqrt(n) Nystrom centers with the standard FALKON algorithm; given that d_eff is often significantly smaller than sqrt(n), this suggests that FALKON-LSG can use far fewer centers to achieve the same accuracy as FALKON.  However, this claim is not validated empirically.  In fact, the paper states that FALKON-LSG and standard FALKON converge to very similar test accuracy, using the same number of centers.  This raises questions about when/why this algorithm would be of practical importance.  It would have been informative to see a plot, with the x axis as the number of Nystrom centers, and the y axis as test performance (at convergence), for FALKON-LSG and FALKON.  One thing the experiments *do show*, however, is that FALKON-LSG converges **faster** than FALKON; but this is a bit confusing, given that the rate of convergence was not something the theory addressed in any way.  The paper does not dig into why this might be the case.  It also shows FALKON-LSG is more robust to the choice of regularizer \lambda than FALKON, but this also isn't explained.  2) Clarity: If I were to ignore the list of small errors (in math and text) summarized below, I would say the paper is in general quite clear.  But unfortunately, the presence of all these small issues made the paper somewhat laborious to get through.  On the bright side, these issues could be fixed quite quickly.  Some more minor comments: - Lines 137-138: I would have liked a more careful explanation of the time and space bounds. - In plots, legends and axis labels have tiny text.  3) Originality: The work is original, giving two new algorithms for leverage score sampling, and proving that these algorithms can achieve the same leverage score approximation error as existing algorithms, but can do so asymptotically faster.  Furthermore, it presents a way of using these approximate leverage scores in the supervised learning setting (regression with least squares loss) in order to achieve optimal rates asymptotically faster than existing work.  4) Significance: From a theoretical perspective, the work is definitely significant.  The paper presents the fastest known algorithm for achieving optimal learning rates for kernel ridge regression.  However, it remains unclear what the practical significance of the work is, given that the paper was unable to show improved test performance over FALKON for a fixed number of Nystrom centers.  [1] Alessandro Rudi, Luigi Carratino, Lorenzo Rosasco: FALKON: An Optimal Large Scale Kernel Method. NIPS 2017.  ========================================================================== Mathematical issues found in paper: - Line 53: Isn't d_eff bounded above by n, not sqrt(n)?  For example, when lambda is infinitesimal Tr(KK^(-1)) = n. - Line 61: The RLS of sample x_i seems to be an indication of how orthogonal phi(x_i) is, not x_i.  I think this was meant as an informal statement, so not a huge deal. - Line 73: It says "corresponds exactly to \tilde{l}", and should say "corresponds exactly to l". - Line 77: |D| not defined.  It seems |D| = |J|. - Algorithm 1: --- l(x_u, lambda) should be l(u,lambda) --- R_h as defined doesn't depend on h.  I believe it should be R_h = q_1 min(\kappa^2/ \lambda_h, n). - Lines 89-90: The summary of lemma 3 is incorrect.  Should be \lambda_h \leq \lambda_{h-1} \Rightarrow l(i, \lambda_h) \leq (\lambda_{h-1}/\lambda_h) l(i,\lambda_{h-1}). - Lines 101-102: Should be O(n M_{h-1}^2).  I would then specify that this is made more efficient by only computing the approximate leverage scores for R_h points, which takes O(R_h M_{h-1}^2) time. - Algorithm 2: --- Failure probability \delta is taken as input but never used. --- Line 3: \lambda_h / q.  What is q?  Only q_1 and q_2 are defined as inputs. --- Line 4: q_1 q / (\lambda_h n).  Once again, what is q? --- Line 6: uniform in what range?  I assume [0,1] --- Line 8: Shouldn't it be min(\beta_h, ...), not min(1, ...)?  This would ensure that p_{h,i}/\beta_h in [0,1]. - Lines 113-118.  h-1 and h are confused in many places.  computing l_{D_{h-1}} requires inverting (K_{J_{h-1}} + \lambda_{h-1}n I), which costs M_{h-1}^3 time, and each of the R_h evaluations takes M_{h-1}^2 time. So Algorithm 1 runs in time O( \sum_{h=1}^H M_{h-1}^3 + R_h M_{h-1}^2) time. - Lines 113-118: I found the derivation of the time and space bounds confusing.  Shouldn't time bound be something like O(\sum_h \lambda_h^{-3})  (and then this should be simplified to be in terms of \lambda). - As written, algorithm 1 returns  (M_h, J_h, A_h), and algorithm 2 returns (J_h, A_h) for h \in {1,...,H}.  This inconsistency is weird.  Also, it seems like the output of these algorithms should simply be (J_H,A_H), the LSG from the final iteration.  This seems to be the way algorithm 3 uses the LSG algorithm in line 1. Another inconsistency is that Algorithm 2 takes as input a failure probability delta (which is never used), and Algorithm 1 doesn't. - Algorithm 3: --- Line 1: "(J,A) = LSG-Alg(X,\lambda,K) with approximation factor T = 2 and step q = 2".  As mentioned above, (J,A) as output of the algorithm currently doesn't type check.  Also, algorithms 1 and 2 require as input {q_1, q_2, \lambda_0}, not {T, q}. - line 172 and Eq (2): Should 'H' be used instead of 'h'?  Issues in text: NIT - In many places, "leverage scores sampling" and "leverage scores generators" is written.  It seems grammatically it should be "leverage score sampling" and "leverage score generators". - Two broken references on page 1 (lines 17 and 31). - Line 42: establish --> establishes - Line 44: "Our second, contribution..." --- comma after "second" is unnecessary. - Algorithm 2 title: Regection --> Rejection. - Line 112: mimik --> mimic. - Algorithm 3: broken equation reference "Eq. ??" - line 122: create --> creates. - line 160: "consists in a efficient" ---> "consists of an efficient". - line 190: "standard basic" --> "standard". - line 208: Broken theorem reference "Thm. ??" - line 209: "In Thm" --  Thm number not specified. - line 256: "Nystromcentres" --> "Nystrom centers". - Figures: --- Unit of time not specified in Figure 1. --- Figure 1 is actually a table (should be Table 2) --- The RLS accuracy box plot should be a separate figure with its own caption probably. --- Instead of calling the proposed algorithms "BLESS" and "BLESS-R" as these methods are called throughout the text, it says "Opt LS" and "REJLS".  This is confusing.  Also, RRLS is called Musco in two of the plots, which is also confusing.


Backpropagation with Continuation Callbacks: Foundations for Efficient and Expressive Differentiable Programming
Training of deep learning models depends on gradient descent and end-to-end differentiation. Under the slogan of differentiable programming, there is an increasing demand for efficient automatic gradient computation for emerging network architectures that incorporate dynamic control flow, especially in NLP. In this paper we propose an implementation of backpropagation using functions with callbacks, where the forward pass is executed as a sequence of function calls, and the backward pass as a corresponding sequence of function returns. A key realization is that this technique of chaining callbacks is well known in the programming languages community as continuation-passing style (CPS). Any program can be converted to this form using standard techniques, and hence, any program can be mechanically converted to compute gradients. Our approach achieves the same flexibility as other reverse-mode automatic differentiation (AD) techniques, but it can be implemented without any auxiliary data structures besides the function call stack, and it can easily be combined with graph construction and native code generation techniques through forms of multi-stage programming, leading to a highly efficient implementation that combines the performance benefits of define-then-run software frameworks such as TensorFlow with the expressiveness of define-by-run frameworks such as PyTorch.

=== Summary  The fields of high-performance computing and systems have played a large role in the recent developments in deep learning. More recently, important research from the compilers and programming languages perspective has been contributing to deep learning, addressing the lack of usability and flexibility that many frameworks have traded for efficiency. This work falls squarely in that category.  The authors propose a new implementation approach to reverse mode automatic differentiation. The technique relies on operator overloading, but does not rely on a tape (and the accompanying interpreter that walks the tape backwards). Instead, any primal computation takes a callback to the rest of the forward pass which it calls before performing the adjoint computation, making the "tape" implicit in the function call stack.  The authors provide a basic implementation of this idea in Scala, along with a cleaner implementation which relies on delimited continuations (using  the shift/reset oeprators) in order to construct the necessary callbacks automatically.  Lastly, lightweight modular staging (LMS) is used to reduce the overhead of these abstractions (dual numbers, callbacks, etc.) and generate efficient C++ code. Experiments highlight that this approach is competitive with existing frameworks such as PyTorch and TensorFlow.  === Review  In the abstract the authors highlight that the use of continuation passing style (CPS) is a key ingredient to their approach. They claim that this can be implemented without any "auxiliary data structures". This claim is repeated in lines 36-37. However, frameworks that rely on having a single global scope (e.g. TensorFlow) or that use closures (e.g. Myia) don't need these data structures either.  The same confusion appears in section 2: The text suggests that TensorFlow uses graph as in figure 1, where nodes hold values for both the forward and backward pass. This is not the case. TensorFlow augments the forward graph with additional primitives i.e. it creates a larger graph in which there is no distinction between the forward and backward pass anymore. Hence, the graph itself is never used as an auxiliary data structure that stores intermediate variables and there is no "graph iterator which performs a forward pass first, then a backward pass" (line 75).  The remainder of the section does a good job at conveying the main ideas of the paper. In the last two paragraphs the authors compare their approach to Siskind's approach. Although it is true that that transformation is more complex, it must be noted that it operates on lambda calculus, whereas the proposed implementation here relies on (1) compiler support for control delimiters and (2) side effects (i.e. instead of requiring a backpropagator to return partial derivatives to a particular scope it can call `that.d += y.d`). The different restrictions under which the transformations apply makes it misleading to compare them without qualifying this more clearly in the text.  In section 3 lightweight modular staging (LMS) is introduced as an approach to achieving some of the performance gains that libraries such as TensorFlow can achieve by optimizing and compiling the computation graph. I think the text would possible benefit from a small code sample that involves a loop or conditional here, but overall the text is clear.  In the last two sections the authors argue that their approach is applicable to other languages. "Fundamentally" almost anything can be done in any language, but I would have appreciated more insight as to what requirements their approach puts on a language. For example, I think it is far less feasible to implement delimited continuations and LMS in Python, given its dynamically typed, interpreted nature.  The authors argue that their approach benefits more from general compiler transformations, but I believe the same is true for e.g. TensorFlow when using the XLA backend, since it lowers the entire graph to LLVM in that case.  In section 4 a few simple models are implemented using the proposed approach and benchmarked against PyTorch, NumPy and TensorFlow. I do not think it is necessary to show training curves; that they decrease actually says very little (neural networks and SGD are very robust, and the loss will decrease just as well even if there are mistakes in the gradients). The correctness of gradients can just be checked using finite differences by the authors and be taken for granted by the reader.  The runtime benchmarks look promising, but as the authors hint at in the text: The results mainly highlight that if you consider very small models, batch sizes, and datasets, your performance will no longer depend on e.g. your compute kernels, high-level linear algebra optimizations, or your ability to exploit large amounts of parallelism (the things that TensorFlow and PyTorch are optimized for). Instead, your performance will be entirely determined by the overhead of your language and AD implementation, in which case a statically typed language like Scala which will use LMS to compile custom compute kernels will always beat a dynamically typed, interpreted language like Python which will be calling e.g. MKL kernels--the authors failed to mention which BLAS library they linked to--which are optimized for larger workloads on more cores. That said, there is no perfect way to do these benchmarks and for people who care about small workloads in low-resource environments, these results are encouraging. That said, statements such as "Lantern performs comptetitively on cutting edge deep learning applications which push the boundaries of existing frameworks [in] performance" could be qualified, considering that neither the models nor hardware in this benchmark are representative of the common deep learning workloads that existing frameworks are optimized for.  In the related work section I think it would be worthwhile to discuss Siskind and Pearlmutter's more recent work on checkpointing, considering that uses both CPS and control operators in the context of reverse mode AD.  Finally, some higher-level questions that this work raised for me:  The listing in figure 4 suggests that the partial derivative is explicitly initialized to zero during the forward pass. For multidimensional arrays this can lead to the allocation of large amounts of zeros, significantly increasing the peak memory usages. Most existing libraries try to be more clever about this (e.g. Tangent uses a special zero object). Have the authors given this any thought?  Higer-order differentiation. An important goal in AD is usually that the gradient operator is closed under its own operation. Also in ML, this is the case e.g. TensorFlow, PyTorch, and more research-oriented libraries such as Tangent, Myia, and Autograd, all have support for higher-order derivatives, and often have support for mixing forward and reverse mode. It is a shame that this is not dicsused at all in this paper. Can a single shift operator be matched to multiple reset operators?   === Minor comments  223: Parentheses around citations. 267: lead -> led  === Conclusion  There are a few misunderstandings in the paper about the internals of TensorFlow that should be addressed, but they don't fundamentally change the authors claim that their approach can naturally support control flow while still supporting multi-stage programming. Other things I would like to see addressed are higher-order gradients, and the supposed applicability of this approach to e.g. Python (which I find hard to believe).  Overall I believe the paper is well-written though, the research question relevant, the approach sufficiently novel, and the experimental results, although preliminary, show promise. Hence, I would recommend this work for publication with minor edits.


Improved Network Robustness with Adversary Critic
Ideally, what confuses neural network should be confusing to humans. However, recent experiments have shown that small, imperceptible perturbations can change the network prediction. To address this gap in perception, we propose a novel approach for learning robust classifier. Our main idea is: adversarial examples for the robust classifier should be indistinguishable from the regular data of the adversarial target. We formulate a problem of learning robust classifier in the framework of Generative Adversarial Networks (GAN), where the adversarial attack on classifier acts as a generator, and the critic network learns to distinguish between regular and adversarial images. The classifier cost is augmented with the objective that its adversarial examples should confuse the adversary critic. To improve the stability of the adversarial mapping, we introduce adversarial cycleconsistency constraint which ensures that the adversarial mapping of the adversarial examples is close to the original. In the experiments, we show the effectiveness of our defense. Our method surpasses in terms of robustness networks trained with adversarial training. Additionally, we verify in the experiments with human annotators on MTurk that adversarial examples are indeed visually confusing.

Work description The paper proposes a new method for improving robustness of NNs to adversarial attacks.  The work progresses towards distinguishing two types of errors. The first type is misclassification of (adversarial) images that may be in or out of the data manifold but are unlikely to be instances of other classes. The second type is misclassification of images that are likely instances of other classes. The second type of errors comes from the inherent overlap of the data and cannot be avoided (e.g. there are digits that are “in between” the classes and are difficult to classify for humans). The first type is the harmful one, which needs to be addressed. The main idea is to achieve that the net misclassifies a perturbed image only if it looks like the target class and is robust to all other perturbations. The proposed learning method explicitly forces the distribution of misclassified images for a given adversarial class to look like the true data samples of that class. The distance between these distributions leads to the regularizer which is expressed as a GAN pair in which the generator is the algorithm that finds adversarial examples leading to the misclassification with high confidence. The experiments on MNIST compare to SOTA methods and show improved robustness results. A human study is conducted to prove that the remaining adversarial perturbations are indeed confusing to the human (and thus are of the second type).  Evaluation I thank the authors for their work. The paper is very clear. It gives an excellent overview of existing methods and how they can be viewed as (approximate) robust optimization methods. The method is still a bit “heuristic” in the sense that the regularizing objectives are appended somewhat arbitrary. However, it is much sounder than many other proposals.  The experiments are on a simple dataset but cleanly made and giving a comparison to the baseline method and 3 robust training techniques. It is not clear what is the computational complexity of the method (i.e. are second order gradients involved?). However, the positive aspect is that the test-time model is the standard NN and is thus efficient. In my opinion the paper makes a good contribution and I see no problems. Clarity (minor) L 35-36: this is confusing since after adversarial training is introduced, it suggests training with adversarial examples that should be similar to regular images. This stays confusing in Fig 1 and lines 43-44. In Fig 1. it is not explained what is what. A good formulation when I finally got the idea is in lines 133-135. The design of the high confidence attack is perhaps not well explained. I see that without the high confidence the “generator” would only produce images on the classification boundary between the classes. Then enforcing the generator distribution to be close to the data distribution would actually enforce the model to have fractured boundaries. Instead it is somehow desirable to cover the whole adversarial target class, i.e. to also generate the images in its interior. What is the purpose of cycle consistency (7) is not clear. Discussion The difference between classifier and the critique is not completely clear. Could the critique be used as a classifier? Could the critique and the classifier be the same?  It seems that works on supervised learning formulations of generative models would be highly relevant. I.e. when one learns the joint mode of data and labels p(y,x). The objective (3) looks like the pseudo-likelihood learning of such generative model: the first part is the conditional likelihood p(y|x), modeled by the classifier, and the second part is the conditional likelihood p(x|y), modeled by the generator (which is the adversarial remapping of the true distribution). In that sense the method performs generative learning – fitting the joint distribution p(y,x) to the data. Such formulations are known to improve generalization properties with scarce data but achieve lower accuracy when more data is available. Alternatively to (3), is it possible to enforce the constraint that “images that are misclassified by the network may not look like the original class”? Technical In (1) max should be under the sum Minor L78: employ utilize  Final comments I believe the rebuttal addresses well all raised questions. I think it is an excellent work both conceptually and practically.


Learning Gaussian Processes by Minimizing PAC-Bayesian Generalization Bounds
Gaussian Processes (GPs) are a generic modelling tool for supervised learning. While they have been successfully applied on large datasets, their use in safetycritical applications is hindered by the lack of good performance guarantees. To this end, we propose a method to learn GPs and their sparse approximations by directly optimizing a PAC-Bayesian bound on their generalization performance, instead of maximizing the marginal likelihood. Besides its theoretical appeal, we find in our evaluation that our learning method is robust and yields significantly better generalization guarantees than other common GP approaches on several regression benchmark datasets.

I  have read the replies from reviewers and I decided to increase the score of this paper from 6 to 7.   -----  By minimizing PAC-Bayesian Generalization Bounds” the authors propose to train Gaussian processes (GPs) using a PAC-Bayesian bound for bounded loss functions in the regression case. The paper first recalls the classic PAC-Bayesian theorem (see, e.g. McAllester (1999), Theorem 1 and Seeger (2003), Theorem 3.1) for [0, 1]-valued loss func- tions. They show, by using a union bound, that the generalization bound holds for all parameters θ in a finite parameter space Θ and they propose to mini- mize this bound to train GP hyper-parameters in case of full GPs and to select hyper-parameters and inducing points for sparse GP models. In order to use the generalization bound the authors restrict their analysis to the 1-0 loss function. Other loss functions are introduced in appendix. They consider a FITC model (Quiñonero-Candela and Rasmussen, 2005) for sparse GP and train by minimizing the generalization bound. They compare the method with standard methods for full GP and sparse GP training on classic data sets and show that their method gives the best generalization guarantees. Quality: The submission is technically sound. The derivations for the training objectives are well explained and an experimental section details the strengths and weaknesses of the method on standard data sets. Clarity: The submission is clearly written and well organized. Below are some points that might need clarification. • The sparse GP section 3.2 could be improved by clarifying the treatment of the inducing points z 1 , . . . , z M during the training. It is not completely clear in line 195, 196 how and if the discretization of Θ influences inducing points selection. • The experimental section details the methods compared, however gives no indication on the programming language and software used. Nonetheless he authors note that the software implementing the experiments will be made available after acceptance. • line 159 it is not clear to me why a coarser rounding should improve the bound or the optimization. • line 200 the FITC model trained with marginal likelihood is known to over-fit, while the kl-PAC-SGP training objective seems to alleviate this behavior it would be interesting to test this on pathological data sets where FITC is known to fail. Originality: The method proposed, to the best of my knowledge, is new for the task of GP regression. However, the use of [0, 1]-valued loss functions renders it a direct extension of GP classification training. Germain et al. (2016) develop a PAC-Bayesian bound for regression with unbounded loss functions which however requires distributional assumptions on the input examples. The authors mention this and note that in order to keep a distribution-free frame- work they use bounded loss functions. The very interesting question raised by this submission is if it is possible to generalize the training method to the unbounded loss functions commonly used when evaluating regression tasks. Significance: The submission introduces a training method for GP regres- sion which, to the best of my knowledge, is a new technique. By shifting the focus on generalization guarantees it provides an interesting point of view in the likelihood dominated landscape of GP training. However, in my opinion, there are several aspects that reduce the significance of this work. • the use of [0, 1]-valued loss functions makes this training method much less appealing for practitioners as it leads to less accurate results in MSE than classic methods. • Germain et al. (2016) show that if we consider as loss function a bounded negative log-likelihood then minimizing the PAC-Bayesian bound is equiv- alent to maximizing the marginal likelihood. I am wondering if by using an appropriately bounded negative log-likelihood we could obtain better results than [0, 1]-valued loss functions in terms of upper bound and Gibbs risk. • The authors show that the discretization of the parameter space Θ does not affect the method drastically on an example. In the same section, Ap- pendix F, they mention that “as long as a minimal discrimination ability is allowed” (line 502) the training and test risks are not affected. However I am not fully convinced that for ARD kernels with high d a minimal dis- crimination ability is within reach computationally because the number of parameter combinations increases drastically.


On Misinformation Containment in Online Social Networks
The widespread online misinformation could cause public panic and serious economic damages. The misinformation containment problem aims at limiting the spread of misinformation in online social networks by launching competing campaigns. Motivated by realistic scenarios, we present an analysis of the misinformation containment problem for the case when an arbitrary number of cascades are allowed. This paper makes four contributions. First, we provide a formal model for multi-cascade diffusion and introduce an important concept called as cascade priority. Second, we show that the misinformation containment problem cannot be approximated within a factor of Ω(2 log 1− n 4 ) in polynomial time unless N P ⊆ DT IM E(n polylog n ). Third, we introduce several types of cascade priority that are frequently seen in real social networks. Finally, we design novel algorithms for solving the misinformation containment problem. The effectiveness of the proposed algorithm is supported by encouraging experimental results.

This paper addresses the problem of minimizing the effect of negative (e.g., false, harmful) information cascades in online social networks, i.e. misinformation containment. In particular, they consider the case of an arbitrary number of cascades, whereas most prior work in NIPS, KDD, INFOCOM, etc. have focused on the cases of one or two cascades. The authors provide a formal model for multi-cascade influence diffusion and cascade priority (Section 3), design an algorithm for minimizing the number of affected nodes (Section 5), provide theoretical bounds in terms of runtime and optimality (Section 4), and demonstrate the superiority of their algorithm against baselines on real-world datasets.  Overall, I think this is a very good submission. The authors provide clear motivation for the multi-cascade misinformation problem, rigorous theory around the proposed algorithm, and a reasonably comprehensive set of experiments on three real-world datasets. The paper is both original and significant because at least to my knowledge, the multi-cascade problem in online social networks has not been mathematically modeled before. Further, the paper is well written, with all ideas and even theorems explained with clarity. For these reasons, I recommend accepting the paper.  At the same time, I also found two areas for improvement. First, I would encourage the authors to consider a more comprehensive set of propagation probabilities in the experimentation. As it stands, it seems that these are set deterministically based on the observed social network, e.g., the number of activities observed from u to v and the out degree of particular nodes. This is not likely to capture nuances in how people influence one another, though, including the susceptibility of individuals, the influential power of individuals, and the fraction of interactions that are actually attempting to spread information. There is of course no way to account for all of these factors, but one idea would be to add random Gaussian noise on each of the link weights, and average the results over multiple trials, to see how robust the algorithms are to fluctuations. Another possibility would be to weight each of the probabilities by a measure of importance of the root node, like their social centrality.  Second, I feel that the authors refer to the supplemental material too often. To rectify this, I would suggest moving Section 4 and a few of the less important theorems entirely to the supplemental material, and instead including more of the proofs from Sections 3 and 5 in the manuscript itself. This way the paper would be more self-contained, and the reader does not have to flip back and forth between the paper and the supplement as much.


Direct Estimation of Differences in Causal Graphs
We consider the problem of estimating the differences between two causal directed acyclic graph (DAG) models with a shared topological order given i.i.d. samples from each model. This is of interest for example in genomics, where changes in the structure or edge weights of the underlying causal graphs reflect alterations in the gene regulatory networks. We here provide the first provably consistent method for directly estimating the differences in a pair of causal DAGs without separately learning two possibly large and dense DAG models and computing their difference. Our two-step algorithm first uses invariance tests between regression coefficients of the two data sets to estimate the skeleton of the difference graph and then orients some of the edges using invariance tests between regression residual variances. We demonstrate the properties of our method through a simulation study and apply it to the analysis of gene expression data from ovarian cancer and during T-cell activation.

The goal of this paper is to construct a DAG that represents the differences between two different but related causal DAG models (representing different causal relations in different contexts) from i.i.d. data. The difference graph has an edge from A to B when the coefficient for the edge is different in the two causal DAGs (including the possibility that the coefficient is zero in one of the DAGs.) This problem has been considered for undirected graphs, but not for directed graphs. For the purpose of planning experiments, directed graphs are essential however. The proposed algorithm (Difference Causal Inference or DCI) can be useful in biological problems where in different contexts some edges might have been removed or had their coefficients changed.   DCI first constructs the adjacencies in the difference DAG by searching for regression coefficients that are invariant across the different contexts, and then identifies the direction of (some) of the edges by searching for regression coefficients that are invariant across the contexts. The obvious alternative to the proposed algorithm is to use standard algorithms such as PC or GES to construct both DAGs from the two different data sets, and then take their difference. However, this naive algorithm only finds edges that occur in only one DAG, rather than edges that appear in both DAGs but differ in their linear coefficients. In addition, because the difference DAG is often much sparser than the two original DAGs in different contexts, there are a number of statistical advantages to estimating the difference directly, as opposed to calculating the difference from estimates of the two original DAGs. (There has been similar work on inferring parts of two DAGs that are invariant across contexts, but it is not possible to infer the difference DAG from these invariants unless the two original DAGs are known.)   One limitation of the algorithm proposed in the paper is that it makes a large number of strong assumptions: 1. Linearity 2. Gaussianity (which can be weakened at the expense of computational complexity). 3. The two causal DAGs have a common super-DAG (i.e. there are no edge reversals). 4. No latent confounders (which can be weakened to no changes in the latent confounder connections). 5. All differences in linear coefficients between the two DAGs entail differences in a certain set of regression coefficients. 6. All differences in noise terms between the two DAGs entail differences in a certain set of conditional variances.  The last two assumptions are similar in spirit to the commonly made faithfulness assumption, in that they assume that certain polynomials are not equal to zero unless they are identically zero for all parameter values. However, they neither entail faithfulness nor are entailed by faithfulness.  In terms of comparing the naive algorithms versus DCI, the paper makes a solid case that where their set of assumptions is applicable, DCI widely outperforms the naive algorithms. However, the paper should mention that these there are conditions under which the naive algorithms based on PC and GES algorithms can be applied where DCI cannot. For example, both of PC and GES can be applied to much larger numbers of variables than DCI can, and PC can be applied to a much wider class of distributions, while modifications of PC drop the assumption of no latent variables.   The paper argues that assumptions 5 and 6 are generally weaker than the faithfulness assumption because the surfaces in the parameter space where the assumptions are violated are fewer in number for assumptions 5 and 6 than for the Faithfulness Assumption. However, the surfaces violating assumptions 5 and 6 are not a subset of the surfaces where faithfulness is violated. Both assumptions 5 and 6 on the one hand and faithfulness on the other hand have surfaces where they are violated of Lebesgue measure 0. If (as is more statistically relevant) one considers the volume of space close to a surface where the assumptions are violated, that depends on the shape of the surface as well as the number of surfaces. So while it seems like a reasonable conjecture that assumptions 5 and 6 are weaker than the faithfulness assumption, it seems like just a conjecture at this point.   The paper notes that the standard version of the DCI algorithm does not have a complete set of orientation rules, since it does not orient edges to prevent cycles to reduce the computational complexity of the algorithm. The article states this is "computationally intractable in the presence of long paths." However, it is not clear why adding this rule would be a computational bottleneck for the algorithm or why the length of paths matter. It would seem that this would only require calculating the transitive closure of the directed part of the graph, which should be no worse than n^3, and could certainly be performed for hundreds of variables, which is in the range that the article applies the algorithm to.   DCI was tested on both simulated and real data. The simulations used both 10 node DAGs and 100 node DAGs. In both cases, DCI significantly outperformed the naive algorithms. My only criticism of these tests is that they made all of the strong assumptions presupposed by DCI. This is quite commonly done, but given that DCI is applied to data where many of these assumptions are certainly false, it would be instructive to see how much violations of these assumptions affect the results, and the comparison with other algorithms. DCI was also applied to several real data sets, including one where there were two groups of ovarian cancer patients with different survival rates, and on where there were activated naive T cells. In each case the results were compared to an algorithm that computed a difference graph for undirected graphs. (It is not clear why the results were not also compared to the naive algorithms based on GES or PC). In each case they found nodes of importance (e.g. hub nodes) that were found to play an important role in the external literature. It is not clear from the exposition whether they found any part of the output surprising or likely to be wrong, or how systematically the output was compared to what was already in the literature.   Overall, the article was written quite clearly. The general approach of DCI is similar to other algorithms which construct graphs representing edges that are invariant across different contexts (rather than different across different contexts) and also uses algorithms for constructing difference graphs for undirected edges. However, the particular application is both novel and significant, and both simulation tests and applications to real data show promise.


Multivariate Convolutional Sparse Coding for Electromagnetic Brain Signals
Frequency-specific patterns of neural activity are traditionally interpreted as sustained rhythmic oscillations, and related to cognitive mechanisms such as attention, high level visual processing or motor control. While alpha waves (8-12 Hz) are known to closely resemble short sinusoids, and thus are revealed by Fourier analysis or wavelet transforms, there is an evolving debate that electromagnetic neural signals are composed of more complex waveforms that cannot be analyzed by linear filters and traditional signal representations. In this paper, we propose to learn dedicated representations of such recordings using a multivariate convolutional sparse coding (CSC) algorithm. Applied to electroencephalography (EEG) or magnetoencephalography (MEG) data, this method is able to learn not only prototypical temporal waveforms, but also associated spatial patterns so their origin can be localized in the brain. Our algorithm is based on alternated minimization and a greedy coordinate descent solver that leads to state-of-the-art running time on long time series. To demonstrate the implications of this method, we apply it to MEG data and show that it is able to recover biological artifacts. More remarkably, our approach also reveals the presence of non-sinusoidal mu-shaped patterns, along with their topographic maps related to the somatosensory cortex.

This work extends convolutional sparse coding to the multivariate case with a focus on multichannel EEG decomposition. This corresponds to a non-convex minimization problem and a local minimum is found via an alternating optimization. Reasonable efficient bookkeeping (precomputation of certain factors, and windowing for locally greedy coordinate descent) is used to improve scalability. The locally greedy coordinate descent cycles time windows, but computes a greedy coordinate descent within each window.  As spatial patterns are essential for understanding EEG, this multivariate extension is an important contribution. The work is clearly presented. The results demonstrate an efficient implementation. Some preliminary experiments show the potential for automatically learning brain waveforms.   Weaknesses:  In previous work, Jas et al. [13] consider different noise whereas this model assumes Gaussian noise versus the alpha stable noise. The authors should comment how this work could be extended to a different noise model.  Parameter/model selection ($L$ and $K$) are not discussed for the real-world signal (lines 235–253). Choosing $K$ too large can lead to cases where some atoms are never used or updated, and other cases where the same waveform appears as different atoms (perhaps at different shifts). Perspectives on these real-world considerations should be mentioned. All the learned waveforms should be reported in the supplement.   There are obvious similarities between greedy coordinate descent and the well-known matching pursuit algorithm, which is not mentioned. With the non-negative restriction, matching pursuit for time-series would have the same update in Eq. 5 without the shrinkage by lambda on the numerator. Assuming unit-norm patterns the greedy coordinate descent strategy (Eq. 7) would match the matching pursuit selection of maximal inner-product. Thus, the only change is that matching pursuit is done without shrinkage (lambda). Using matching pursuit, the remainder of the multivariate framework would not need to change. However, to be as efficient as the locally greedy version, a windowed version of matching pursuit should be done, which itself would be a novel algorithm. At least the issues with $\lambda_{max}$ could be replaced by using a fixed cardinality on the support. The authors could consider a discussion of the relation to matching pursuit as it has been used for both template matching (including the multivariate case) and learning waveforms in neural signals. Also to facilitate future comparisons the cardinality of support of the sparse code for different values of lambda could be investigated and reported.  Relevant reference: Piotr J. Durka, Artur Matysiak, Eduardo Martínez Montes, Pedro Valdés Sosa, Katarzyna J. Blinowska, Multichannel matching pursuit and EEG inverse solutions, Journal of Neuroscience Methods, Volume 148, Issue 1, 2005, Pages 49-59.  Based on my understanding of the reasoning, the authors believe the improved convergence efficiencies of a complete greedy coordinate descent would not compensate for the increased computational complexity of the complete greedy search. However, the gap between the local and globally greedy version could be narrowed: Couldn't the algorithm be improved by recomputing the differences in Eq. 7 after the update in each window, keeping them in a priority queue, and then proceeding to choose the best window? This could be implemented after the first loop through the $M$ windows in Algorithm 1.  Lines 218–227 How are the spatial maps in this example created? From the text it appears they are distinct for the two atoms. But the multivariate approach still would be better than single channel even if the spatial maps were the same due to spatial independence of noise.  This could be clarified.  Figure 4. How was this atom selected (by manual inspection)? Is it representative or simply the most mu-like waveform? How often does this atom appear? Are there similar atoms to it with different spatial patterns?    Minor issues: Line 20 "is central" is perhaps an overstatement in the context of neuroscience. "can be used to investigate" would be more reasonable. Line 22 duration -> durations Line 83 "aka" Line 98 overlapping -> overlapped Line 104 multivariate with -> multivariate CSC with Equations 2, 3 & 4 Should indicate the full sets of parameters as arguments in the minimizations. Especially to distinguish the case for a single multivariate signal in Equation 4.  Line 246 "9 Hz.," Line 252 "In notably includes" Line 178 values -> value Line 179 "allows to use" Line 335 'et al.' in author list for reference [29] without additional authors. Line 356 'et al.' in author list for reference [38] with only one more author.  ----- The authors have answered many of my questions. I look forward to a final version that will incorporate the additional discussion.