When Worlds Collide: Integrating Different Counterfactual Assumptions in Fairness
Machine learning is now being used to make crucial decisions about people's lives. For nearly all of these decisions there is a risk that individuals of a certain race, gender, sexual orientation, or any other subpopulation are unfairly discriminated against. Our recent method has demonstrated how to use techniques from counterfactual inference to make predictions fair across different subpopulations. This method requires that one provides the causal model that generated the data at hand. In general, validating all causal implications of the model is not possible without further assumptions. Hence, it is desirable to integrate competing causal models to provide counterfactually fair decisions, regardless of which causal "world" is the correct one. In this paper, we show how it is possible to make predictions that are approximately fair with respect to multiple possible causal models at once, thus mitigating the problem of exact causal specification. We frame the goal of learning a fair classifier as an optimization problem with fairness constraints entailed by competing causal explanations. We show how this optimization problem can be efficiently solved using gradient-based methods. We demonstrate the flexibility of our model on two real-world fair classification problems. We show that our model can seamlessly balance fairness in multiple worlds with prediction accuracy.

The authors consider a novel supervised learning problem with fairness constraints, where the goal is to find an optimal predictor that is counterfactually fair from a list of candidate causal models. The parameterizations of each candidate causal model are known. The authors incorporate the fairness constraint as a regularization term in the loss function. Evaluations are performed on two real-world datasets, and results show that the proposed method balance fairness in multiple worlds with prediction accuracy.
While the idea of exploring a novel fairness measure in counterfactual semantics and enforcing it over multiple candidate models is interesting, there are a few issues I find confusing, which is listed next:
1. The Counterfactual Fairness definition (Def 1) is not clear. It is not immediate to see which counterfactual quantity the authors are trying to measure. Eq2, the probabilistic counterfactual fairness definition, measures the total causal effect (P(Y_x)) of the sensitive feature X on the predicted outcome Y. It is a relatively simple counterfactual quantity, which can be directly computed by physically setting X to a fixed value, without using Pearl’s algorithm of three steps.
2. If the authors are referring to the total causal effect, it is thus unnecessary to use a rather complicated algorithm to compute counterfactuals (line87-94).
If the authors are indeed referring to the counterfactual fairness defined in [1], the motivation of using this novel counterfactual fairness has not been properly justified. A newly proposed fairness definition should often fall into one of following categories: i. It provides a stronger condition for existing fairness definitions; ii. It captures discriminations which are not covered by existing definitions; or iii. It provides a reasonable relaxation to improve prediction accuracy. I went back to the original counterfactual fairness paper [1] and found discussions regarding this problem. Since this a rather recent result, it would be better if the authors could explain it a bit further in the background section.
3. The contribution of this paper seems to be incremental unless I am missing something. The authors claim that the proposed technique “learns a fair predictor without knowing the true causal model”, but it still requires a finite list of known candidate causal models and then ranges over them. The natural question at this point is how to obtain a list of candidate models? In causal literature, “not knowing the true causal model” often means that only observational data is available, let alone a list of fully-parametrized possible models exists. The relaxation considered in this paper may be a good starting point, but it does not address the fundamental challenge of the unavailability of the underlying model.
Minor comments:
- All references to Figure2 in Sec2.2 should be Figure1.
[1] Matt J Kusner, Joshua R Loftus, Chris Russell, and Ricardo Silva. Counterfactual fairness. arXiv preprint arXiv:1703.06856, 2017.
Post-rebuttal:
 Some of the main issues were addressed by the authors. One of the issues was around Definition 1, and I believe the authors can fix that in the camera-ready version. Connections with existing fairness measures can be added in the Supplement.
Still, unless I am missing something, it seems the paper requires that each of the causal models is *fully* specified, which means that they know precisely the underlying structural functions and distributions over the exogenous variables. This is, generally, an overly strong requirement.
I felt that the argument in the rebuttal saying that “These may come from expert knowledge or causal discovery algorithms like the popular PC or FCI algorithms [P. Spirtes, C. Glymour, and R. Scheines. “Causation, Prediction, and Search”, 2000]”  is somewhat misleading.  Even when the learning algorithms like FCI can pin down a unique causal structure (almost never the case), it’s still not the accurate to say that they provide the fully specified model with the structural functions and distributions over the exogenous. If one doesn’t have this type of knowledge and setting, one cannot run Pearl’s 3-step algorithm. I am, therefore, unable to find any reasonable justification or setting that would support the feasibility of the proposed approach.


Real-Time Bidding with Side Information
We consider the problem of repeated bidding in online advertising auctions when some side information (e.g. browser cookies) is available ahead of submitting a bid in the form of a d-dimensional vector. The goal for the advertiser is to maximize the total utility (e.g. the total number of clicks) derived from displaying ads given that a limited budget B is allocated for a given time horizon T . Optimizing the bids is modeled as a contextual Multi-Armed Bandit (MAB) problem with a knapsack constraint and a continuum of arms. We develop UCB-type algorithms that combine two streams of literature: the confidence-set approach to linear contextual MABs and the probabilistic bisection search method for stochastic root-finding. Under mild assumptions on the underlying unknown distribution, we establish distributionindependent regret bounds of orderÕ(d · √ T ) when either B = ∞ or when B scales linearly with T .

The paper studies the problem of repeated bidding in online ad auctions where the goal is to maximize the accumulated reward over a fixed horizon and in the presence of budget constraint. It is also assumed that a context vector is available at each auctions which summarizes the specific information about the targeted person and the website. The expected value of an ad is considered to be a linear function of this context vector. This is a generalization of the setting considered in Weed et al (2016) in two directions: first having a budget constraint and second the availability of the side information. The paper proposes a bidding strategy in such a setting and provide a regret bound for it.
While the paper technically sounds and is well-written, here are a few comments on the formulated problem.
1. It has been assumed in lines 51-53 that p_t is an iid process. This seems to be far from reality as p_t is the maximum bid of the competitors each of which might be a learning agent (just like us). In this case, competitors select their bids based on his observed history. This suggests that competitors bid (and hence p_t) is not a stationary process and more importantly is correlated with our bids. This is true even if each competitors values the ad differently and in a way specific to him. This adversarial correlation is actually one of the main differences between auctions and conventional bandit problems which has been ruled out in the paper by assuming that p_t is generated from a stationary distribution in an iid manner.
2. Assumption 1 which plays an important role in the results of the paper mentions that p_t and v_t are conditionally independent given the side information x_t. Aside from the above reason (comment 1) and even if we assume that the competitors are not learning agents, this seems a very restrictive assumption. x_t is commonly shared between all the bidders and all of them select their bids based on it. Consider for example a scenario that a group of the bidders (including us) are interested in a specific type of (target person, website) pair and they will bid higher when  the associated x_t is reported. Thus, there is a correlation between those competitors bids (and hence p_t) and our bids given x_t. There is clearly a correlation between our bid and our value v_t (because we are a smart bidder) and hence, v_t and p_t cannot be independent given x_t.
A few typos:
1. Assumption 1 in the Supplementary Materials is i fact Assumption 2 of the main paper.
2. There is a . missing at the end of line 313 and one at the end of Lemma 8.
3. At the end of Section C in the Supplementary Materials it should be Lemma 3 of [2] not Lemma 11 of [2].


A Universal Analysis of Large-Scale Regularized Least Squares Solutions
A problem that has been of recent interest in statistical inference, machine learning and signal processing is that of understanding the asymptotic behavior of regularized least squares solutions under random measurement matrices (or dictionaries). The Least Absolute Shrinkage and Selection Operator (LASSO or least-squares with 1 regularization) is perhaps one of the most interesting examples. Precise expressions for the asymptotic performance of LASSO have been obtained for a number of different cases, in particular when the elements of the dictionary matrix are sampled independently from a Gaussian distribution. It has also been empirically observed that the resulting expressions remain valid when the entries of the dictionary matrix are independently sampled from certain non-Gaussian distributions. In this paper, we confirm these observations theoretically when the distribution is sub-Gaussian. We further generalize the previous expressions for a broader family of regularization functions and under milder conditions on the underlying random, possibly non-Gaussian, dictionary matrix. In particular, we establish the universality of the asymptotic statistics (e.g., the average quadratic risk) of LASSO with non-Gaussian dictionaries.

The paper studies the universality of high-dimensional regularized least squares problems with respect to the distribution of the design matrix. The main result of the paper is that, under moment conditions on the entries of the design matrix A, the optimal value of the least squares problem, and characteristics of the optimal solution are independent of the details of the distribution of the A_ij’s.
The proof uses the Lindeberg interpolation method, in a clever incremental way, to transform the problem to its Gaussian counterpart while preserving the characteristics of the optimization, then uses Gaussian comparison theorems (variants of Gordon’s escape through a mesh result) to analyze the Gaussian setting.
The consequences are nice formulas predicting the exact values of the cost, the limiting empirical distribution of the entries of the solution and of the error vector.
These formulas were previously known to be correct only in restricted settings.
These exact asymptotic characterizations of random estimation problems have been studied by several authors and in different settings. There are similarities, at least in spirit, to the work of El Karoui [1], and Donoho-Montanari [2] on robust high-dimensional regression (a problem that could be seen as the dual of the one considered here), and to many papers that have appeared in the information-theory community on “single letter formulas” characterizing the MMSE of various estimation channels (linear regression, noisy rank-one matrix estimation, CDMA,…) See e.g., [3,4,5] and references therein.
Although most of these efforts have mainly dealt with the Gaussian design setting, some have proved “channel universality”, which is essentially the focus of this paper. The authors should probably fill this bibliographic gap.
Second, I would like to point out a different approach to the problem, inspired by rigorous work in statistical physics, and that will make the connection to the above mentioned work clearer: one could consider the log-sum-exp approximation to the optimization problem, with a “inverse temperature” parameter \beta. And one recovers the optimization in the limit \beta -> \infty. One could study the convergence of log-partition function obtained in this way, and obtain a formula for it, which in this setting would bare the name of “the replica-symmetric (RS) formula”.
The “essential optimization” problem found by the authors is nothing else but the zero-temperature limit of this RS formula. (One would of course have to justify swapping the limits n \to \infty and \beta \to \infty.)
This being said, the techniques used in this work seem to be of a different nature, and add to the repertoire of approaches to compute asymptotic values of these challenging problems.
[1] Asymptotic behavior of unregularized and ridge-regularized high-dimensional robust regression estimators : rigorous results https://arxiv.org/abs/1311.2445
[2] High Dimensional Robust M-Estimation: Asymptotic Variance via Approximate Message Passing https://arxiv.org/abs/1310.7320
[3] The Mutual Information in Random Linear Estimation : https://arxiv.org/abs/1607.02335
[4] Mutual information for symmetric rank-one matrix estimation: A proof of the replica formula https://arxiv.org/abs/1606.04142
[5] Fundamental limits of symmetric low-rank matrix estimation : https://arxiv.org/abs/1611.03888


Pixels to Graphs by Associative Embedding
Graphs are a useful abstraction of image content. Not only can graphs represent details about individual objects in a scene but they can capture the interactions between pairs of objects. We present a method for training a convolutional neural network such that it takes in an input image and produces a full graph definition. This is done end-to-end in a single stage with the use of associative embeddings. The network learns to simultaneously identify all of the elements that make up a graph and piece them together. We benchmark on the Visual Genome dataset, and demonstrate state-of-the-art performance on the challenging task of scene graph generation.

This paper proposes the use of a Hourglass net with associative embeddings to generate a graph (relating objects and their relationships) from an image. The model is presented as one of  end-to-end learning. The Hourglass net provides heatmaps for objects and relationships, feature vectors are extracted from the top locations in the heatmaps and used with FC layers for predicting object-classes, bounding boxes and relationships among objects. Associate embeddings are used to link vertexes and edges, each vertex having an unique embedding.  Experimental results show the high performance of the proposed methodology.
I think this is a valuable work, yet I have doubts on whether it meets the NIPS standards. On the one hand,
* the performance of this methodology is quite good in the approached  subtasks. On the other hand, the novelty/originality of this work is somewhat limited:
* it is based in a standard Hourglass network, standard in the sense that nothing novel seems to be added to the network, it simply estimates heatmaps and the FC layers provide estimates
* it incorporates associative embeddings to generate the graph. Whereas this is an interesting approach, it is quite similar as the way it is used in [20] https://arxiv.org/pdf/1611.05424.pdf
Hence, this work is another use for Hourglass-nets + associative embeddings as proposed in [20] obtaining good performance. I do not think this is enough contribution to be presented at NIPS.
Further detailed comments:
* The authors themselves acknowledge one of their contribution is a "November el use" of associative embeddings. Whereas I agree there is novelty on the usage of this methodology, the contribution in this side is limited (in the nips sense)
* The authors fail at explaining both, the details and intuition of the proposed methodology (in all parts of the paper, but mostly in the intro, where this info is mostly important). The authors, assume the reader has the same degree of understanding of their proposed methodology ( not only of general technical stuff related to the approach, but also of methodology  itself), making it kind of tedious to read the paper, and a bit complicated to follow it
* Authors should make more explicit the differences and emphasize the importance of the contribution of this paper, certainly it is a different problem than that approached in [20], but is that difference enough contribution as to publish a paper in NIPS?, please argue on the differences and provide arguments supporting the strength of your contribution
I removed a comment on overlap, my recommendation is not based on such assumption
Typo
* "them. [18]"


The Neural Hawkes Process: A Neurally Self-Modulating Multivariate Point Process
Many events occur in the world. Some event types are stochastically excited or inhibited-in the sense of having their probabilities elevated or decreased-by patterns in the sequence of previous events. Discovering such patterns can help us predict which type of event will happen next and when. We model streams of discrete events in continuous time, by constructing a neurally self-modulating multivariate point process in which the intensities of multiple event types evolve according to a novel continuous-time LSTM. This generative model allows past events to influence the future in complex and realistic ways, by conditioning future event intensities on the hidden state of a recurrent neural network that has consumed the stream of past events. Our model has desirable qualitative properties. It achieves competitive likelihood and predictive accuracy on real and synthetic datasets, including under missing-data conditions.

The proposed submission deals with an interesting and important problem: how to automatically learn the potentially complex temporal influence structures for the multivariate Hawkes process. The proposed neutrally self-modulating multivariate point process model can capture a range of superadditive, subadditive, or even subtractive influence structures from the historical events on the future event, and the model is quite flexible. Also, the model in evaluated on both the synthetic and the real data, and yields a competitive likelihood and prediction accuracy under missing data.
Comments:
1.	Compared with existing work, one potential contribution of this submission is in the increased flexibility of the proposed model. First, in modeling the intensity function, a non-linear transfer function is introduced and is applied to the original defined intensity for multivariate Hawkes processes.  Second, it models the hidden state h(t) as a continuous-time LSTM.
2.	Since the model is more flexible and has more parameters to estimate, it will be harder to accurately learn the parameters. The algorithm basically follows the previous Back Propagation Through Time (Du et al, 2016) strategy and proposes using the Monte Carlo trick in evaluating the integral quantify in the likelihood. I think the convergence of the current algorithm might be very slow since they have too many parameters. Since the learning issue in this model is challenging, it is worthwhile discussing more on the computational efficiency. A more tractable and efficient learning algorithm might be needed in dealing with large-scale problems.
3.	For the numerical section, only the likelihood under missing data is evaluated. It will be more convincing to add the results for the intensity accuracy evaluation.


Hindsight Experience Replay
Dealing with sparse rewards is one of the biggest challenges in Reinforcement Learning (RL). We present a novel technique called Hindsight Experience Replay which allows sample-efficient learning from rewards which are sparse and binary and therefore avoid the need for complicated reward engineering. It can be combined with an arbitrary off-policy RL algorithm and may be seen as a form of implicit curriculum. We demonstrate our approach on the task of manipulating objects with a robotic arm. In particular, we run experiments on three different tasks: pushing, sliding, and pick-and-place, in each case using only binary rewards indicating whether or not the task is completed. Our ablation studies show that Hindsight Experience Replay is a crucial ingredient which makes training possible in these challenging environments. We show that our policies trained on a physics simulation can be deployed on a physical robot and successfully complete the task. The video presenting our experiments is available at https://goo.gl/SMrQnI.

Summary:
This paper introduces a method called hindsight experience replay (HER), which is designed to improve performance in sparse reward, RL tasks. The basic idea is to recognize that although a trajectory through the state-space might fail to find a particular goal, we can imagine that the trajectory ended at some other goal state-state. This simple idea when realized via the experience replay buffer of a deep q learning agent improves performance in several interesting simulated arm manipulation domains.
Decision:
I think this paper is an accept. The main idea is conceptually simple (in a good way). The paper is clear, and includes results on a toy domain to explain the idea, and then progresses to show good performance on simulated arm tasks. I have several questions that I invite the authors to respond to, in order to refine my final judgement of the paper.
I will start with high-level questions. The auxiliary task work [1] (and hinted at with the Horde paper), shows that learning to predict and control many reward functions in parallel, results in improvement performance on the main task (for example the score in an atari game). One can view the auxiliary tasks as providing dense reward signals for training a deep network and thus regularizing the agent’s representation of the world to be useful for many task. In the auxiliary task work, pixel control tasks seems to improve performance the most, but the more general idea of feature control was also presented. I think the paper under review should discuss the connections with this work, and investigate what it might mean to use the auxiliary task approach in the simulated arm domain. Are both methods fundamentally getting at the same idea? What are the key differences?
The work under-review is based on episodic tasks with goal states. How should we think about continuing tasks common in continuous control domains? Is the generalization straightforward under the average reward setting?
The related work section mentions prioritized DQN is orthogonal. I don’t agree. Consider the possibility that prioritized DQN performs so well in all the domains you tested, that there remains no room for improvement when adding HER. I think this should be validated empirically.
One could consider options as another approach to dealing with sparse reward domains: options allow the agent to jump through the state space, perhaps making it easier to find the goal state. Clearly there are challenges with discovering options, but the connection is worth mention.
The result on the real robot, is too quickly discussed and doesn’t seem to well support the main ideas of the paper. It feels like: “lets add a robot result to make the paper appear stronger”. I suggest presenting a clearer result or dropping it and pulling in something else from the appendix.
Footnote 3 says count-based exploration method was implemented by first discretizing the state. My understanding of Bellemare’s work was that it was introduced as a generalization of tabular exploration bonuses to the case of general function approximation. Was discretization required here, and if not what effect does it have on performance?
The paper investigates the combination of HER with shaped rewards, showing the shaping actually hurts performance. This is indeed interesting, but a nice companion result would be investigating the combination of HER and shaping in a domain/task where we know shaping alone works well.
[1] Jaderberg, M., Mnih, V., Czarnecki, W. M., Schaul, T., Leibo, J. Z., Silver, D., & Kavukcuoglu, K. (2016). Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint arXiv:1611.05397.


On Fairness and Calibration
The machine learning community has become increasingly concerned with the potential for bias and discrimination in predictive models. This has motivated a growing line of work on what it means for a classification procedure to be "fair." In this paper, we investigate the tension between minimizing error disparity across different population groups while maintaining calibrated probability estimates. We show that calibration is compatible only with a single error constraint (i.e. equal false-negatives rates across groups), and show that any algorithm that satisfies this relaxation is no better than randomizing a percentage of predictions for an existing classifier. These unsettling findings, which extend and generalize existing results, are empirically confirmed on several datasets.

Note: Equal opportunity as used in this paper was originally defined as “Equalized odds” in Hardt et al. 2017. This needs to be fixed in multiple places.
The paper analyzes the interaction between relaxations of equalized odds criteria and classifier calibration. The work extends and builds on the results of a previous work [23].
1. Incompatibility of calibration and non-discrimination: The previous work [23] already establish incompatibility of matching both FPR and FNR (equalized odds) and calibration in exact as well as approximate sense (except in trivial classifiers). This result has been extended to satisfying non-discrimination with respect to two generalized cost functions expressed as a linear combinations of FPR and TPR of respective groups. This generalization while new is not entirely surprising .
2. The authors show that if the equalized odds is relaxed to non-discrimination with respect to a single cost function expressed as a linear combination of FPR and FNR, then the relaxation and calibration can be jointly achieved under certain feasibility conditions.
3. Additional experiments are provided to compare the effects of enforcing non-discrimination and/or calibration.
While there are some new results and additional experiments, the conclusions are fairly similar to [23], essentially establishing incompatibility of satisfying equalized-odds style non-discrimination and classification calibration.
Additional comments:
1. Why was EO trained not implemented and compared for the heart disease and income datasets?
2. Line 165-166: there seems to be a missing explanation or incorrect assumption here as if only one of a_t or b_t is required to be nonzero then g_t can be zero with just FPR=0 OR FNR=0, respectively.
[Post rebuttal] Thanks for the clarifications on the contributions and the condition on a_t,b_t.
I see the significance of the quantification of degradation performance of classifiers even when training is mindful of fairness and calibration conditions -- a similar quantification of accuracy of "best" fair predictor for exclusively EO condition (without calibration) would form a good complementary result (I dont expect the authors to attempt it for this paper, but may be as future work). That said, given the existing literature, I am still not very surprised by the other conclusions on incompatibility of two fairness conditions with calibration.


Reliable Decision Support using Counterfactual Models
Decision-makers are faced with the challenge of estimating what is likely to happen when they take an action. For instance, if I choose not to treat this patient, are they likely to die? Practitioners commonly use supervised learning algorithms to fit predictive models that help decision-makers reason about likely future outcomes, but we show that this approach is unreliable, and sometimes even dangerous. The key issue is that supervised learning algorithms are highly sensitive to the policy used to choose actions in the training data, which causes the model to capture relationships that do not generalize. We propose using a different learning objective that predicts counterfactuals instead of predicting outcomes under an existing action policy as in supervised learning. To support decision-making in temporal settings, we introduce the Counterfactual Gaussian Process (CGP) to predict the counterfactual future progression of continuous-time trajectories under sequences of future actions. We demonstrate the benefits of the CGP on two important decision-support tasks: risk prediction and "what if?" reasoning for individualized treatment planning.

Thanks to the authors for a very interesting paper. The main contribution that I see is the proposal of a counterfactual GP model, which can take into account discrete actions, events, and outcomes, and as the authors argue allows for a much better model of discrete event data, e.g. electronic medical records, by explicitly accounting for future actions.
A minor point: the citation for Neyman shouldn't be  to 1990, as the paper is from 1923. Yes, the accessible translated version is from 1990, but the citation should include the fact that it's of an article from 1923, seeing as Neyman died in 1981. ;)
I have some suggestions for strengthening the presentation and outlined some of what confused me. Hopefully addressing this will improve the paper.
The framework adopted, of counterfactual predictions, is a very reasonable way to think about causal inference. I believe that the two main assumptions are standard in some of the literature, but, following Pearl, I do not think that they are the best way to think about what is going on. Overall, I would very much like to see a graphical model in Section 2.
Regarding Assumption 1, I would like an account following or at least engaging with Pearl's critique of VanderWeele's claim about the consistency rule as an assumption rather than a theorem.
Regarding Assumption 2, you cite Pearl so I am sure you are aware of his viewpoint, but I think it is very important to make clear what the no unmeasured confounders assumption is, and I believe that a graphical model is the best way to do this. It is worth pointing out as well that, following Pearl, we can go beyond the (often unrealistic) assumption of no unmeasured confounders (see e.g. the back-door and front-door criteria) as long as we're willing to state clearly what our structural assumptions are. Far from a criticism of how you've set things up, I believe that this viewpoint will mean you can cover more cases, as long as there is a way to setup the appropriate regression using a GP (which there should be!)
Having I think understood Assumptions 1 and 2, it now seems like the claim is that we can estimate P(Y[a]) by marginalizing P(Y | A = a, X = x) with respect to X. But I'm having trouble connecting this with the claim that the CGP will do better than the RGP. The end of the "Model." section has the key insight, which I think you ought to expand a bit on and move earlier. It'd be helpful, I think, to write down the likelihood of the RGP so that it can be compared to Eq (3). It would also help to refer to Figure 1 (which is indeed best viewed in color, but could be made quite a bit more legible in grayscale!)---what would RGP do for Figure 1, for example?
Now having read the experiments I'm confused again. In the "Experimental" scenario it is the case that after 12 hours there is a structural break, and all patients are assigned to a control condition, since no actions (treatments) are taken. But why are the RGP and CGP equivalent here?
The Observational scenario makes sense, and the fact that the CGP performs well is reassuring, but why does this support your claim that the CGP is able to "learn a counterfactual predictive model"? Shouldn't you ask the CGP to make predictions under different treatment regimens---to take the example from Figure 1, shouldn't you ask the CGP to predict what will happen if Drug A is administered at hour 13, if Drug B is administered at hour 13, or if no drug is administered? It seems to me that the CGP is a better predictive model than the RGP, and thus does a better job predicting. I think it's true that this means that under your assumptions it will also do a better job on counterfactuals, but you should evaluate this.
A question: how did you fit your models? I guess you maximized Equation 3, but there's some mixture modeling stuff going on here, and the model is highly non-convex. Were there issues with optimization? Did you need to do random restarts? How did you pick initial values? Where did the confidence intervals on parameters come from?


Learned in Translation: Contextualized Word Vectors
Computer vision has benefited from initializing multiple deep layers with weights pretrained on large supervised training sets like ImageNet. Natural language processing (NLP) typically sees initialization of only the lowest layer of deep models with pretrained word vectors. In this paper, we use a deep LSTM encoder from an attentional sequence-to-sequence model trained for machine translation (MT) to contextualize word vectors. We show that adding these context vectors (CoVe) improves performance over using only unsupervised word and character vectors on a wide variety of common NLP tasks: sentiment analysis (SST, IMDb), question classification (TREC), entailment (SNLI), and question answering (SQuAD). For fine-grained sentiment analysis and entailment, CoVe improves performance of our baseline models to the state of the art.

This paper proposes to pretrain sentence encoders for various NLP tasks using machine translation data. In particular, the authors propose to share the whole pretrained sentence encoding model (an LSTM with attention), not just the word embeddings as have been done to great success over the last few years. Evaluations are carried out on sentence classification tasks (sentiment, entailment & question classification) as well as question answering. In all evaluations, the pretrained model outperforms a randomly initialized model with pretrained GloVe embeddings only.
This is a good paper that presents a simple idea in a clear, easy to understand manner. The fact that the pretraining helps all tasks across the board is an important finding, and therefore I recommend this paper for publication.
Are the authors going to release their pretrained models?
The authors should cite https://arxiv.org/pdf/1611.02683.pdf, where the encoders are pretrained as language models.


VAIN: Attentional Multi-agent Predictive Modeling
Multi-agent predictive modeling is an essential step for understanding physical, social and team-play systems. Recently, Interaction Networks (INs) were proposed for the task of modeling multi-agent physical systems. One of the drawbacks of INs is scaling with the number of interactions in the system (typically quadratic or higher order in the number of agents). In this paper we introduce VAIN, a novel attentional architecture for multi-agent predictive modeling that scales linearly with the number of agents. We show that VAIN is effective for multiagent predictive modeling. Our method is evaluated on tasks from challenging multi-agent prediction domains: chess and soccer, and outperforms competing multi-agent approaches.

This paper extends interaction networks (INs) with an attentional mechanism so that it scales linearly (as opposed to quadratically in vanilla INs) with the number of agents in a multi-agent predictive modeling setting: the embedding network is evaluated once per agent rather than once for every interaction. This allows to model higher-order interactions between agents in a computationally efficient way. The method is evaluated on two new non-physical tasks of predicting chess piece selection and soccer player movements.
The paper proposes a simple and elegant attentional extension of Interaction Networks, and convincingly shows the benefit of the approach with two interesting experiments. The idea is not groundbreaking but seems sufficiently novel, especially in light of its effectiveness. The paper is clear and well written. I appreciate that the authors provide an intuitive yet precise explanation (in words) of their mechanism instead of resorting to unnecessarily complicated notation and formulas.
Some confusion comes from the discussions about single-hop and multi-hop, terms that should probably be defined somewhere. On line 105, the authors write: "The architecture can be described as a single-hop CommNet with attention. Although CommNets typically use more hops for learning communications, more than one hop did not help on our predictive modeling tasks", but on lines 136 and 262 it says that VAIN uses multiple hops.
The experiments are well thought-out and interesting. I'm only missing some more insights or visualizations on what kind of interactions the attention mechanism learns to focus on or ignore, as the attention mechanism is the main contribution of the paper.
Further, I wonder if a disadvantage of the proposed attention mechanism (with the fixed-length softmax layer depending on the number of agents) does not take away one of the benefits of vanilla INs, namely that it is straightforward to dynamically add or remove agents during inference. If so, perhaps this can be mentioned.
Some minor remarks and typo's:
- typo in sentence on line 103-104: two times "be"
- equation (3) should be C = (P_i, F_i), or the preceding sentence should be changed
- line 138: sentence should start with "*While* the original formulation ..."
- line 284: generally *fares* poorly.
Overall, I believe this work would be a valuable contribution to NIPS.


Collaborative Deep Learning in Fixed Topology Networks
There is significant recent interest to parallelize deep learning algorithms in order to handle the enormous growth in data and model sizes. While most advances focus on model parallelization and engaging multiple computing agents via using a central parameter server, aspect of data parallelization along with decentralized computation has not been explored sufficiently. In this context, this paper presents a new consensus-based distributed SGD (CDSGD) (and its momentum variant, CDMSGD) algorithm for collaborative deep learning over fixed topology networks that enables data parallelization as well as decentralized computation. Such a framework can be extremely useful for learning agents with access to only local/private data in a communication constrained environment. We analyze the convergence properties of the proposed algorithm with strongly convex and nonconvex objective functions with fixed and diminishing step sizes using concepts of Lyapunov function construction. We demonstrate the efficacy of our algorithms in comparison with the baseline centralized SGD and the recently proposed federated averaging algorithm (that also enables data parallelism) based on benchmark datasets such as MNIST, CIFAR-10 and CIFAR-100.

This paper explores a fixed peer-to-peer communication topology without parameter server. To demonstrate convergence, it shows that the Lyaounov functions that is minimized includes a regularizer term that incorporates the topology of the network. This leads to convergence rate bounds in the convex setting and convergence guarantees in the non-convex setting.
This is original work of high technical quality, well positioned with a clear introduction. It is very rare to see proper convergence bounds in such a complex parallelization setting, the key to the proof is really neat (I did not check all the details). As we know, insight from convex-case proofs usually apply to the non-convex case, especially as the end convergence should be approximately convex. Thanks to the proof, one can predict the impact of the topology over convergence (through the eigenvalues). This proof should also open to further work. Experiments are also well designed, thought they do not cover the hypothetical situation where a parameter server could fail.
However, several critical limitations will impact the short term significance. Some of them could probably be addressed by the time of publication.
First, the loss in both training speed and convergence accuracy of CDSGD compared to plain SGD is very large, and is only mitigated with the use of a momentum. While the authors show how the algorithm can be modified to support both Polyak and Nesterov momentums, I could not find anywhere which momentum CDMSGD actually use! There is no explanation of why this momentum helps performance so much.
Second, compared to other data-distributed SGD approaches such as Federated Averaging, the main practical contribution of the paper is to get rid of the parameter server, at some efficiency cost on controlled experiments (figure 2.b). So this work significance would greatly improve if the authors clearly identified the reasons for not using a parameter server. In the introduction, they hint at better privacy preserving (if I understood right), but Federated Averaging also claims privacy preserving WITH a parameter server. In the experiments, final accuracy with momentum is better than Federated averaging: is this because of the decentralized approach or the use of a momentum? Note also that this paper does not seem to mention the main reason I have seen for using decentralized approach, which is robustness in case of failure of the central server.
Last, the stochastic Lyapunov gradient (eq 7) shows that the regularization can be very strong especially with small learning rates, and the effects are visible in the experiments. What this regularization amounts to would be worth studying, as this is a striking side-effect of the algorithm.
Potential issues:
Line 92: f_j(x)=1/N f(x) does not make sense to me.
It should be  f_j(x)=1/N sum_i f^i_j(x) ??
Line 119: it implies that all eigenvalues are non-zero, which contradicts the fully connected topology where they are all zero but one.
Line 143: "Gradient of V(x) also has a Lipshitz": you either remove gradient or replace 'has' with 'is'.


Learning Combinatorial Optimization Algorithms over Graphs
The design of good heuristics or approximation algorithms for NP-hard combinatorial optimization problems often requires significant specialized knowledge and trial-and-error. Can we automate this challenging, tedious process, and learn the algorithms instead? In many real-world applications, it is typically the case that the same optimization problem is solved again and again on a regular basis, maintaining the same problem structure but differing in the data. This provides an opportunity for learning heuristic algorithms that exploit the structure of such recurring problems. In this paper, we propose a unique combination of reinforcement learning and graph embedding to address this challenge. The learned greedy policy behaves like a meta-algorithm that incrementally constructs a solution, and the action is determined by the output of a graph embedding network capturing the current state of the solution. We show that our framework can be applied to a diverse range of optimization problems over graphs, and learns effective algorithms for the Minimum Vertex Cover, Maximum Cut and Traveling Salesman problems.

The authors propose a reinforcement learning strategy to learn new heuristic (specifically, greedy) strategies for solving graph-based combinatorial problems. An RL framework is combined with a graph embedding approach. The RL approach effectively learns a greedy policy for selecting constructing an approximate solution. The approach is innovative and the empirical results appear promising. An important advantage of the work is that the learned policy is not restricted to a fixed problem size, in contrast to earlier work.
One drawback of the paper is that I found the writing unnecessarily dense and unclear at various places. In particular, it would be good to include more details on the intuitive description of the approach. I believe the ideas can be stated clearly in words because the concept of learning a greedy policy is not that different from learning any policy as done in RL (Learning the “next move to make” in a game is quite analogous to learning what is the next node in the graph to select. So, the authors can focus more on what makes this work different from learning a game strategy.) The over reliance on formal notation does not help.
I did not fully check the formal details but some points were unclear. For example, a state is defined as a sequence of action nodes on the graph. Each node in the tagged graph is represented by its p-dimensional embedding (a p-dimensional vector). A state is then defined as the sum of the vectors corresponding to the set of action nodes so far. Why is this choice of state definition the “right one”?
The empirical results are promising but some further insights would be helpful. For most combinatorial problems there is a known basic greedy strategy that already performs quite well. Does RL rediscover those strategies for the domains under consideration? (If the framework has the potential to discover new algorithmic strategy, it would be good to know that the approach can also re-discover known strategies.)


A Learning Error Analysis for Structured Prediction with Approximate Inference
In this work, we try to understand the differences between exact and approximate inference algorithms in structured prediction. We compare the estimation and approximation error of both underestimate (e.g., greedy search) and overestimate (e.g., linear relaxation of integer programming) models. The result shows that, from the perspective of learning errors, performances of approximate inference could be as good as exact inference. The error analyses also suggest a new margin for existing learning algorithms. Empirical evaluations on text classification, sequential labelling and dependency parsing witness the success of approximate inference and the benefit of the proposed margin.

This paper is on the important topic of learning with approximate
inference. Previous work, e.g., Kulesza and Pereira (2007), has demonstrated the
importance of matching parameter update rules and inference approximation
methods. This paper presents a new update rule based on PAC Bayes bounds, which
is fairly agnostic to the inference algorithm used -- it assumes a
multiplicative error bound on model score and supports both over and under
approximations.
The example given in section 3.2 is a great illustration of how approximation
error is more subtle than we might think it is. Sometimes an approximate
predictor can fit the training data better because it represents a different
family of functions!
The experiments presented in the paper are on NLP problems. However, in NLP the
"workhorse" approximate inference algorithm used is beam search. I strongly
encourage the authors to extend their experiments to include beam search
(varying the beam size to control the approximation quality).
    If, for some reason, beam search does not match their method, it should be
    made very clear.
    Related to beam search is work done by Liang Huang (e.g.,
    http://www.aclweb.org/anthology/N12-1015) about how perceptron with beam
    search can diverge and how to fix the learning algorithm (see "early update"
    and "max violation" updates).
    This paper is pretty influential in NLP.
    This is a case where the learning rule is modified. I think the NLP
    community (at least) would be very interested if beam search could actually
    work with your approach.
Overall, I like this paper and I think that it could have a real impact because
learning with approximate inference is such an important topic and the approach
describe here seems to be pretty effective. Unfortunately, the current state of
the paper is a bit too rough for me to recommend acceptance. (If only we had the
option to "accept with required revisions")
Question
========
I'm confused about Theorem 6.
  a) Isn't this bound looser than the one in theorem 4?  Should making *more*
     assumptions (about \rho *and* \tau) lead to a tighter bound?
  b) I expected \rho to be replaced with a bound based solely on \tau. What's
     the deal?
Am I missing something?
Lower-level comments
====================
The authors assume that the reader is already familiar with the concepts of
under-estimation and over-estimation. I do not believe most readers will know
about this. I strongly recommend defining these terms very early in the paper.
I would also recommend explaining early on why these two classes of
approximations have been studied in the past (and why they are typically studied
separately) -- this would be much more useful than the current discussion in the
related work section.
bottom of page 1: Please define risk? In particular is e(h) /empirical/ risk or
true risk?
39: typo? "We investigate three wildly used structured prediction models"
    -> I think you meant "widely", but I suppose "wildly" also works :P
41: At this point in the paper, I'm very surprised to see "text classification"
    on a list of things that might use approximate inference... It might be
    useful to explain how this is relevant -- it seems as if the answer is that
    it's just a synthetic example.
    There are some settings, such as "extreme classification", where the number
    of labels is huge and approximate inference might be used (e.g., based on
    locality sensitive hashing or approximate nearest neighbors).
42: typo "high order" -> "higher-order"
45: It's tough to say that "Collins [5]" was "the first" since many results for
the /unstructured/ multi-label case technically do transfer over.
47,49,50,etc: Citations are not nouns use use \citet instead of \citep and
\usepackage{natbib}.
   "[37] provided..." -> "Collins (2001) provided ..."
49: "Taskar’s bound" is undefined, presumably it's in one of the citations in
    the paragraph. (Also, you probably mean "Taskar et al.'s bound".)
There are many excellent citations in related work (section 2), but without much
discussion and clear connection to the approach in the paper.
68: "w the parameter." -> "w is the parameter vector." (probably want to say $w
\in \mathbb{R}^d$ as well)
70: clearly over and under estimation method do not partition the space of
approximate inference algorithms: what sorts of algorithms does it omit? does it
matter? Also, what is the role of \rho? Do we need to know it? does it have
constant for all x? (Looks like later, e.g., theorem 4, it's allowed to depend
on the example identifier) Why are multiplicative bounds of interest? Which
approx methods give such bounds? (Later it is generalized based on
stability. Probably worth mentioning earlier in the paper.)
The notation h[w](x) is unusual, I'd expect/prefer h_w(x), or even h(w, x).
73, 74: definitions of H, H- and H+ are a little sloppy and verbose. The
definition of H is pretty useless.
73: The typesetting of R doesn't mach line 68. Please be consistent.
section 3.1
 - L(Q,S,h[w]) seems unnecessary (it's fine to just say that the expectation is over S).
 - Also, S is undefined and the relationship between S and x_i is not explained
   (I assume x_i \in S and |S|=m). Alternatively E_{x \sim S} would do the
   trick.
Lemma 2: Missing citation to PAC-Bayes Theorem.
  Please explain the relationship between the prior p(w') and Q(w'|w). The use
  of a prior comes out of the blue for readers who are unfamiliar with PAC Bayes
  bounds.
  A few words about why I should even be thinking about priors at this point
  would be useful.
  Also, if I don't care about priors can't I optimize the prior away?
77: Should technically pass Q(.|w) to KL divergence, i.e., $D(Q(.|w) || P(.))$.
80: It might be more useful to simply define under and over estimation with this
definition of margin.
theorem 4:
 - assume h'[w] be a ..." -> "theorem 4: assume h'[w] is ..."
 - Should be a \rho_i approximation of h[w](x_i)?
 - what is \bar \cal Y ?
 - why bother subscripting \lambda_S? just use \lambda
90: a better/more-intuitive name for square root term?
107: definition 8, "contains" is not a good term since it's easy to mistaking it
for set containment or something like that. Might be better to call it
"dominates".
Section 3.2 could be summarized as approximate inference algorithms give us a
/different/ family of predictors. Sometimes the approximate inference family has
a better predictor (for the specific dataset) than the exact family.
   Giving high-level intuitions such as the one I provided with resonate more
   with the reader.
Figures 1 and 2 take a little too much effort to understand.
    Consider hoisting some of the discussion from the main text into the
    caption.
    At the very least, explain what the visual elements used mean mean (e.g.,
    What is the gray arc? What are the columns and rows?)
159: "... or by tuning." on development data or training data?
246: "non-comparableness" -> "incomparability"


Neural Variational Inference and Learning in Undirected Graphical Models
Many problems in machine learning are naturally expressed in the language of undirected graphical models. Here, we propose black-box learning and inference algorithms for undirected models that optimize a variational approximation to the log-likelihood of the model. Central to our approach is an upper bound on the logpartition function parametrized by a function q that we express as a flexible neural network. Our bound makes it possible to track the partition function during learning, to speed-up sampling, and to train a broad class of hybrid directed/undirected models via a unified variational inference framework. We empirically demonstrate the effectiveness of our method on several popular generative modeling datasets.

# Overview
The paper presents approximate inference methods for learning undirected graphical models. Learning a Markov random field (MRF)  p(x) involves computing the partition function -- an intractable integral $Z(\theta) = \int \bar p_\theta(x) dx$ (where $\bar p_\theta(x)$ is the energy function). The authors use a tractable importance distribution $q_\phi(x)$ to derive an upperbound on the partition function. They argue that optimising the parameters of p and q under this bound can be seen as variational inference with \chi^2-divergence (instead of KL).
Essentially the proposed methodology circumvents sampling from an MRF (which would require computing the partition function or a form of MCMC) by sampling from an approximation q and using the same approximation to estimate an upperbound on the log partition function of the MRF (which arises in VI's objective). To obtain an approximations q that is potentially multi-modal the authors employ mixture of K components (termed "non-parametric VI" Gershman et al 2012) and auxiliary variables (in the approximation).
The paper then presents 3 applications:
1. 5x5 Ising model: the idea behind this toy application is to assess the quality of the approximate partition function since one can compute the exact one.
2. RBM for MNIST: here they model an RBM p(x) over MNIST-digits using their upperbound on partition function to optimise a lowerbound on the likelihood of the data.
3. Variational auto encoder for (binarised) MNIST: here they model p(x|z)p(z) where the prior p(z) is an RBM.
The main point of the empirical comparison was to show that the proposed method can be used to fit a variational auto-encoder with an RBM prior. The authors contrast that with a variational auto-encoder (with the standard Gaussian prior) whose approximate posterior is augmented with auxiliary random variables as to be able to model multiple posterior modes.
# Qualitative assessment
* Quality
The paper is mostly technically sound, but I would like to have a few points clarified:
1. You refer to equation just before line 54 as ELBO, but the ELBO would involve computations about the posterior, which is really not the case here. I can see the parallel to the ELBO, but perhaps this should be clarified in the text.
2. In line 91, I find the text a bit misleading: "the variance is zero, and a single sample computes the partition function".  If p = q, the bound in Eq (2) is tight and the expectation equals Z(theta)^2, but that does not mean you should be able to compute it with a single sample.
3. Most of the text in Section 3.4 (optimisation) assumes we can simulate x ~ q(x) directly, but that's not really the case when auxiliary variables are employed (where we actually simulate a ~ q(a) and then x ~ q(x|a)). I am curious about how you estimated gradients, did you employ a reparameterisation for q(a)? How did you deal with the KL term from q(a,x) in Section 5.3 where you have a VAE with an RBM prior?
4. Is the gradient in line 159 correct? I think it might be missing a minus.
5. The energy function in line 204 should have been exponentiated, right?
6. Can you clarify why $\log B(\bar p, q)$ in Equation 7 does not affect p's gradient? As far as I can tell, it is a function of p's parameters (\theta).
* Clarity
A few results could have been derived explicitly: e.g. gradients of upperbound with respect to q in each of q's formulation that the paper explored; similarly, the estimator for the case where q is q(a,x).
* Significance
The main advance is definitely interesting and I can imagine people will be interested in the possibility of using MRF priors.


A graph-theoretic approach to multitasking
A key feature of neural network architectures is their ability to support the simultaneous interaction among large numbers of units in the learning and processing of representations. However, how the richness of such interactions trades off against the ability of a network to simultaneously carry out multiple independent processes -a salient limitation in many domains of human cognition -remains largely unexplored. In this paper we use a graph-theoretic analysis of network architecture to address this question, where tasks are represented as edges in a bipartite graph G = (A ∪ B, E). We define a new measure of multitasking capacity of such networks, based on the assumptions that tasks that need to be multitasked rely on independent resources, i.e., form a matching, and that tasks can be multitasked without interference if they form an induced matching. Our main result is an inherent tradeoff between the multitasking capacity and the average degree of the network that holds regardless of the network architecture. These results are also extended to networks of depth greater than 2. On the positive side, we demonstrate that networks that are random-like (e.g., locally sparse) can have desirable multitasking properties. Our results shed light into the parallel-processing limitations of neural systems and provide insights that may be useful for the analysis and design of parallel architectures.

The authors lay an ambitious groundwork for studying the multitasking capability of general neural network architectures. They prove a variety of theorems concerning behavior of their defined criteria--the multitasking capacity \alpha--and how it relates to graph connectivity and size via graph matchings and degree.
Pros:  The problem being studied is quite clearly motivated, and the authors present a genuinely novel new measure for attempting to understand the multitasking capability of networks.  The authors fairly systematically calculate bounds on their parameter in a variety of different graph-theoretic scenarios.
Cons:  It's unclear how practically useful the parameter \alpha will be for real networks currently in use.  Admittedly, this work is preliminary, and the authors point out this shortcoming in their conclusion.  While it is likely the topic of an upcoming study, it would benefit the thrust of the paper considerably if some \alpha's were calculated for networks that are actually in use, or *some* sort of numerical comparison were demonstrated (ex: it's recently become popular to train neural networks to perform multiple tasks simultaneously--what does \alpha have to say about the topologies of these networks?)
Plainly, it's not obvious from a first read of the paper that \alpha *means* anything short of being an abstract quantity one can calculate for graphs.  Additional reads clarify the connection, but making some sort of empirical contact with networks used in practice (beyond the throwaway lines in the conclusion about pseudorandom graphs) would greatly clarify the practical use of (or even intuition for) the \alpha parameter.
That said, this reviewer thinks the article should still be accepted in the absence of such an empirical study, because performing such a study would likely (at least) double the length of the paper (which would be silly).  At the minimum, though, this reviewer would appreciate a bit more discussion of the practical use of the parameter.


PRUNE: Preserving Proximity and Global Ranking for Network Embedding
We investigate an unsupervised generative approach for network embedding. A multi-task Siamese neural network structure is formulated to connect embedding vectors and our objective to preserve the global node ranking and local proximity of nodes. We provide deeper analysis to connect the proposed proximity objective to link prediction and community detection in the network. We show our model can satisfy the following design properties: scalability, asymmetry, unity and simplicity. Experiment results not only verify the above design properties but also demonstrate the superior performance in learning-to-rank, classification, regression, and link prediction tasks.

The paper presents a NN model for learning graph embeddings that preserves the local graph structure and a global node ranking similar to PageRank. The model is based on a Siamese network, which takes as inputs two node embeddings and compute a new (output) representation for each node using the Siamese architecture. Learning is unsupervised in the sense that it makes use only of the graph structure. The loss criterion optimizes a weighted inner product of the output representations (local loss) plus a node ranking criterion (global loss). Some links with a community detection criterion are also discussed. The model is evaluated on a series of tasks: node ranking, classification and regression, link prediction, and compared to other families of unsupervised embedding learning methods.
The paper introduces new loss criteria and an original method for learning graph embeddings. The introduction of a node ranking term based on the graph structure is original. The authors find that this term brings interesting benefits to the learned embeddings and allows their method to outperform the other unsupervised baselines for a series of tasks. The links with the community criteria is also interesting. On the other hand, the organization and the form of the paper makes it somewhat difficult to follow at some places. The motivation for using a Siamese architecture is not explained. Two representations are learned for each node (notations u and z in the paper). Since the z representation is a deterministic function of the u representation, why are these two different representations needed, and why not directly optimizing the initial u representation? The explanation for the specific proximity preserving loss term is also unclear. For example, it is shown that this criterion optimizes a “second-order” proximity, but this term is never properly defined. Even if the result looks interesting, lemma 3.4 which expresses links with a community detection criterion is confusing.
In the experimental section, a more detailed description of the baselines is needed in order to understand what makes them different from the proposed method. The latter outperforms all the baselines on all the tests, but there is no analysis of the reasons for this good behavior.
Overall the paper reveals intriguing and interesting findings. The proposed method has several original aspects, but the paper requires a better organization/presentation in order to fully appreciate the proposed ideas.


PASS-GLM: polynomial approximate sufficient statistics for scalable Bayesian GLM inference
Generalized linear models (GLMs)-such as logistic regression, Poisson regression, and robust regression-provide interpretable models for diverse data types. Probabilistic approaches, particularly Bayesian ones, allow coherent estimates of uncertainty, incorporation of prior information, and sharing of power across experiments via hierarchical models. In practice, however, the approximate Bayesian methods necessary for inference have either failed to scale to large data sets or failed to provide theoretical guarantees on the quality of inference. We propose a new approach based on constructing polynomial approximate sufficient statistics for GLMs (PASS-GLM). We demonstrate that our method admits a simple algorithm as well as trivial streaming and distributed extensions that do not compound error across computations. We provide theoretical guarantees on the quality of point (MAP) estimates, the approximate posterior, and posterior mean and uncertainty estimates. We validate our approach empirically in the case of logistic regression using a quadratic approximation and show competitive performance with stochastic gradient descent, MCMC, and the Laplace approximation in terms of speed and multiple measures of accuracy-including on an advertising data set with 40 million data points and 20,000 covariates.

[EDIT] I have read the rebuttal and I have subsequently increased my score to 7.

# Summary of the paper
When data are too numerous or streaming, the authors investigate using approximate sufficient statistics for generalized linear models. The crux is a polynomial approximation of the link function.
# Summary of the review
The paper is well written and the method is clear and simple. Furthermore, it is computationally super-cheap when the degree of the polynomial approximation is low, a claim which is backed by interesting experiments. I like the underlying idea of keeping it simple when the model allows it. My main concern is the theoretical guarantees, which 1) lack clarity, 2) involve strong assumptions that should be discussed in the main text, and 3) are not all applicable to the recommended experimental setting.
# Major comments
- L47 "constant fraction": some subsampling methods have actually been found to use less than a constant fraction of the data when appropriate control variates are available, see e.g. [Bardenet et al, MCMC for tall data, Arxiv 2015], [Pollock et al., the scalable Langevin [...], Arxiv 2016]. Actually, could your polynomial approximation to the likelihood could be useful as a control variate in these settings as well? Your objection that subsampling requires random data access still seems to hold for these methods, though.
- L119 "in large part due to...": this is debatable. I wouldn't say MCMC is expensive 'because of' the normalizing constant. "Approximating this integral": it is not the initial purpose of MCMC to approximate the normalizing constant, this task is typically carried out by an additional level above MCMC (thermodynamic integration, etc.).
- Section 4: the theoretical results are particularly hard to interpret for several reasons. First, the assumptions are hidden in the appendix, and some of them are quite strong. The strongest assumptions at least should be part of the main text. Second, in -say- Corollary 4.1, epsilon depends on R, and then L207 there's a necessary condition involving R and epsilon, for which one could wonder whether it is ever satisfied. Overall, the bounds are hard to make intuitive sense of, so they should be either simplified or explained. Third, some results like Corollary 4.2 are "for M sufficiently large", while the applications later typically consider M=2 for computational reasons. This should be discussed.
- The computational issues of large d and large M should be investigated further.
- Corollary 4.2 considers a robust loss (Huber's) but with a ball assumption on all data points, and a log concave prior on theta. I would typically use Huber's loss in cases where there are potential outlier inner products of x and theta; is that not in contradiction with the assumptions?
- Corollary 4.3 Assumption (I) is extremely strong. The appendix mentions that it can be relaxed, I think it should be done explicitely.


Learning multiple visual domains with residual adapters
There is a growing interest in learning data representations that work well for many different types of problems and data. In this paper, we look in particular at the task of learning a single visual representation that can be successfully utilized in the analysis of very different types of images, from dog breeds to stop signs and digits. Inspired by recent work on learning networks that predict the parameters of another, we develop a tunable deep network architecture that, by means of adapter residual modules, can be steered on the fly to diverse visual domains. Our method achieves a high degree of parameter sharing while maintaining or even improving the accuracy of domain-specific representations. We also introduce the Visual Decathlon Challenge, a benchmark that evaluates the ability of representations to capture simultaneously ten very different visual domains and measures their ability to perform well uniformly.

Paper address the problem of learning a joint recognition model across variety of domains and tasks. It develops a tunable deep network architecture that, by means of adapter residual modules, can be steered for a variety of visual domains. Methods archives good performance across variety of domains (10 in total) while having high degree of parameter sharing. Paper also introduces a corresponding challenge benchmark - Decathlon Challenge. The benchmark evaluates the ability of a representation to be applied across 10 very different visual domains.

Generally the paper is very well written and presents an interesting approach and insights. I particularly applaud drawing references to related works and putting the approach in true context. Additional comments are below.
> References
Generally the references are good and adequate. The following references are thematically related and should probably be added:
Learning Deep Feature Representations with Domain Guided Dropout for Person Re-identification,
Tong Xiao, Hongsheng Li, Wanli, Ouyang, Xiaogang Wang,
CVPR, 2016.
Expanding Object Detector's Horizon: Incremental Learning Framework for Object Detection in Videos,
Alina Kuznetsova, Sung Ju Hwang, Bodo Rosenhahn, and Leonid Sigal,
CVPR, 2015.
> Clarifications
Lines 201-202 talk about shrinking regularization. I think this is very interesting and could be a major benefit of the model. However, it is unclear if this is actually tested in practice in experiments. Can you please clarify.
Also, authors claim that the proposed model ensures that performance does not degrade in the original domain. This does not actually appears to be the case (or supported experimentally). For example, performance on ImageNet in Table 1 goes from 59.9 to 59.2. This should be discussed and explained.
> Experiments
In light of the comments above, Table 2 should include the proposed model (or models), e.g., Res. adapt. decay and Res. adapt. finetune all.
Finally, explanation of Scratch+ is unclear and should be clarified in Lines 303-305. I read the sentence multiple times and still unsure what was actually done.


Interactive Submodular Bandit
In many machine learning applications, submodular functions have been used as a model for evaluating the utility or payoff of a set such as news items to recommend, sensors to deploy in a terrain, nodes to influence in a social network, to name a few. At the heart of all these applications is the assumption that the underlying utility/payoff function is known a priori, hence maximizing it is in principle possible. In many real life situations, however, the utility function is not fully known in advance and can only be estimated via interactions. For instance, whether a user likes a movie or not can be reliably evaluated only after it was shown to her. Or, the range of influence of a user in a social network can be estimated only after she is selected to advertise the product. We model such problems as an interactive submodular bandit optimization, where in each round we receive a context (e.g., previously selected movies) and have to choose an action (e.g., propose a new movie). We then receive a noisy feedback about the utility of the action (e.g., ratings) which we model as a submodular function over the context-action space. We develop SM-UCB that efficiently trades off exploration (collecting more data) and exploration (proposing a good action given gathered data) and achieves a O( √ T ) regret bound after T rounds of interaction. More specifically, given a bounded-RKHS norm kernel over the context-action-payoff space that governs the smoothness of the utility function, SM-UCB keeps an upperconfidence bound on the payoff function that allows it to asymptotically achieve no-regret. Finally, we evaluate our results on four concrete applications, including movie recommendation (on the MovieLense data set), news recommendation (on Yahoo! Webscope dataset), interactive influence maximization (on a subset of the Facebook network), and personalized data summarization (on Reuters Corpus). In all these applications, we observe that SM-UCB consistently outperforms the prior art.

Pros:
        - The paper combines submodular function maximization with contextual bandit, and propose the problem of interactive submodular bandit formulation and SM-UCB solution. It includes several existing studies as special cases.
        - The empirical evaluation considers several application contexts and demonstrates the effectiveness of the SM-UCB algorithm.
        Cons:
        - The discussion on the key RKHS condition is not enough, and its applicability for actual applications is unclear. More importantly, there is no discussion about constant B for the RKHS norm for various applications, and also the maximum information gain \gamma_t, but both of them are needed in the algorithm.
        - The technical analysis is heavily dependent on prior study, in particular [33]. The additional technical contribution is small.

        The paper studies a contextual bandit learning setting, where in each round a context \phi is presented, which corresponds to an unknown submodular set function f_\phi. The user needs to select one item x to add to the existing selected set S_\phi, with the purpose of maximizing the marginal gain of x given S_\phi. After selecting x, the user obtains a noisy feedback of the marginal gain. The regret is comparing against the (1-1/e) fraction of the optimal offline solution, when all the submodular functions for all contexts are known. The key assumption to make the learning work is that the marginal contribution \Delta in terms of the subset S, the new item x, and the context \phi, has a bounded RKHS norm. Intuitively, this norm makes sure that similar contexts, similar subsets, and similar item would give similar marginal results.
        The authors propose the SM-UCB algorithm to solve the problem. The proposed algorithm and its regret analysis is heavily based on the prior work on contextual Gaussian process bandit optimization [33]. In fact, it looks like all the authors need to do is to treat existing subset S already selected as part of the context, and then plugging it into the algorithm and the analysis of [33]. The proof of Theorem 1 given in the supplementary material is only to connect their current regret definition with the regret definition of [33], using the submodularity property. Then they just apply Theorem 1 of [33] to obtain the result. Therefore, from the technical point of view, the paper's new contribution is small.
        [Note: After reading the authors' feedback, I get a bit clearer understanding, in that the difference in model from [33] seems to be that the current action may affect future contexts. However, I am still not quite sure how much the analysis differs from [33], since there is only a two-and-a-half page analysis in the supplementary material, and it mostly relies on [33] to achieve the main result. Please clarify the technical contribution of the current paper, and what are the novelty in the analysis that differs from [33]]
        Thus, the main contribution of the paper, in my view, is to connecting submodular maximization with prior work on contextual bandit and give the formulation and the result of interactive submodular bandit problem. Their empirical evaluation covers several application cases and demonstrate advantages of the AM-UCB algorithm, giving readers some concrete examples on how to apply their general results in different contexts.
        However, one important thing the authors did not discuss much is the key assumption on the bounded RKHS norm. First, RKHS norm should be better defined and explained. Second, its implication on several application contexts should be discussed. Third, the choice of kernel functions should be discussed. More importantly, the bound B should be evaluated for the applications they discuss and use in the experiment section. This is because this complexity measure B is not only used in the regret bound, but also appears as part of \beta_t in the algorithm. Similar situation occurs for the maximum information gain paramter \gamma_t. The authors do not discuss at all how to obtain these parameters for the algorithm. Without these parameters, it is not even clear how the authors evaluate their SM-UCB algorithm in the experiments. This is a serious ommission, and the readers would not be able to apply their algorithm, or reproduce their experiments without knowing how to properly set B and \gamma_t.
        Furthermore, In the empirical evaluation section, the authors use linear kernel and Jaccard kernel, but why these are reasonable ones? What are their implications to the application context? What are the other possible kernels to use? In general, I do not have a good idea how to pick a good kernel function for an application, and what are the impact and limitations of this assumption. I think this should be the key contribution of the paper, since the other technical contribution is minor, but the authors fail to provide satisfactory answers in this regard.
        Other minor comments:
        - line 133, "T_i" --> "T_j"
        - line 164, I(y_A | \Delta) --> I(y_A; \Delta)
        - Related work
        It would be nice if the authors could compare their approach with other related approaches combining multi-armed bandit with combinatorial optimization tasks, such as
        a) combinatorial multi-armed bandit with application to influence maximization and others: Combinatorial Multi-Armed Bandit and Its Extension to Probabilistically Triggered Arms Wei Chen, Yajun Wang, Yang Yuan, Qinshi Wang, in Journal of Machine Learning Research, 2016
        b) Online influence maximization. Siyu Lei, Silviu Maniu, Luyi Mo, Reynold Cheng, and Pierre Senellart. In KDD, 2015.
        c) Stochastic Online Greedy Learning with Semi-bandit Feedbacks. Tian Lin, Jian Li, Wei Chen, In NIPS'2015


Bayesian GAN
Generative adversarial networks (GANs) can implicitly learn rich distributions over images, audio, and data which are hard to model with an explicit likelihood. We present a practical Bayesian formulation for unsupervised and semi-supervised learning with GANs. Within this framework, we use stochastic gradient Hamiltonian Monte Carlo to marginalize the weights of the generator and discriminator networks. The resulting approach is straightforward and obtains good performance without any standard interventions such as label smoothing or mini-batch discrimination. By exploring an expressive posterior over the parameters of the generator, the Bayesian GAN avoids mode-collapse, produces interpretable and diverse candidate samples, and provides state-of-the-art quantitative results for semi-supervised learning on benchmarks including SVHN, CelebA, and CIFAR-10, outperforming DCGAN, Wasserstein GANs, and DCGAN ensembles.
over the network weights as a point mass centred on a single mode. Thus even if the generator does not memorize training examples, we would expect samples from the generator to be overly compact relative to samples from the data distribution. Moreover, each mode in the posterior over the network weights could correspond to wildly different generators, each with their own meaningful interpretations. By fully representing the posterior distribution over the parameters of both the generator and discriminator, we can more accurately model the true data distribution. The inferred data distribution can then be used for accurate and highly data-efficient semi-supervised learning.
In this paper, we propose a simple Bayesian formulation for end-to-end unsupervised and semisupervised learning with generative adversarial networks. Within this framework, we marginalize the posteriors over the weights of the generator and discriminator using stochastic gradient Hamiltonian Monte Carlo. We interpret data samples from the generator, showing exploration across several distinct modes in the generator weights. We also show data and iteration efficient learning of the true distribution. We also demonstrate state of the art semi-supervised learning performance on several benchmarks, including SVHN, MNIST, CIFAR-10, and CelebA. The simplicity of the proposed approach is one of its greatest strengths: inference is straightforward, interpretable, and stable. Indeed all of the experimental results were obtained without many of the ad-hoc techniques often used to train standard GANs.
We have made code and tutorials available at https://github.com/andrewgordonwilson/bayesgan.

Summary:
The paper introduces a Bayesian type of GAN algorithms, where the generator G and discriminator D do not have any fixed initial set of weights that gets gradually optimised. Instead, the weights for G and for D get sampled from two distributions (one for each), and it is those distributions that get iteratively updated. Different weight realisations of G may thus generate images with different styles, corresponding to different modes in the dataset. This, and the regularisation effect of the priors on the weights, promotes diversity and alleviates the mode collapse issue. The many experiments conducted in the paper support these claims.
Quality, Originality, Clarity:
While most GAN papers today mainly focus on slight variations of the original net architecture or of the original GAN objective function, this paper really presents a new, original and very interesting algorithm that combines features of GANs with a strongly Bayesian viewpoint of Neural Nets (similar in spirit, for example, to the viewpoint conveyed by Radford Neal in his PhD thesis), where the weights are parameters that should be sampled or integrated out, like any other parameter in the Bayesian framework. This leads to a very elegant algorithm. The paper contains many good experiments that prove that the algorithm is highly effective in practice (see Fig 1 and Table 1), and the produced images (though not always state of the art it seems) indeed reflect a high diversity of the samples. Moreover, the paper is very clearly written, in very good english, with almost no typos, and the algorithm is well explained. Overall: a very good paper!
Other remarks:
- Am I right in understanding that at each \theta_g sampling step, you are actually not really sampling from p(\theta_g | \theta_d), but rather from something more like p(\theta_g | \theta_d) * p(theta_d | theta_g^old) (and similar for p(\theta_d | theta_g)) ? In that case, this should be stressed and explained more clearly. In particular, I got confused at first, because I did not understand how you could sample from p(\theta_g | \theta_d) if \theta_d was not fixed.
- Algorithm 1: the inner loop over M does not stick out clearly. The 'append \theta_g^k to sample set' should be inside this loop, but seems outside. Please ensure that both inner- and outer loop are clearly visible from first eye sight.
- l.189 & l.192: clash of notations between z (of dimension d, to generate the synthetic dataset) and z (of dimension 10, as random input to the generator G). Use different variable for the first z
- Figure 1: What is p_{ML-GAN} ? I guess that the 'ML' stays for 'Maximum-Likelihood' (like the maximum-likelihood gan mentioned on l.267 without further introduction.) and that the MLGAN is just the regular GAN. In that case, please call it simply p_{GAN}, or clearly introduce the term Maximum-Likelihood GAN and its abbreviation MLGAN (with a short justification of why it is a maximum likelihood, which would actually be an interesting remark).
- l.162-171: Describe the architectures more carefully. Anyone should be able to implement *exactly* your architecture to verify the experiments. So sentences like ('The networs we use are slightly different from [10] (e.g. we have 4 hidden layers and fewer filters per layer)', without saying exactly how many filters, is not ok. Also, the architecture for the synthetic dataset is described partly at l.162-163 and partly at l.191. Please group the description of the network in one place. The authors may consider describing roughly their architectures in the main part, and put a detailed description in the appendix.
Response to rebuttal:
We thank the authors for answering our questions. We are happy to hear that the code will be released (as there is usually no better way to actually reproduce the experiments with the same parameter settings). Please enclose the short explanations of the rebuttal on GANs being the Maximum Likelihood solution with improper priors into your final version.


Learning Causal Structures Using Regression Invariance
We study causal discovery in a multi-environment setting, in which the functional relations for producing the variables from their direct causes remain the same across environments, while the distribution of exogenous noises may vary. We introduce the idea of using the invariance of the functional relations of the variables to their causes across a set of environments for structure learning. We define a notion of completeness for a causal inference algorithm in this setting and prove the existence of such algorithm by proposing the baseline algorithm. Additionally, we present an alternate algorithm that has significantly improved computational and sample complexity compared to the baseline algorithm. Experiment results show that the proposed algorithm outperforms the other existing algorithms.

The authors propose using regression systematically in a causal setup where an intervention refers to a change in the distribution of some of the exogenous variables. They propose two algorithms for this, one is complete but computationally intractable, whereas the other is a heuristic.
One of the most important problems is the motivation for this new type of intervention. Please provide real world examples of when this type of intervention is realistic.
Another important problem is about Assumption 1. This assumption is very convenient for the analysis but it is very unclear what it entails. How should the variances be related to one another for Assumption 1 to hold? And under what perturbation? Authors should provide a more detailed alternate assumption instead that clearly lays out how noise variances relate to each other, while also considering the causal graph. Otherwise, it is very difficult to assess how restrictive this assumption is.
Line 38: "Note that this model is one of the most problematic models in the literature of causal inference"
-> Additive noise models with Gaussian exogenous variables are arguably one of the most restrictive models. It is not clear if the hardness of identification in this case is relevant for practical questions.
Line 62, 63, 64: "Also since the exogenous noises of both variables have changed, commonly used interventional tests are also not capable of distinguishing between these two structures [4]"
This is not true. As long as the noise variables are independent (they are always independent in Pearlian setting under causal sufficiency), a simple conditional independence test in the two environments, one pre-interventional and one post-interventional, yields the true causal direction (assuming either X1 -> X2 or X1 <- X2). The fact that exogenous variables have different distributions does not disturb this test: Simply observe X1 and X2 are independent under an intervention on X2 to conclude X1->X2.
Line 169: "ordinary interventional tests" are described as the changes observed in the distribution of effect given a perturbation of the cause distribution. However, the more robust interventional test commonly used in the literature checks the conditional independence of the two variables in the post-interventional distribution, as explained in the above comment. Please see "Characterization and Greedy Learning of Interventional Equivalence Classes" by Buhlmann for details.
Based on the two comments above, please do not include "discovering causal graph where interventional methods cannot" in the list of contributions, this is incorrect and severely misleading. I believe the authors should simply say that methods used under perfect interventions cannot be used any more under this type of intervention - which changes the distribution of the exogenous variable - since no new conditional independencies are created compared to the observational distribution. This would be the correct statement, but it is not what the paper currently reads. Please edit the misleading parts of the text.
Example 1 is slightly confusing: Authors use I = {X1} which does not have any subscript as the previous notation I_{i,j}. I believe it would be more clear to write it as I = {\emptyset, X1} instead.
It is not clear if the algorithm LRE is complete, I believe not. Please explicitly mention this in the paper.
The regression invariance set - R - can be indicative of the set of parents of a variable. Authors already make use of this observation in designing the LRE algorithm. What if every exogenous variable changed across every two intervention, i.e., I_{i,j} = [n]. How would you change LRE algorithm for this special case. I would intuitively believe it would greatly simplify it. This may be a useful special case to consider that can boost the usefulness of regression invariant sets.
Another extension idea for the paper is to design the experiments I_{i,j}. What is the minimum number of interventions required?
Simulation results are very preliminary. The results of the real dataset simulations is not clear: Authors have identified a set of possible causes of the number of years of education. Since the true causal graph is unknown it is not clear how to evaluate the results.


OnACID: Online Analysis of Calcium Imaging Data in Real Time
Optical imaging methods using calcium indicators are critical for monitoring the activity of large neuronal populations in vivo. Imaging experiments typically generate a large amount of data that needs to be processed to extract the activity of the imaged neuronal sources. While deriving such processing algorithms is an active area of research, most existing methods require the processing of large amounts of data at a time, rendering them vulnerable to the volume of the recorded data, and preventing real-time experimental interrogation. Here we introduce OnACID, an Online framework for the Analysis of streaming Calcium Imaging Data, including i) motion artifact correction, ii) neuronal source extraction, and iii) activity denoising and deconvolution. Our approach combines and extends previous work on online dictionary learning and calcium imaging data analysis, to deliver an automated pipeline that can discover and track the activity of hundreds of cells in real time, thereby enabling new types of closed-loop experiments. We apply our algorithm on two large scale experimental datasets, benchmark its performance on manually annotated data, and show that it outperforms a popular offline approach.

This paper proposes an online framework for analyzing calcium imaging data. This framework is built upon the popular and now widely used constrained non-negative matrix factorization (CNMF) method for cell segmentation and calcium time-series analysis (Pnevmatikakis, et al., 2016). While the existing CNMF approach is now being used by many labs across the country, I’ve talked to many neuroscientists that complain that this method cannot be applied to large datasets and thus its application has been limited. This work extends this method to a real-time decoding setting, making it an extremely useful contribution for the neuroscience community.
The paper is well written and the results are compelling. My only concern is that the paper appears to combine multiple existing methods to achieve their result. Nonetheless, I think the performance evaluation is solid and an online extension of CNMF for calcium image data analysis and will likely be useful to the neuroscience community.
Major comments:
- There are many steps in the described method that rely on specific thresholds or parameters. It would be useful to understand the sensitivity of the method to these different hyperparameters. In particular, how does the threshold used to include new components affect performance?
- Lines 40-50: This paragraph seems to wander from the point that you nicely set up before this paragraph (and continue afterwards). Not sure why you’re bringing up closed loop, except for saying that by doing this in real time you can do closed-loop experiments. I’m not sure the idea that you want to communicate here.
- The performance evaluations do not really address the use of isotonic regression for deconvolution. In Figure 3C, the traces appear to be demixed calcium after segmentation and not the deconvolved spike trains. Many of the other comparisons focus on cell segmentation. Please comment on the use of deconvolution and how it possibly might help in cell segmentation.
- The results show that the online method outperforms the original CNMF method. Can the authors comment on where the two differ?
Minor comments:
Figure 1: The contours cannot be seen in B. Perhaps a white background in B (and later in the results) can help to see the components and contours. C is hard to see and interpret initially as the traces overlap significantly.