Avoiding Discrimination through Causal Reasoning
Recent work on fairness in machine learning has focused on various statistical discrimination criteria and how they trade off. Most of these criteria are observational: They depend only on the joint distribution of predictor, protected attribute, features, and outcome. While convenient to work with, observational criteria have severe inherent limitations that prevent them from resolving matters of fairness conclusively. Going beyond observational criteria, we frame the problem of discrimination based on protected attributes in the language of causal reasoning. This viewpoint shifts attention from "What is the right fairness criterion?" to "What do we want to assume about our model of the causal data generating process?" Through the lens of causality, we make several contributions. First, we crisply articulate why and when observational criteria fail, thus formalizing what was before a matter of opinion. Second, our approach exposes previously ignored subtleties and why they are fundamental to the problem. Finally, we put forward natural causal non-discrimination criteria and develop algorithms that satisfy them.

The paper proposes a causal view on the question of fair machine learning. Its main contribution is the introduction of causal language to the problem, and specifically the notion of "resolving variable": a variable that mediates the causal effect of a protected attribute in a way the user deems "fair". An example would be the variable "choice of departments" in the famous Simpson's paradox college-admission scenario.
Overall I think this paper has a lot of potential, but still needs some work on clarifying many of the concepts introduced. Fairness is a difficult subject as it clearly goes beyond the math and into societal and normative discussions. As such, I am looking forward to the authors replies to my comments and questions below, hoping to have a fruitful discussion.
Pros:
1. This paper present an important contribution to the discussion on fairness, by introducing the notion of resolving variable. I have already found this term useful in discussions about the subject.
2. The paper shows how in relatively simple scenarios the proposed definitions and methods reduce to very reasonable solutions, for example the result in Theorem 2 and its corollaries.
Cons:
1. No experimental results, not even synthetic. Other papers in the field have experiments, e.g. Hardt Price & Srebro (2016) and Chulechova (2016).
2. Lack of clarity in distinction between proxies and non-resolving variables.
3. I am not convinced of the authors view on the role of proxy variables.
4. I am not convinced that some of the proposed definitions capture the notion of fairness the authors claim to capture.
Specific comments:
1. What exactly is the difference between a proxy variable and a non-resolving variable? Does the difference lie only in the way they are used?
2. Why is the definition of proxy discrimination (def. 3) the one we want?
Wouldn't the individual proxy discrimination be the one which more closely correlates with our ideas of fairness?
E.g., say name (variable P) affects past employment (variable X), which in turn affects the predictor. Couldn't averaging this effect over X's (even considering the do-operator version) lead to a predictor which is discriminating with respect to a specific X?
3. Why is the notion of proxy variables the correct one to control for? I think on the contrary, in many cases we *do* have access to the protected attribute (self-declared race, gender, religion etc.). It's true that conceiving intervention on these variables is difficult, but conceiving the paths through which these variables affect outcomes is more plausible. The reason gender affects employment goes beyond names: it relates to opportunities presented, different societal attitudes, and so far. The name is but one path (an unresolved path) through which gender can affect employment.
4. The wording in the abstract: "what do we want to assume about the causal DGP", seems odd. The causal DGP exists in the world, and we cannot in general change it. Do the authors mean the causal DGP *that gives rise to a predictor R* ?
5. Why have the mentioned approaches to individual fairness not gained traction?
6. The paper would be better if it mentioned similar work which recently appeared on arxiv by Nabi & Shpitser, "Fair Inference on Outcomes". In general I missed a discussion of mediation, which is a well known term in causal inference and seems to be what this paper is aiming at understanding.
7. I am confused by the relation between the predictions R and the true outcomes Y. Shouldn't the parameters leading to R be learned from data which includes Y ?
8. In figure 4, why does the caption say that "we generically do not want A to influence R"? Isn't the entire discussion premised on the notion that sometimes it's ok if A influences R, through a resolving variable?
Minor comments:
1. line 215 I think there's a typo, N_A should be N_P?


Q-LDA: Uncovering Latent Patterns in Text-based Sequential Decision Processes
In sequential decision making, it is often important and useful for end users to understand the underlying patterns or causes that lead to the corresponding decisions. However, typical deep reinforcement learning algorithms seldom provide such information due to their black-box nature. In this paper, we present a probabilistic model, Q-LDA, to uncover latent patterns in text-based sequential decision processes. The model can be understood as a variant of latent topic models that are tailored to maximize total rewards; we further draw an interesting connection between an approximate maximum-likelihood estimation of Q-LDA and the celebrated Q-learning algorithm. We demonstrate in the text-game domain that our proposed method not only provides a viable mechanism to uncover latent patterns in decision processes, but also obtains state-of-the-art rewards in these games.

This paper targets on two text games and propose a new reinforcement learning framework Q-LDA to discover latent patterns in sequential decision process. The proposed model uses LDA to convert action space into a continuous representation and subsequently use Q-learning algorithm to iteratively make decision in a sequential manner.

Authors apply the proposed model to two different text games, and achieve better performance than previous proposed baseline models.

The paper is a little bit hard to follow with some missing or inconsistent information. The paper is not self-contained, for a reader that is not familiar with the problem domain, one may need to refer to the Appendix or prior works almost all the time.
Some detailed comments:

- I would suggest authors to include a detailed section highlighting the contribution of the paper.
- Authors provide some screenshots on the text game interface in the appendix material, but the information of the text games is still short. The reference [11] also doesn't provide much useful context neither. I would recommend authors to include some example text flow (at least in the Appendix) from these games to better illustrate the target scenario. What are and how many is the possible conversation flow of each text game?

- In the context of the game, the agent only receives a reward at the end of game. This is consistent with the text in line 74-75. However, in the graphically model shown in Figure 1. It seems like there is a reward after each turn. I assume this graphical illustration is for general process, but it should be nice to include an explanation in the text.

- In the graphically illustration, it is unclear to me which variables are observable and which are not. For example, all \beta_A and \beta_S are not observable. The rewards r_t's are not observable in my understanding.

- In the generative process, it occurs to me that the observation text W are generated following LDA process as well, but in the graphically model illustration, there is a missing plate in the figure.

- I would encourage authors to include a section of model complexity analysis, for example, what is the complexity of parameter space. Given the size of the dataset and the complexity of the proposed model, it is hard to judge if the learned policy is generalizable.

- In the experiments, what are the vocabulary size of the dataset, in terms of observed text and action text? In Table 2 and Figure 2, authors demonstrate some interesting outcome and observation from the generated topics. I am wondering are these topics being used in other episode, since these topics look very fine-grained and hence may not be applicable to other scenario.

- I would also recommend to add in a list of all topics (maybe in Appendix), rather than the cherry-picked ones, to display.

- In the experiments, authors mention that the upper bound for reward is 20 for "Saving John" and 30 for "Machine of Death". Are these reward numbers objectively specified in the game flow or assigned in a post-hoc manner? If latter, how do you justify this measure (especially if the reward are assigned upon game termination).

- Missing reference in line 76. Missing space in line 106. {D}irichlet in line 291.
%%%%%%%%%%%%%%%%%%%%%%%%
The authors' response have clarified some of my questions and they also agree to improve the paper to make it more self contained. I have adjusted the score accordingly.


Learning Graph Representations with Embedding Propagation
We propose Embedding Propagation (EP), an unsupervised learning framework for graph-structured data. EP learns vector representations of graphs by passing two types of messages between neighboring nodes. Forward messages consist of label representations such as representations of words and other attributes associated with the nodes. Backward messages consist of gradients that result from aggregating the label representations and applying a reconstruction loss. Node representations are finally computed from the representation of their labels. With significantly fewer parameters and hyperparameters an instance of EP is competitive with and often outperforms state of the art unsupervised and semi-supervised learning methods on a range of benchmark data sets.
Despite its conceptual simplicity, we show that EP generalizes several existing machine learning methods for graph-structured data. Since EP learns embeddings by incorporating different label types (representing, for instance, text and images) it is a framework for learning with multi-modal data [31].

The authors introduce embedding propagation (EP), a new message-passing method for learning representations of attributed vertices in graphs.   EP computes vector representations of nodes from the 'labels' (sparse features) associated with nodes and their neighborhood.  The learning of these representations is facilitated by two different types of messages sent along edges: a 'forward' message that sends the current representation of the node, and a 'backward' message that passes back the gradients of some differentiable reconstruction loss.  The authors report results that are competitive with or outperform baseline representation learning methods such as deepwalk and node2vec.
Quality:
The quality of the paper is high.  The experimental technique is clearly described, appears sound, and the authors report results that are competitive with strong baseline methods.  In addition, the authors provide clear theoretical analysis of the worst-case complexity of their method.
Clarity:
The paper is generally clearly written and well-organized.  The proposed model is clearly delineated.
Originality:
The work is moderately original - the authors draw inspiration from message-passing inference algorithms to improve on existing graph embedding methods.
Significance:
On the positive side, the proposed technique is creative and likely to be of interest to the NIPS community. Furthermore, the authors report results that are competitive or better when compared with strong baseline methods, and the computational complexity of the technique is low (linear in both the maximum degree of the graph and the number of nodes).  However, while the results are competitive, it is unclear which method (if any) is offering the best performance.  This is a little troubling because EP is constructing the representations using both node features and graph structure while the baselines only consider graph structure itself, and this raises the question of whether EP is adequately capturing the node features when learning the node representation.
Overall, I believe that the paper is significant enough to warrant publication at NIPS.
Some questions and suggestions for improvement for the authors:
 - It seems like this method could be implemented in a distributed asynchronous manner like standard belief propagation.  If so, the scalability properties are nice - linear computational complexity and a low memory footprint due to the sparse feature input and low-dimensional dense representations.  This could be a good area to explore.
 - It seems like there are really two representations being learned: a representation for a node's labels, and an overall representation for the node that takes the label representation of the node and the representations of the neighboring nodes into account.  How necessary is it to learn a representation for the node labels as a part of the pipeline?  Would it be possible to just use the raw sparse features (node labels)?  If so, how would that affect performance?
 - Is it possible to learn this representation in a semi-supervised manner?
 - Learning curves would be nice.  If the method is using fewer parameters, perhaps it's less inclined to overfit in a data-poor training regime?

Overall impression:
A good paper that proposes an interesting technique that gives solid (although not spectacular) results.  An interesting idea for others to build on.  Accept.


Gradient Episodic Memory for Continual Learning
One major obstacle towards AI is the poor ability of models to solve new problems quicker, and without forgetting previously acquired knowledge. To better understand this issue, we study the problem of continual learning, where the model observes, once and one by one, examples concerning a sequence of tasks. First, we propose a set of metrics to evaluate models learning over a continuum of data. These metrics characterize models not only by their test accuracy, but also in terms of their ability to transfer knowledge across tasks. Second, we propose a model for continual learning, called Gradient Episodic Memory (GEM) that alleviates forgetting, while allowing beneficial transfer of knowledge to previous tasks. Our experiments on variants of the MNIST and CIFAR-100 datasets demonstrate the strong performance of GEM when compared to the state-of-the-art.

The authors of the manuscript consider the continuum learning setting, where the learner observes a stream of data points from training, which are ordered according to the tasks they belong to, i.e. the learner encounters any data from the next task only after it has observed all the training data for the current one. The authors propose a set of three metrics for evaluation performance of learning algorithms in this setting, which reflect their ability to transfer information to new tasks and not forget information about the earlier tasks. Could the authors, please, comment on the difference between continuum and lifelong learning (the corresponding sentence in line 254 seems incomplete)?
The authors also propose a learning method, termed Gradient of Episodic Memory (GEM). The idea of the method is to keep a set of examples from every observed task and make sure that at each update stage the loss on the observed tasks does not increase. It seems like there are some typos in eq. (6), because in its current form I don’t see how it captures this idea.
GEM is evaluated on a set of real-world datasets against a set of state-of-the-art baselines. I have a few questions regarding this evaluation:
1. comparison to icarl: if I understand correctly, icarl solves a multi-class classification problem with new classes being added over time, i.e. by the end of training on CIFAR-100 it would be able to solve a full 100-classes problem, while GEM would solve only any of 20 5-classes problems. How was the evaluation of icarl performed in this experiment? Was it ever given a hint which of 5 classes to look at?
2. it also seems from table 1 that the results in figure 1 for GEM and icarl were obtained for different memory sizes. If this is true, could the authors comment on why?
3. on MNIST rotations task multi-task method has less negative transfer than GEM, but its accuracy is lower. Could the authors, please, comment on why? Or these differences are non-significant?
4. It would be very interesting to see performance of baseline (multi-task for MNIST and/or multi-class for CIFAR) that gets all the data in a batch, to evaluate the gap that remains there between continuum learning methods, like GEM, and standard batch learning.
5. were the architectures, used for baseline methods (multi-task and icarl), the same as for GEM?


Efficient and Flexible Inference for Stochastic Systems
Many real world dynamical systems are described by stochastic differential equations. Thus parameter inference is a challenging and important problem in many disciplines. We provide a grid free and flexible algorithm offering parameter and state inference for stochastic systems and compare our approch based on variational approximations to state of the art methods showing significant advantages both in runtime and accuracy.

***Update following reviewer discussion and author feedback***
I am happy to revise my score for the paper provided the authors add some discussion of the number of OU processes used in the simulations, the \delta term in the RODE system and add the volatility to the Lorenz96 model (in addition to the other changes to the text recommended by myself and the other reviewers).
**************************************************************
The authors propose a method for combined state and parameter estimation for stochastic differential equation (SDE) models.  The SDE is first transformed into a random ordinary differential equation.  Several solution paths are then simulated to generate a large number of ordinary differential equations, and each of these is then solved using an EM algorithm type approach that was introduced in an earlier paper.  The method is tested on two systems, the Lorenz96 and Lorenz63 models, and compared to a competitor method showing that the new approach can be faster and more accurate.
There are some interesting ideas in the paper but I can’t accept it for publication in its current form.  The general approach seems reasonable, but there are some details of it that the authors don’t really mention that I think need to be explored.  There are also lots of things that I found unclear in the manuscript and I think these need to be fixed.  Since there is no major revision option for the conference I must therefore recommend rejection.  I think if the authors revise the paper accordingly it could be accepted at a similar conference in the future.
Detailed comments:
I think is section 3 you need to spend more time discussing two things:
How many OU processes you should be simulating.  You don’t seem to discuss it but it must be crucial to the performance of the approach in practice.
Why the additional \delta term is added to each RODE system, and how its variance should be chosen.
The main reason for these suggestions is that the approach itself borrows innovations from other sources (e.g. Gorbich et al. 2017).  Because of this, the specifics of how these tools are applied in the SDE context is to my mind the main contribution of the present paper. If you don’t discuss these things then it isn’t clear how useful the paper is.
The first line states ‘A dynamical system is represented by a set of K stochastic differential equations…’ This is not true in general, many dynamical systems cannot be represented by stochastic differential equation models (e.g. when the driving noise is not Brownian).
line 56 why is equation (2) a scalar RODE?  I would have thought that it was a d-dimensional system of RODEs.  Or does scalar not mean what I think it means here?
In (2) the vector field is described as f(x(t),w(t)), whereas in (2) it is written f(x(t),\eta(w)).  Shouldn’t it instead be f(x(t),\eta(t)) in (2) and f(x(t),\eta(t,w)) in (3), since \eta(t) is a r.v. and \eta(t,w) will be the fixed outcome for w \in \Omega?
Line 60.  Here you introduce Example 1, but then fail to discuss it for the next paragraph.  Then you introduce Example 2 which is essentially the second part of example 1.  I think it would be better if you combined example 1 and 2, since the purpose of the example is to show how a RODE can be re-written as an SDE when the driving noise is Brownian. And also put it where example 2 currently is (i.e. below the paragraph of text).
Line 87.  ‘Stationary' for the OU process can either mean it is started at equilibrium or that it is positive recurrent (i.e. it will be ergodic).  Which of these do you mean here?  From the definition I think it should be the latter since the SDE could be conditioned to start at any point, but this could certainly cause some confusion I think.
Line 89.  Why an OU process?  Can the authors offer any intuition here, as it seems to come out of nowhere and I think the reader would be very interested to understand the origins.
Line 167.  Is the Lorenz96 system therefore an SDE with unit volatility?  I think it would be good to clarify this for non-experts.
Typos
line 24 diffusions processes -> diffusion processes
line 44 the fact the we -> the fact that we
line 48 ‘both frameworks are highly related’ doesn’t make sense.  If two things are related then the statement clearly applies to both of them.
line 54 is a R^m valued -> be an R^m valued
line 99 an computationally efficient -> a computationally efficient
line 113 an product of experts -> a product of experts
line 119 an lower bound -> a lower bound
line 174 in pur approach -> in our approach (I think?)
line 190 robuts -> robust
line 198 random ordinary differential equation -> random ordinary differential equations


When Cyclic Coordinate Descent Outperforms Randomized Coordinate Descent
The coordinate descent (CD) method is a classical optimization algorithm that has seen a revival of interest because of its competitive performance in machine learning applications. A number of recent papers provided convergence rate estimates for their deterministic (cyclic) and randomized variants that differ in the selection of update coordinates. These estimates suggest randomized coordinate descent (RCD) performs better than cyclic coordinate descent (CCD), although numerical experiments do not provide clear justification for this comparison. In this paper, we provide examples and more generally problem classes for which CCD (or CD with any deterministic order) is faster than RCD in terms of asymptotic worst-case convergence. Furthermore, we provide lower and upper bounds on the amount of improvement on the rate of CCD relative to RCD, which depends on the deterministic order used. We also provide a characterization of the best deterministic order (that leads to the maximum improvement in convergence rate) in terms of the combinatorial properties of the Hessian matrix of the objective function.

In this paper, the authors analyze cyclic and randomized coordinate descent and show that despite the common assumption in the literature, cyclic CD can actually be much faster than randomized CD. The authors show that this is true for quadratic objectives when A is of a certain type (e.g., symmetric positive definite with diagonal entries of 1, irreducible M-matrix, A is consistently ordered, 2-cyclic matrix).
Comments:
- There are more than 2 variants of CD selection rules: greedy selection is a very valid selection strategy for structured ML problems (see "Coordinate Descent Converges Faster with the Gauss-Southwell Rule than Random Selection, ICML 2015 by Nutini et. al.)
- The authors state that it is common perception that randomized CD always dominates cyclic CD. I don't agree with this statement. For example, if your matrix is diagonal, cyclic CD will clearly do better than randomized CD.
- The authors make several assumptions on the matrices considered in their analysis (e.g., M-matrix, 2-cyclic). It is not obvious to me that these matrices are common in machine learning applications when solving a quadratic problem. I think the authors need to do a better job at convincing the reader that these matrices are important in ML applications, e.g., precision matrix estimation https://arxiv.org/pdf/1404.6640.pdf "Estimation of positive definite M-matrices and structure learning for attractive Gaussian Markov Random fields" by Slawski and Hein, 2014.
- The authors should be citing "Improved Iteration Complexity Bounds of Cyclic Block Coordinate Descent for Convex Problems" NIPS 2015 (Sun & Hong)
- I have several issues with the numerical results presented in this paper. The size of the problem n = 100 is small. As coordinate descent methods are primarily used in large-scale optimization, I am very curious why the authors selected such a small system to test. Also, it seems that because of the structure of the matrices considered, there is equal progress to be made regardless of the coordinate selected -- thus, it seems obvious that cyclic would work better than random, as random would suffer from re-selection of coordinates, while cyclic ensures updating every coordinate in each epoch. In other words, the randomness of random selection would be unhelpful (especially using a uniform distribution). Can the authors please justify these decisions.
Overall, I'm not convinced that the analysis in this paper is that informative. It seems that the assumptions on the matrix naturally lend to using cyclic selection as there is equal progress to be made by updating any coordinate. At least that is what the numerical results are showing. I think the authors need to further convince that this problem setting is important and realistic to ML application and that their numerical results emphasize again the *realistic* benefits of cyclic selection.
============
POST REBUTTAL
============
I have read the author rebuttal and I thank the authors for their details comments. My comment regarding "equal progress to be made" was with respect to the structure in the matrix A and the authors have address this concern, pointing my attention to the reference in [20]. I think with the inclusion of the additional references mentioned by the authors in the rebuttal that support the applicability of the considered types of matrices in ML applications, I can confidently recommend this paper for acceptance.


Variational Walkback: Learning a Transition Operator as a Stochastic Recurrent Net
We propose a novel method to directly learn a stochastic transition operator whose repeated application provides generated samples. Traditional undirected graphical models approach this problem indirectly by learning a Markov chain model whose stationary distribution obeys detailed balance with respect to a parameterized energy function. The energy function is then modified so the model and data distributions match, with no guarantee on the number of steps required for the Markov chain to converge. Moreover, the detailed balance condition is highly restrictive: energy based models corresponding to neural networks must have symmetric weights, unlike biological neural circuits. In contrast, we develop a method for directly learning arbitrarily parameterized transition operators capable of expressing nonequilibrium stationary distributions that violate detailed balance, thereby enabling us to learn more biologically plausible asymmetric neural networks and more general non-energy based dynamical systems. The proposed training objective, which we derive via principled variational methods, encourages the transition operator to "walk back" (prefer to revert its steps) in multi-step trajectories that start at datapoints, as quickly as possible back to the original data points. We present a series of experimental results illustrating the soundness of the proposed approach, Variational Walkback (VW), on the MNIST, CIFAR-10, SVHN and CelebA datasets, demonstrating superior samples compared to earlier attempts to learn a transition operator. We also show that although each rapid training trajectory is limited to a finite but variable number of steps, our transition operator continues to generate good samples well past the length of such trajectories, thereby demonstrating the match of its non-equilibrium stationary distribution to the data distribution. Source

This paper proposes an extension of the information destruction/reconstruction processes introduced by Sohl-Dickstein et al. (i.e. NET). Specifically, the authors propose learning an explicit model for the information destroying process, as opposed to defining it a priori as repeated noising and rescaling. The authors also propose tying the parameters of the forwards/backwards processes, with the motivation of more efficiently seeking to eliminate spurious modes (kind of like contrastive divergence in undirected, energy-based models).
The general ideas explored in this paper are interesting -- i.e. training models which generate data through stochastic iterative refinement, and which provide mechanisms for "wandering around" the data manifold. However, the proposed model is a fairly straightforward extension of NET and does not produce compelling quantitative or qualitative results. The general perspective taken in this paper, i.e. reinterpreting the NET approach in the context of variational inference, was presented earlier in Section 2.3 of "Data Generation as Sequential Decision Making" by Bachman et al. (NIPS 2015). If more ambitious extensions of the ideas latent in NET were explored in this paper, I would really like it. But, I don't see that yet. And, e.g., the CIFAR10 results are roughly what one gets when simply modelling the data using a full-rank Gaussian distribution. This would make an interesting workshop paper, but I don't think there's enough strong content yet to fill a NIPS paper.


Net-Trim: Convex Pruning of Deep Neural Networks with Performance Guarantee
We introduce and analyze a new technique for model reduction for deep neural networks. While large networks are theoretically capable of learning arbitrarily complex models, overfitting and model redundancy negatively affects the prediction accuracy and model variance. Our Net-Trim algorithm prunes (sparsifies) a trained network layer-wise, removing connections at each layer by solving a convex optimization program. This program seeks a sparse set of weights at each layer that keeps the layer inputs and outputs consistent with the originally trained model. The algorithms and associated analysis are applicable to neural networks operating with the rectified linear unit (ReLU) as the nonlinear activation. We present both parallel and cascade versions of the algorithm. While the latter can achieve slightly simpler models with the same generalization performance, the former can be computed in a distributed manner. In both cases, Net-Trim significantly reduces the number of connections in the network, while also providing enough regularization to slightly reduce the generalization error. We also provide a mathematical analysis of the consistency between the initial network and the retrained model. To analyze the model sample complexity, we derive the general sufficient conditions for the recovery of a sparse transform matrix. For a single layer taking independent Gaussian random vectors of length N as inputs, we show that if the network response can be described using a maximum number of s non-zero weights per node, these weights can be learned from O(s log N ) samples.

The paper presents a technique to sparsify a deep ReLU neural network by solving a sequence of convex problems at each layer. The convex problem finds the sparsest set of weights that approximates the mapping from one layer to another. The ReLU nonlinearity is dealt with by treating the activated and deactivated cases as two separate sets of constraints in the optimization problem, thus, bringing convexity.
Two variants are provided, one that considers each layer separately, and another that carries the approximation in the next layer optimization problem to give the chance to the next layer to counterbalance this error. In both cases, the authors provide bounds on the approximation error after sparsification.
The paper is very clear and well-written. A number of theoretical results come in support to the proposed method.
The experiments section shows results for various networks (fully connected, and convolutional) on the MNIST data. The results are compared with a baseline which consists of removing weights with smallest magnitude from the network. The authors observe that their method works robustly while the baseline methods lands in some plateau with very low accuracy.
The baseline proposed here looks particularly weak. Setting a large number of weights to zero slashes the variance of the neuron preactivations, and the negative biases will tend to drive activations zero. Actually, in the HPTD paper, the authors obtain the best results by iteratively pruning the weights.
A pruning ratio of 80% is quite low. Other approaches such as SqueezeNet have achieved 98% parameter reduction on image data without significant drop of performance.


Multitask Spectral Learning of Weighted Automata
We consider the problem of estimating multiple related functions computed by weighted automata (WFA). We first present a natural notion of relatedness between WFAs by considering to which extent several WFAs can share a common underlying representation. We then introduce the novel model of vector-valued WFA which conveniently helps us formalize this notion of relatedness. Finally, we propose a spectral learning algorithm for vector-valued WFAs to tackle the multitask learning problem. By jointly learning multiple tasks in the form of a vector-valued WFA, our algorithm enforces the discovery of a representation space shared between tasks. The benefits of the proposed multitask approach are theoretically motivated and showcased through experiments on both synthetic and real world datasets.

SUMMARY
The paper studies the problem of multitask learning of WFAs. It defines a notion of relatedness among tasks, and designs a new algorithm that can exploit such relatedness. Roughly speaking, the new algorithm stacks the Hankel matrices from different tasks together and perform an adapted version of spectral learning, resulting in a vv-WFA that can make vector-valued predictions with a unified state representation. A post-processing step that reduces the dimension of the WFA for each single task is also suggested to reduce noise. The algorithm is compared to the baseline of learning each task separately on both synthetic and real-world data.
COMMENTS
Overall this is a well written paper. However, I do have a concern in the experiment section: it is important to compare to the baseline where all data from different tasks are bagged together and treated as if they came from the same task. At least when all the tasks are the same, this should outperform everyone else as it makes full use of all the data. Of course, when the tasks are not related, such a practice may lead to asymptotic approximation error, but how large is this error practically? If this error is small on the datasets used in the experiment section, then such datasets are not interesting as any algorithm that does some kind of data aggregation would show improvement over single-task learning. If possible I would like to see some results (even if they are primary) on this comparison during rebuttal.
It would be good to also compare to Alg 1 without the projection step to see how much improvement this post-processing procedure brings.
The paper's presentation may be improved by discussing the application scenario of multi-task learning of WFAs. As a starter, one could consider natural language modeling tasks where we need to make predictions in different contexts (e.g., online chat vs newspaper articles) and have access to datasets in each of them. In this example, it is natural to expect that basic grammar is shared across the datasets and can be learned together. Of course, one can always aggregate all datasets into a big one and build a single model (which corresponds to the baseline I mentioned above), and the disadvantage is that the model cannot leverage the context information available at prediction phase.
Two additional suggestions:
- The current algorithm implicitly assumes equal weights among all the tasks. This should work well when the size of the datasets are roughly the same across tasks, but when they differ a lot I suspect that the algorithm could misbehave. In this case you might want to consider a weighted approach; see Kulesza et al, Low-Rank Spectral Learning with Weighted Loss Functions.
- Here is another reason for doing the projection step: consider the case when the m tasks are completely unrelated, and each of them requires n states. Single-task learning would need n*m^2 parameters for each character in the alphabet, while the multi-task learning uses a model of size (nm)^2. The projection step eliminates such redundancy.
MINOR ISSUE
Line 93: as far as I know, it is not required that empty string is included in prefixes or suffixes. (At least this is true in the PSR literature which I am more familiar with.) The author(s) might want to double check on this.
==============================
Thanks for the rebuttal and the additional results. No complaints! Will keep arguing for acceptance.


Nonbacktracking Bounds on the Influence in Independent Cascade Models
This paper develops upper and lower bounds on the influence measure in a network, more precisely, the expected number of nodes that a seed set can influence in the independent cascade model. In particular, our bounds exploit nonbacktracking walks, Fortuin-Kasteleyn-Ginibre type inequalities, and are computed by message passing algorithms. Nonbacktracking walks have recently allowed for headways in community detection, and this paper shows that their use can also impact the influence computation. Further, we provide parameterized versions of the bounds that control the trade-off between the efficiency and the accuracy. Finally, the tightness of the bounds is illustrated with simulations on various network models.

The paper develops upper and lower bounds, using correlation inequalities, on the value of the influence function (expected influenced set size) under the Independent Cascade model. The work is a clear improvement on the upper bounds of Lemonnier et al., NIPS 2014. The theoretical results are a solid contribution towards a bound-based analysis of influence computation and maximization, though the evaluation could be improved to better evaluate and communicate the impact of the results.
1) As future work building on this line of bounds, It would be interesting to see work connecting them to actual influence maximization, e.g. running a greedy IM procedure, and comparing the seed sets under greedy IM vs. seed sets returned by greedy maximization over the UB or LB measures. It would have been nice to see some dialogue with maximization as part of the work, but it would have been difficult to fit into the short format.
2) In the Experimental Results (Section 4), it would be preferable to see the evaluation decomposed over the two main sources of randomness being studied, the graph and the seed node {s]. It would be interesting to dig into properties of the seed nodes for which the bounds are tighter or looser (degree? Centrality? etc.) It would also be preferable to see an evaluation of larger seed sets, as part of the difficulty with computing influence is interactions between seed nodes.
3) An obvious omission is the study of real world graphs, which have higher degrees and contain many more triangles than any of the simulated graphs considered in the paper. The higher degrees make backtracking steps in random walks less frequent. The triangles and other cycles generally impede message passing algorithms. It’d be worth evaluating and commenting on these considerations.
4) The state of the art for fast IM computation with approximation guarantees is, I believe, (Tang-Xiao-Shi, SIGMOD 14). Worth including in the citations to IM papers. Furthermore, the use of the word “heuristic” on Line 20 is usually reserved for algorithms without formal guarantees. The methods in the citation block [10, 17, 3, 7, 20] almost all feature formal guarantees for IM in terms of worst-case approximation ratios (some in probabilistic senses). To call all these papers “heuristic” feels overly dismissive. As a related comment on the literature review, on Line 32 "A few exceptions include” makes it sound like there is going to be more than one exception, but only one is given. Fine to rewrite to only expect on exception, better to give more cites.


Contrastive Learning for Image Captioning
Image captioning, a popular topic in computer vision, has achieved substantial progress in recent years. However, the distinctiveness of natural descriptions is often overlooked in previous work. It is closely related to the quality of captions, as distinctive captions are more likely to describe images with their unique aspects. In this work, we propose a new learning method, Contrastive Learning (CL), for image captioning. Specifically, via two constraints formulated on top of a reference model, the proposed method can encourage distinctiveness, while maintaining the overall quality of the generated captions. We tested our method on two challenging datasets, where it improves the baseline model by significant margins. We also showed in our studies that the proposed method is generic and can be used for models with various structures.

The paper proposed a contrastive learning approach for image captioning models. Typical image captioning models utilize log-likelihood criteria for learning, which tends to result in preferring a safer generation that lacks specific and distinct concept in an image. The paper proposes to introduce contrastive learning objective, where the objective function is based on density ratio to the reference, without altering the captioning models. The paper evaluates multiple models in MSCOCO and InstaPIC datasets, and demonstrates the effectiveness as well as conducts ablation studies.
The paper is well-written and has strength in the following points.
* Proposing a generalizable learning method
* Convincing empirical study
The paper could be improved on the following respect.
* Results might look insignificant depending on how to interpret
* Insufficient discussion on distinctiveness vs. human-like description
For the purpose of introducing distinctiveness in the image captioning problem, the paper considers altering the learning approach using existing models. The paper takes contrastive learning ideas from NCE [5], and derives an objective function eq (10). By focusing on the contrastive component in the objective, the paper solves the problem of learning under MLE scheme that results in a generic description. Although the proposed objective is similar to NCE, the approach is general and can benefit in other problem domains. This is certainly a technical contribution.
In addition, the paper conducts a thorough empirical study to show the effectiveness as well as to generalization across base models and datasets. Although the result is not necessarily the best all the time depending on the evaluation scenario, I would point out the proposed approach is independent of the base model yet consistently improving the performance over the MLE baseline.
One thing I would point out is that the paper could discuss more on the nature of distinctiveness in image captions. As discussed in the introduction, distinctiveness is certainly one component overlooked in caption generation. However, the paper’s view on distinctiveness is something that can be resolvable by algorithms. I would like to argue that the nature of caption data can induce generic description due to the vagueness of image content [Jas 2015]. Even if an image can be described by distinctive phrases, people can describe the content by comfortable words that are not always distinctive [Ordonez 2013]. In this respect, I would say the choice of the dataset may or may not be appropriate for evaluating distinctive phrases. The paper can add more on data statistics and human behavior on image description.
* Jas, Mainak, and Devi Parikh. "Image specificity." CVPR 2015.
* Ordonez, Vicente, et al. "From large scale image categorization to entry-level categories." ICCV 2013.
Another concern is that image captioning is extensively studied in the past and unfortunately the paper might not be able to impact a lot in the community.
In overall, the paper is well written and presenting convincing results for the concerned problem. Some people might not like the small improvements in the performance, but I believe the result also indicates a good generalization ability. I think the paper is above the borderline.


REBAR: Low-variance, unbiased gradient estimates for discrete latent variable models
Learning in models with discrete latent variables is challenging due to high variance gradient estimators. Generally, approaches have relied on control variates to reduce the variance of the REINFORCE estimator. Recent work (Jang et al., 2016;Maddison et al., 2016) has taken a different approach, introducing a continuous relaxation of discrete variables to produce low-variance, but biased, gradient estimates. In this work, we combine the two approaches through a novel control variate that produces low-variance, unbiased gradient estimates. Then, we introduce a modification to the continuous relaxation and show that the tightness of the relaxation can be adapted online, removing it as a hyperparameter. We show state-of-the-art variance reduction on several benchmark generative modeling tasks, generally leading to faster convergence to a better final log-likelihood.
to significantly improve its effectiveness. We call this the REBAR gradient estimator, because it combines REINFORCE gradients with gradients of the Concrete relaxation. Next, we show that a modification to the Concrete relaxation connects REBAR to MuProp in the high temperature limit. Finally, because REBAR is unbiased for all temperatures, we show that the temperature can be optimized online to reduce variance further and relieve the burden of setting an additional hyperparameter.
In our experiments, we illustrate the potential problems inherent with biased gradient estimators on a toy problem. Then, we use REBAR to train generative sigmoid belief networks (SBNs) on the MNIST and Omniglot datasets and to train conditional generative models on MNIST. Across tasks, we show that REBAR has state-of-the-art variance reduction which translates to faster convergence and better final log-likelihoods. Although we focus on binary variables for simplicity, this work is equally applicable to categorical variables (Appendix C).

This paper introduces a control variate technique to reduce the variance of the REINFORCE gradient estimator for discrete latent variables. The method, called REBAR, is inspired by the Gumble-softmax/Concrete relaxations; however, in contrast to those, REBAR provides an unbiased gradient estimator. The paper shows a connection between REBAR and MuProp. The variance of the REBAR estimator is compared to state-of-the-art methods on sigmoid belief networks. The paper focuses on binary discrete latent variables.
Overall, I found this is an interesting paper that addresses a relevant problem; namely, non-expensive low-variance gradient estimators for discrete latent variables. The writing quality is good, although there are some parts that weren't clear to me (see comments below). The connections to MuProp and Gumble-softmax/Concrete are clear.
Please find below a list with detailed comments and concerns:
- The paper states several times that p(z)=p(z|b)p(b). This is confusing, as p(z|b)p(b) should be the joint p(z,b). I think that the point is that the joint can be alternatively written as p(z,b)=p(b|z)p(z), where the first term is an indicator function, which takes value 1 if b=H(z) and zero otherwise, and that it motivates dropping this term. But being rigorous, the indicator function should be kept. So p(b|z)p(z)=p(z|b)p(b), and when b=H(z), then p(z)=p(z|b)p(b). I don't think the derivations in the paper are wrong, but this issue was confusing to me and should be clarified.
- It is not clear to me how the key equation in the paper was obtained (the unnumbered equation between lines 98-99). The paper just reads "Putting the terms together, we arrive at", but this wasn't clear to me. The paper would definitely be improved by adding these details in the main text or in the Appendix.
- In the same equation, the expectation w.r.t. p(u,v), where u, v ~ Uniform(0,1) is also misleading, because they're not independent. As Appendix 7.5 reads, "a choice of u will determine a corresponding choice of v which produces the same z", i.e., p(u,v)=p(u)p(v|u), with p(v|u) being deterministic. In other words, I don't see how "using this pair (u,v) as the random numbers" is a choice (as the paper suggests) rather than a mathematical consequence of the procedure.
- The paper states that REBAR is equally applicable to the non-binary case (lines 44-45). I think the paper can be significantly improved by including the mathematical details in the Appendix to make this explicit.
- In the experimental section, all figures show "steps" (of the variational procedure) in the x-axis. This should be replaced with wall-clock time to get a sense of the computational complexity of each approach.
- In the experiments, it would be interesting to see a comparison with "Local expectation gradients" [Titsias & Lazaro-Gredilla].
- In the experimental section, do the authors have any intuition on why MuProp excels in the structured prediction task for Omniglot? (Figs. 6 and 7)
- The procedure to optimize temperature (Sec 3.2) seems related to the procedure to optimize temperature in "Overdispersed black-box VI" [Ruiz et al.]; if that's the case, the connection can be pointed out.
- The multilayer section (Sec 3.3) seems a Rao-Blackwellization procedure (see, e.g., [Ranganath et al.]). If that's the case, the authors should mention that.
- In line 108, there seems to be a missing 1/lambda term.
- In lines 143-144 and 193, there are missing parenthesis in the citations.
- Consider renaming Appendix 7 as Appendix 1.


Random Permutation Online Isotonic Regression
We revisit isotonic regression on linear orders, the problem of fitting monotonic functions to best explain the data, in an online setting. It was previously shown that online isotonic regression is unlearnable in a fully adversarial model, which lead to its study in the fixed design model. Here, we instead develop the more practical random permutation model. We show that the regret is bounded above by the excess leave-one-out loss for which we develop efficient algorithms and matching lower bounds. We also analyze the class of simple and popular forward algorithms and recommend where to look for algorithms for online isotonic regression on partial orders.

I am not an expert in online learning, and did not read the proofs in the appendix. My
overall impression of the paper is positive, but I am not able to judge the importance of
the results or novelty of the analysis techniques. My somewhat indifferent score is more a
reflection of this than the quality of the paper.
Summary: The authors study isotonic regression in an online setting, where an adversary
initially chooses the dataset but the examples are shown in random order. Regret is
measured against the best isotonic function for the data set.
The main contribution of the paper seems to be in Section 4, i.e a class of "forward
algorithms" which encompass several well known methods, achieve sqrt(T) regret. The authors
also prove a bunch of complementary results such as lower bounds and results for different
settings and loss functions.
Can the authors provide additional motivation for studying the random permutation model?
I don't find the practical motivation in lines 46-47 particularly convincing. It might
help to elaborate on what is difficult/interesting for the learner this setting and what
makes the analysis different from existing work (e.g. [14]).
Section 3: How do Theorems 3.1 and 3.2 fit into the story of the paper? Are they simply
some complementary results or are they integral to the results in Section 4?
- The estimator in (3) doesn't seem computationally feasible. Footnote 2 states that the
  result holds in expectation if you sample a single data and permutation but this is
  likely to have high variance. Can you comment on how the variance decreases when you
  sample multiple data points and multiple permutations?
Clarity: Despite my relative inexperience in the field, I was able to follow most of the
details in the paper. That said, I felt the presentation was math driven and could be
toned down in certain areas, e.g. lines 149-157, 242-250

While there seem to a some gaps in the results, the authors have been fairly thorough in
exploring several avenues (e.g. sections 4.3-4.6). The paper makes several interesting
contributions that could be useful for this line of research.
----------------------------------------------------
Post rebuttal: I have read the authors' rebuttal and am convinced by their case for the setting. I have upgraded my score.


Best of Both Worlds: Transferring Knowledge from Discriminative Learning to a Generative Visual Dialog Model
We present a novel training framework for neural sequence models, particularly for grounded dialog generation. The standard training paradigm for these models is maximum likelihood estimation (MLE), or minimizing the cross-entropy of the human responses. Across a variety of domains, a recurring problem with MLE trained generative neural dialog models (G) is that they tend to produce 'safe' and generic responses ('I don't know', 'I can't tell'). In contrast, discriminative dialog models (D) that are trained to rank a list of candidate human responses outperform their generative counterparts; in terms of automatic metrics, diversity, and informativeness of the responses. However, D is not useful in practice since it can not be deployed to have real conversations with users. Our work aims to achieve the best of both worlds -the practical usefulness of G and the strong performance of D -via knowledge transfer from D to G. Our primary contribution is an end-to-end trainable generative visual dialog model, where G receives gradients from D as a perceptual (not adversarial) loss of the sequence sampled from G. We leverage the recently proposed Gumbel-Softmax (GS) approximation to the discrete distribution -specifically, a RNN augmented with a sequence of GS samplers, coupled with the straight-through gradient estimator to enable end-to-end differentiability. We also introduce a stronger encoder for visual dialog, and employ a self-attention mechanism for answer encoding along with a metric learning loss to aid D in better capturing semantic similarities in answer responses. Overall, our proposed model outperforms state-of-the-art on the VisDial dataset by a significant margin (2.67% on recall@10).

This paper describes an improved training procedure for visual dialogue models.
Rather than maximizing the likelihood of a collection of training captions, this
approach first trains a discriminator model to rank captions in a given context
by embedding them in a common space, then uses scores from this discriminator as
an extra component in a loss function for a generative sequence prediction
model. This improved trainin procedure produces modest improvements on an
established visual dialogue benchmark over both previous generative approaches
as well as adversarial training.
I think this is a pretty good paper, though there are a few places in which
the presentation could be improved.
SPECIFIC COMMENTS
The introduction claims that the discriminator "has access to more information
than the generator". I'm not clear on what this means. It is
true that in a single update step, the discriminator here is trained to
distinguish the ground truth answer from a collection of answers, while in the
standard conditional GAN setting the discriminator compares only one sample at a
time to the ground truth. But over the course of training a normal GAN
discriminator will also get access to lots of samples from the training
distribution. More generally, there's an equivalence between the problem of
learning to assign the right probability to a sample from a distribution and
learning to classify whether a sample came from the target distribution or a
noise distribution (see e.g. the NCE literature). Based on this section, I
expected the discriminator to be able to learn features on interactions between
multiple candidates, and was confused when it wound up assigning an embedding to
each candidate independently.
Related work: given the offered interpretation of D as a "perceptual loss", I
was surprised not to see any mention in the related work section of similar loss
functions used image generation tasks (e.g. Johnson et al). I was not able to
find anything in a quick lit search of similar losses being used for natural
language processing tasks, which is one of the things that's exciting about this
paper, but you should still discuss similar work in other areas.
@137 it's standard form
The explanation offered for the objective function at 177 is a little strange.
This is a logistic regression loss which encourages the model to assign highest
probability to the ground-truth caption, and I think it will be most
understandable to readers if presented in that familiar form. In particular, it
does not have an interpretation as a metric-learning procedure unless the norms
of the learned representations are fixed (which doesn't seem to be happening
here), and is not a margin loss.
I don't understand what "assigned on-the-fly" @186 means. Just that
you're reusing negative examples across multiple examples in the same batch?
@284 modifications to encoder alone
Table captions should explain that MRR -> "mean reciprocal rank" and especially
"mean" -> "mean rank".


DPSCREEN: Dynamic Personalized Screening
Screening is important for the diagnosis and treatment of a wide variety of diseases. A good screening policy should be personalized to the features of the patient and to the dynamic history of the patient (including the history of screening). The growth of electronic health records data has led to the development of many models to predict the onset and progression of different diseases. However, there has been limited work to address the personalized screening for these different diseases. In this work, we develop the first framework to construct screening policies for a large class of disease models. The disease is modeled as a finite state stochastic process with an absorbing disease state. The patient observes an external information process (for instance, self-examinations, discovering comorbidities, etc.) which can trigger the patient to arrive at the clinician earlier than scheduled screenings. The clinician carries out the tests; based on the test results and the external information it schedules the next arrival. Computing the exactly optimal screening policy that balances the delay in the detection against the frequency of screenings is computationally intractable; this paper provides a computationally tractable construction of an approximately optimal policy. As an illustration, we make use of a large breast cancer data set. The constructed policy screens patients more or less often according to their initial risk -it is personalized to the features of the patient -and according to the results of previous screens -it is personalized to the history of the patient. In comparison with existing clinical policies, the constructed policy leads to large reductions (28-68%) in the number of screens performed while achieving the same expected delays in disease detection.

The objective of the paper is to find the best policy for patient screening given the pertinent information. To provide the policy, a disease should be modeled as a finite state stochastic process. The policy is trying to minimize screening costs and delays. The authors propose an approximate solution that is computationally efficient. The experiments on simulated data related to breast cancer indicate that the proposed algorithm could reduce delays while keeping the same cost when compared to a trivial baseline
  Positives:
+ personalized screening is an important topic of research
+ the proposed algorithm is reasonable and well-developed
+ the results are promising
+ the paper is well written
  Negatives:
- the proposed algorithm is purely an academic exercise. It assumes that the disease model of a given patient is known, which could be never assumed in practice. The main obstacle in personalized screening is not coming up with a policy when the model is known, but inferring the model from data.
- the methodological novelty is not large
- the experiments only compare the proposed algorithm to a baseline doing annual screening. Given the previous point about the disease model, an important question that should be studies experimentally is the robustness of the proposed method to inaccuracies in the disease model. In other words, how sensitive is the policy to uncertainties in the disease model.
  Overall, the topic is interesting and the proposed algorithm is reasonable. However, the methodological novelty is not large and the paper would need to do a better work to convince readers that this work is practically relevant.


Variational Laws of Visual Attention for Dynamic Scenes
Computational models of visual attention are at the crossroad of disciplines like cognitive science, computational neuroscience, and computer vision. This paper proposes a model of attentional scanpath that is based on the principle that there are foundational laws that drive the emergence of visual attention. We devise variational laws of the eye-movement that rely on a generalized view of the Least Action Principle in physics. The potential energy captures details as well as peripheral visual features, while the kinetic energy corresponds with the classic interpretation in analytic mechanics. In addition, the Lagrangian contains a brightness invariance term, which characterizes significantly the scanpath trajectories. We obtain differential equations of visual attention as the stationary point of the generalized action, and we propose an algorithm to estimate the model parameters. Finally, we report experimental results to validate the model in tasks of saliency detection.

Variational Laws of Visual Attention for Dynamic Scenes
The authors investigate what locations in static and dynamic images tend to be attended to by humans. They derive a model by first defining three basic principles for visual attention (defined as an energy function to be minimized by the movements of the eye): (1) Eye movements are constrained by a harmonic oscillator at the borders of the image within a limited-sized retina. (2) a “curiosity driven principle” highlighting the regions with large changes in brightness in both a fine and blurred version of the image, and (3) brightness invariance, which increases as a function of changes in brightness. Using a cost function derived from these three functions, the authors derive differential equations for predicting the eye movements across static or dynamic images (depending on the starting location and initial velocity). The authors evaluate their technique quantitatively on data sets of static and dynamic scenes coupled with human eye movements. They demonstrate that their method performs comparable to the state-of-the-art.
Formal definitions of saliency and modeling eye movements are critical issues in computational vision and cognitive science. Psychologists have long been plagued by vague definitions of saliency, and the authors propose a novel and innovative model (as far as I am aware) that could aid the development of better understanding how what makes something salient and a formal model for eye movements (within the bottom-up tradition). Although it is not necessarily state-of-the-art on every metric for every data set, it performs well and provides a refreshingly different perspective on the problem.
Unfortunately, some of what I wrote above is based on conjecture as the paper is poorly written and hard to follow. I recommend the authors have others proofread the paper and expand on abbreviations (both within the equations and also those used in Section 3). I would recommend they move the 2 page Appendix to supplementary material and use those extra pages to define each variable and function used (even if it is a convention within your own field – the NIPS audience comes from many different disciplines and some will have trouble following the mathematics otherwise).
As a psychologist, I would have liked the authors to connect their work to some of the psychological literature on eye movements as optimal steps for gathering information. See for example, Najemnik, J., & Geisler, W. S. (2008). Eye movement statistics in humans are consistent with an optimal search strategy. Journal of Vision, 8(3), 1-14. There are other relevant articles (particularly from Geisler’s lab, but that should give the authors a pointer to follow into that literature). I would be interested to see a discussion of how their approach compares to their findings (e.g., are they restatements of similar models or provide independent information that could be integrated to produce a better model?)


A framework for Multi-A(rmed)/B(andit) Testing with Online FDR Control
We propose an alternative framework to existing setups for controlling false alarms when multiple A/B tests are run over time. This setup arises in many practical applications, e.g. when pharmaceutical companies test new treatment options against control pills for different diseases, or when internet companies test their default webpages versus various alternatives over time. Our framework proposes to replace a sequence of A/B tests by a sequence of best-arm MAB instances, which can be continuously monitored by the data scientist. When interleaving the MAB tests with an online false discovery rate (FDR) algorithm, we can obtain the best of both worlds: low sample complexity and any time online FDR control. Our main contributions are: (i) to propose reasonable definitions of a null hypothesis for MAB instances; (ii) to demonstrate how one can derive an always-valid sequential p-value that allows continuous monitoring of each MAB test; and (iii) to show that using rejection thresholds of online-FDR algorithms as the confidence levels for the MAB algorithms results in both sample-optimality, high power and low FDR at any point in time. We run extensive simulations to verify our claims, and also report results on real data collected from the New Yorker Cartoon Caption contest.

The paper looks at continuous improvement using a sequence of A/B tests, and proposes instead to implement adaptive testing such as
      multi-armed bandit problem while controlling the false discovery rate. This is an important problem discussed in statistical literature, but still unsolved.
      The approach proposed in this paper seems to apparently solve the issues. This is a very interesting paper, that despite minor concerns listed below, could lead to a potentially new avenue of research.

      Lines 24-37: There are well-known issues, and it would be desirable to add citations. Although authors clearly focus on CS/ML literature, there is also a relevant body of literature in biometrics, see e.g. survey by Villar, Bowden and Wason (Statistical Science 2015), the references therein and the more recent papers citing this survey.
      Line 37: "testing multiple literature" -> "multiple testing literature"
      Line 38-47: A similar concept exists in biometrics, called "platform trials" - please describe how your concept differs
      Line 112: "and and" -> "and"
      Line 115: please provide reference for and description of LUCB
      Line 153: "samplesas" -> "samples as"
      Line 260: "are ran" -> ?
      Line 273: It is not clear what "truncation time" is and why it is introduced - it seems to have a huge effect on the results in Figure 2
      Line 288-291: While this motivation is interesting, it seems to be mentioned at an inappropriate place in the paper - why not to do it in the introduction, alongside website management and clinical trials?


Recursive Sampling for the Nyström Method
We give the first algorithm for kernel Nyström approximation that runs in linear time in the number of training points and is provably accurate for all kernel matrices, without dependence on regularity or incoherence conditions. The algorithm projects the kernel onto a set of s landmark points sampled by their ridge leverage scores, requiring just O(ns) kernel evaluations and O(ns 2 ) additional runtime. While leverage score sampling has long been known to give strong theoretical guarantees for Nyström approximation, by employing a fast recursive sampling scheme, our algorithm is the first to make the approach scalable. Empirically we show that it finds more accurate kernel approximations in less time than popular techniques such as classic Nyström approximation and the random Fourier features method.

The authors provide an algorithm which learns a provably accurate low-rank approximation to a PSD matrix in sublinear time. Specifically, it learns an approximation that has low additive error with high probability by sampling columns from the matrix according to a certain importance measure, then forming a Nystrom approximation using these kernels. The importance measure used is an estimate of the ridge leverage scores, which are expensive to compute exactly ( O(n^3) ). Their algorithm recursively estimates these leverage score by starting from a set of columns, using those to estimate the leverage scores, sampling a set of columns according to those probabilities, and repeating ... the authors show that when this process is done carefully, the leverage score estimates are accurate enough that they can be used to get almost as good an approximation as using the true ridge leverage scores. The cost of producing the final approximation is O(ns^2) computation time and O(ns) computations of entries in the PSD matrix.
This is the first algorithm which allows touching only a linear number of entries in a PSD matrix to obtain a provably good approximation --- previous methods with strong guarantees independent of the properties of the PSD matrix required forming all n^2 entries in the PSD matrix. The experimental results show that the method provides more accurate kernel approximations than Nystrom approximations formed using uniform column samples, and kernel approximations formed using Random Fourier Features. These latter two are currently the most widely used randomized low-rank kernel approximations in ML, as they are cheap to compute. A heuristic modification of the algorithm brings its runtimes down to close to that of RFFs and uniform sampled Nystrom approximations, but retains the high accuracy.
I see one technical problem in the paper: the proof of Lemma 5, in the display following line 528, the first inequality seems to use the fact that if W is a diagonal matrix and 0 <= W <= Id in the semidefinite sense, then for any PSD matrix A, we have WAW <= A. This is not true, as can be seen by taking A to be the rank one outer product of [1;-1], W to be diag(1, 1/2), and x = [1, 1]. Then x^TWAWx > x^tAx = 0. Please clarify the proof if the first inequality holds for some other reason, or otherwise establish the lemma.
Comments:
-In section 3.2, when you introduce effective dimension, you should also cite "Learning Bounds for Kernel Regression Using Effective Data Dimensionality", Tong Zhang, Neural Computation, 2005. He calls the same expression the effective dimension of kernel methods
-Reference for the claim on lines 149-150? Or consider stating it as a simple lemma for future reference
-line 159, cite "On the Impact of Kernel Approximation on Learning Accuracy", Cortes et al., AISTATS, 2010 and "Efficient Non-Oblivious Randomized Reduction for Risk Minimization with Improved Excess Risk Guarantee", Xu et al., AAAI, 2017
-line 163, the reference to (13) should actually be to (5)
-on line 3 of Alg 2, I suggest changing the notation of the "equals" expression slightly so that it is consistent with the definition of \tilde{l_i^lambda} in Lemma 6
-the \leq sign in the display preceding line 221 is mistyped
-the experimental results with Gaussian kernels are very nice, but I think it would be useful to see how the method performs on a wider range of kernels.
-would like to see "Learning Kernels with Random Features" by Sinha and Duchi referenced and compared to experimentally
-would like to see "Learning Multidimensional Fourier Series With Tensor Trains", Wahls et al., GlobalSIP, 2014 cited as a related work (they optimize over frequencies to do regression versus 'random' features)


Regret Minimization in MDPs with Options without Prior Knowledge
The option framework integrates temporal abstraction into the reinforcement learning model through the introduction of macro-actions (i.e., options). Recent works leveraged the mapping of Markov decision processes (MDPs) with options to semi-MDPs (SMDPs) and introduced SMDP-versions of exploration-exploitation algorithms (e.g., RMAX-SMDP and UCRL-SMDP)  to analyze the impact of options on the learning performance. Nonetheless, the PAC-SMDP sample complexity of RMAX-SMDP can hardly be translated into equivalent PAC-MDP theoretical guarantees, while the regret analysis of UCRL-SMDP requires prior knowledge of the distributions of the cumulative reward and duration of each option, which are hardly available in practice. In this paper, we remove this limitation by combining the SMDP view together with the inner Markov structure of options into a novel algorithm whose regret performance matches UCRL-SMDP's up to an additive regret term. We show scenarios where this term is negligible and the advantage of temporal abstraction is preserved. We also report preliminary empirical results supporting the theoretical findings.

Overview:
The authors attempt to improve current regret estimation for HRL methods using options. In particular they attempt to do so in the absence of a distribution of cumulative rewards and of option durations, which is a requirement of previous methods (UCRL and SUCRL).
After assuming that options are well defined, the authors proceed to transform the inner MDP of options, represented by a transition matrix Po, into an equivalent irreducible Markov chain with matrix P'o. This is done by merging the terminal states to the initial state.
By doing so, and assuming that any state with a termination probability lower than one can be reached, the stationary distribution mu_o of the chain is obtainable; which in turn is utilized to estimate the optimistic reward gain. With respect to previous methods, this formulation is more robust to ill-defined estimates of the parameters, and better accounts for the correlation between cumulative reward and duration. This method is coined as FSUCRL.
The algorithm is complemented by estimating confidence intervals for the reward r, for the SMDP transition probabilities, and for the inner Markov Chain P'o. Two versions of this same algorithm, FUSCRL Lvl1 and Lvl2, are proposed. The first version requires directly computing an approximated distribution mu_o from the estimate of P'o. The second version nests two empirical value iterations to obtain the optimal bias.
The paper concludes with a theoretical analysis and a numerical application of FSUCLR. On a theoretical ground, FSUCRL Lvl2 is compared to SUCRL. The authors argue that the goodness of the bound on the regret predicted by FSUCRL Lvl2 compared to that of SUCRL depends various factors, including the length of the options and the accessibility of states within the option itself, and provide conditions where FSUCRL Lvl2 is likely to perform better than SUCRL.
As an indication, 4 algorithms, UCRL, SUCRL, FSUCRL Lvl1 and FSUCRL Lvl 2 are tested on a gridworld taken from ref. 9, where the maximum duration of the options is assumed known. Result confirm the theoretical expectations previously discussed: FSUCRL Lvl2 performs better of both SUCRL and FSUCRL Lvl1, partially due to the fact that the options' actions overlap.
Evaluation:
- Quality: The paper appears to be theoretically sound and the problem discussion is complete. The authors discuss in depth strength and weaknesses of their approach with respect to the previous SUCRL. The provided numerical simulation is not conclusive but supports the above considerations;
- Clarity: the paper could be clearer but is sufficiently clear. The authors provide an example and a theoretical discussion which help understanding the mathematical framework;
-  Originality: the work seems to be sufficiently original with respect to its predecessor (SUCRL) and with respect to other published works in NIPS;
- Significance: the motivation of the paper is clear and relevant since it addresses a significant limitation of previous methods;
Other comments:
- Line 140: here the first column of Qo is replaced by vo to form P'o, so that the first state is not reachable anymore but from a terminating state. I assume that either Ass.1 (finite length of an option) or Ass. 2 (the starting state is a terminal state) clarify this choice. In the event this is the case, the authors should mention the connection between the two;
- Line 283: "four" -> "for";
- Line 284: "where" s-> "were";


Mixture-Rank Matrix Approximation for Collaborative Filtering
Low-rank matrix approximation (LRMA) methods have achieved excellent accuracy among today's collaborative filtering (CF) methods. In existing LRMA methods, the rank of user/item feature matrices is typically fixed, i.e., the same rank is adopted to describe all users/items. However, our studies show that submatrices with different ranks could coexist in the same user-item rating matrix, so that approximations with fixed ranks cannot perfectly describe the internal structures of the rating matrix, therefore leading to inferior recommendation accuracy. In this paper, a mixture-rank matrix approximation (MRMA) method is proposed, in which user-item ratings can be characterized by a mixture of LRMA models with different ranks. Meanwhile, a learning algorithm capitalizing on iterated condition modes is proposed to tackle the non-convex optimization problem pertaining to MRMA. Experimental studies on MovieLens and Netflix datasets demonstrate that MRMA can outperform six state-of-the-art LRMA-based CF methods in terms of recommendation accuracy.

This is an excellent paper, proposing a sound idea of approximating a partially defined rating matrix with a combination of multiple low rank matrices of different ranks in order to learn well the head user/item pairs (users and items with lots of ratings) as well as the tail user/item pairs (users and items we few ratings). The idea is introduced clearly. The paper makes a good review of the state-of-the-art, and the experiment section is solid with very convincing results.
In reading the introduction, the reader could find controversial the statement in lines 25-27 about the correlation between the number of user-item ratings and the desired rank. One could imagine that a subgroup of users and items have a large number of ratings but in a consistent way, which can be explained with a low rank matrix. The idea is getting clear further in the paper, when explained in the light of overfitting and underfitting. The ambiguity could be avoided in this early section by adding a comment along the line of “seeking a low rank is a form of regularization”.


A Meta-Learning Perspective on Cold-Start Recommendations for Items
Matrix factorization (MF) is one of the most popular techniques for product recommendation, but is known to suffer from serious cold-start problems. Item cold-start problems are particularly acute in settings such as Tweet recommendation where new items arrive continuously. In this paper, we present a meta-learning strategy to address item cold-start when new items arrive continuously. We propose two deep neural network architectures that implement our meta-learning strategy. The first architecture learns a linear classifier whose weights are determined by the item history while the second architecture learns a neural network whose biases are instead adjusted. We evaluate our techniques on the real-world problem of Tweet recommendation. On production data at Twitter, we demonstrate that our proposed techniques significantly beat the MF baseline and also outperform production models for Tweet recommendation.

This is an interesting and well-written paper but there are some parts that are not well explained, hence my recommendation. These aspects are not clear:
1. I am not sure about the "meta-learning" part. The recommendation task is simply formulated as a binary classification task (without using matrix factorization). The relation to meta-learning is not convincing to me.
2. "it becomes natural to take advantage of deep neural networks (the common approach in meta-learning)" - this is not a valid claim  - deep learning is not the common approach for meta-learning; please see the papers by Brazdil and also the survey by Vilaltra & Drissi.
3. What is the input to the proposed 2 neural network architectures and what is its dimensionality? This should be clearly described.
4. I don't understand why in the first (liner) model the bias is constant and the weights are adapted and the opposite applies for the second (non-linear) model - the weights are fixed and the biases are adapted. This is an optimisation task and all parameters are important. Have you tried adaption all parameters?
5. The results are not very convincing - a small improvement, that may not be statistically significant.
6. Evaluation
-Have you sampled the same number of positive and negative examples for each user when creating the training and testing data?
-How was the evaluation done for items that were neither liked or disliked by a user?
-Why are the 3 baselines called "industrial"? Are they the typically used baselines?
-Is your method able to generate recommendations to all users or only for some? Is it able to recommend all new items? In other words, what is the coverage of your method?
-It will be useful to compare you method with a a pure content-based recommender. Is any of the beselines purely content-based?
7. Discuss the "black-box" aspect of using neural networks for making recommendations (lack of interpretability)
These issues need to be addressed and explained during the rebuttal.


Linear regression without correspondence
This article considers algorithmic and statistical aspects of linear regression when the correspondence between the covariates and the responses is unknown. First, a fully polynomial-time approximation scheme is given for the natural least squares optimization problem in any constant dimension. Next, in an average-case and noise-free setting where the responses exactly correspond to a linear function of i.i.d. draws from a standard multivariate normal distribution, an efficient algorithm based on lattice basis reduction is shown to exactly recover the unknown linear function in arbitrary dimension. Finally, lower bounds on the signal-to-noise ratio are established for approximate recovery of the unknown linear function by any estimator.

The article "Linear regression without correspondence" considers the problem of estimation in linear regression model in specific situation where the correspondence between the covariates and the responses is unknown. The authors propose the fully polynomial algorithms for the solution of least squares problem and also study the statistical lower bounds.
The main emphasis of the article is on the construction of fully polynomial algorithms for least squares problem in noisy and noiseless case, while previously only the algorithms with exponential complexity were known for the cases with dimension d > 1. For the noisy case the authors propose the algorithm which gives a solution of least squares problem with any prespecified accuracy. For noiseless case another algorithm is proposed, which gives the exact solution of the least squares problem. Finally, the authors prove the upper bound for the range of signal to noise ratio values for which the consistent parameter recovery is impossible.
In general, the proposed algorithms, though being not practical, help to make an important step in understanding of computational limits in the linear regression without correspondence. The statistical analysis is limited to lower bound while for the upper bound the authors refer to the paper by Pananjady et al. (2016). What puzzles me a lot is that provided lower bound for parameters recovery is d / log log n, while in Pananjady et al. (2016) the lower bound for permutation recovery is proved to be n^c for some c > 0. Moreover, in another paper Pananjady et al. (2017) the constant upper bound on prediction risk is proved. While all these results consider the different quantities to be analyzed, it is seems that the fundamental statistical limits in this problem is far from being well understood.
To sum up, I think that the paper presents a strong effort in the interesting research direction and the results are sound. However, I believe that the main impact of the paper is on computational side, while some additional statistical analysis is highly desired for this problem. Also I believe that such type of paper, which includes involved theoretical analysis as well as algorithms which are more technical than intuitive, is much more suitable for full journal publication than for short conference paper.


An Empirical Study on The Properties of Random Bases for Kernel Methods
Kernel machines as well as neural networks possess universal function approximation properties. Nevertheless in practice their ways of choosing the appropriate function class differ. Specifically neural networks learn a representation by adapting their basis functions to the data and the task at hand, while kernel methods typically use a basis that is not adapted during training. In this work, we contrast random features of approximated kernel machines with learned features of neural networks. Our analysis reveals how these random and adaptive basis functions affect the quality of learning. Furthermore, we present basis adaptation schemes that allow for a more compact representation, while retaining the generalization properties of kernel machines.

Summary:
The authors provided an empirical study contrasting neural networks and kernel methods, with a focus on how random and adaptive schemes would make efficient use of features in order to improve quality of learning, at four levels of abstraction: data-agnostic random basis (baseline kernel machines with traditional random features), unsupervised data-adaptive basis for better approximation of kernel function, supervised data-label-adaptive basis by kernel target alignment, discriminatively adaptive basis (neural nets). The paper concluded with several suggestions and caveats for efficient use of random features in practice.
Comments:
- 1 -
Line 123, especially for sake of comparing UAB case where the underlying assumption is that using the true kernel function k in prediction yields the "best" performance so that UAB tries to approximate it, I would suggest testing in experiments a baseline model that utilizes the true kernel function k in prediction. Also this would suggest, for example in Fig. 1 at which point of the KAE curve the accuracy is sufficiently good (despite many theoretical results available).
- 2 -
The four datasets chosen in the paper certainly demonstrate proof for conclusions finally drawn. However, in order to support those conclusions to a definitively convincing extent, more datasets should be needed. For example, the performance scores in Tab. 1 do not seem to be too significantly different marginally for each task. And reasons for inconsistent behaviors across different tasks (CoverType vs others in Fig. 1 for instance, Line 222) are not well explained through empirical exploration.
- 3 -
It is not clear how big the difference can be between the four adaptive schemes in terms of training time, which can be crucial in practice. In addition, as the number of (approximate) features D increases, how does the training time increase accordingly in general for each scheme empirically? It would be interesting to also report this.


Hiding Images in Plain Sight: Deep Steganography
Steganography is the practice of concealing a secret message within another, ordinary, message. Commonly, steganography is used to unobtrusively hide a small message within the noisy regions of a larger image. In this study, we attempt to place a full size color image within another image of the same size. Deep neural networks are simultaneously trained to create the hiding and revealing processes and are designed to specifically work as a pair. The system is trained on images drawn randomly from the ImageNet database, and works well on natural images from a wide variety of sources. Beyond demonstrating the successful application of deep learning to hiding images, we carefully examine how the result is achieved and explore extensions. Unlike many popular steganographic methods that encode the secret message within the least significant bits of the carrier image, our approach compresses and distributes the secret image's representation across all of the available bits.

The authors present a new steganography technique based on deep neural networks to simultaneously conduct hiding and revealing as a pair. The main idea is to combine two images of the same size together. The trained process aims to compress the information from the secret image into the least noticeable portions of the cover image and consists of three processes: a prep-Network for encoding features, the Hiding Network creates a container image, and a Reveal Network for decoding the transmitted container image. On the positive side, the proposed technique seems novel and clever, although it uses/modifies existing deep learning frameworks and therefore should be viewed as an application paper. The experiments are comprehensive and the results are convincing.
The technique, however, resembles greatly the image decomposition problem, e.g., for separating mixtures of the intrinsic and reflection layers in previous literatures. I'd hope the authors to clarify how the problems are different and if the proposed technique can be used to resolve the layer separation problem. More important, I wonder if existing approaches on layer separations can be used to directly decode your encrypted results.
I am also a bit concerned about the practicality of the proposed technique.  First, since the container image will be transmitted and potentially intercepted. It appears that one can tell directly that the container image contains hidden information (the image contains ringing type visual artifacts). if that is the case, the approach is likely to undermine the effort. Second, if the cover and secret images appear similar, the technique may fail to robustly separate them. So a more interesting question is how to pick a suitable cover image for a specific secret image. But such discussions seem largely missing. Third, the requirement the cover and the secrete images should have the same size seems to be a major limitation. One would have to resize the images to make them match, it may increase file sizes, etc.
Overall, I think the proposed approach is interesting but I have concerns on the practicality and would like to see comparisons with state-of-the-art layer separation techniques.


Submultiplicative Glivenko-Cantelli and Uniform Convergence of Revenues
In this work we derive a variant of the classic Glivenko-Cantelli Theorem, which asserts uniform convergence of the empirical Cumulative Distribution Function (CDF) to the CDF of the underlying distribution. Our variant allows for tighter convergence bounds for extreme values of the CDF. We apply our bound in the context of revenue learning, which is a well-studied problem in economics and algorithmic game theory. We derive sample-complexity bounds on the uniform convergence rate of the empirical revenues to the true revenues, assuming a bound on the kth moment of the valuations, for any (possibly fractional) k > 1. For uniform convergence in the limit, we give a complete characterization and a zero-one law: if the first moment of the valuations is finite, then uniform convergence almost surely occurs; conversely, if the first moment is infinite, then uniform convergence almost never occurs.

This paper provides a generalization of the Glivenko-Cantelli theorem, a "submultiplicative" compromise between additive and multiplicative errors. The two main results go hand in hand, the first dealing with existence of a sufficient index which guarantees submultiplicativity and the second providing an upper bound on such an index in order to provide ease in applying the results. It is clear that only submultiplicative results are possible due to a simple counterexample given. The proofs of the main results are technical, but mathematically clear.
Throughout the paper, the author(s) familiarity with previous work with generalizing the Glivenko-Cantelli theorem is verified, and the novelty of the work is demonstrated.
The author(s) apply these results to the problem of estimating revenue in an auction via empirical revenues. It is clear that understanding revenue estimation better will lend itself to revenue maximization. Moreover, investigating the connection between finite moments and uniform convergence in this example can offer a framework for exploring other estimators in a similar way.
While the paper is well-written, and I believe the results to be significant and even fundamental, I felt that the example given, while interesting, was quite specialized and detracted somewhat from generality of the paper. Even more detail in the discussion section on the potential impact to the learning theory community would be great.
Additional notes:
The definition of F_n(t) on lines 22 and 23 is hard to read.
The phrasing on lines 66-68 is awkward.
Between lines 104 and 105, why not continue with notation similar to that introduced in Theorem 1.3 for consistency?


First-Order Adaptive Sample Size Methods to Reduce Complexity of Empirical Risk Minimization
This paper studies empirical risk minimization (ERM) problems for large-scale datasets and incorporates the idea of adaptive sample size methods to improve the guaranteed convergence bounds for first-order stochastic and deterministic methods. In contrast to traditional methods that attempt to solve the ERM problem corresponding to the full dataset directly, adaptive sample size schemes start with a small number of samples and solve the corresponding ERM problem to its statistical accuracy. The sample size is then grown geometrically -e.g., scaling by a factor of two -and use the solution of the previous ERM as a warm start for the new ERM. Theoretical analyses show that the use of adaptive sample size methods reduces the overall computational cost of achieving the statistical accuracy of the whole dataset for a broad range of deterministic and stochastic first-order methods. The gains are specific to the choice of method. When particularized to, e.g., accelerated gradient descent and stochastic variance reduce gradient, the computational cost advantage is a logarithm of the number of training samples. Numerical experiments on various datasets confirm theoretical claims and showcase the gains of using the proposed adaptive sample size scheme.

The paper proposes an adaptive sample size strategy applied to first order methods to reduce the complexity of solving the ERM problems. By definition, the ERM problem respect to a given dataset D is to minimize the average loss over it. Instead of handling the full dataset all at once, the proposed algorithm starts with a small subset of D and first minimize the ERM problem respect to this subset. After reaching the desired statistical accuracy on this small problem, it doubles the size of the subset and update the problem with the new subset. The strategy repeats such procedure until the full dataset is included. The paper shows an improvement both in theoretical analysis and in experiments.
The paper is very clear, the theoretical proof is correct and well presented. I find the paper very interesting and I like the idea of taking into account the statistical problem behind the ERM problem. A question I have is that the algorithm requires to predefine the statistical accuracy and uses it explicitly as a parameter in the construction of the algorithm, is there any way to adapt it by removing its dependency in the algorithm? Because it is demonstrated in the experiments that the choice of such accuracy do influence the performance. (In MNIST, the performance of taking 1/n is better than 1/sqrt{n}) The current algorithm will stop once the predefine accuracy is attained which is eventually improvable by varying it.
Besides, I am a bit concerned about the novelty of the paper. As mentioned by the author, there is a big overlap with the reference [12]. The main strategy, including the regularized subproblem and the proposition 1, is the same as in [12]. The only difference is to replace the Newton's method by first order methods and provide the analysis of the inner loop complexity.
Overall, I find the idea interesting but the contribution seems to be limited, therefore I vote for a weakly accept.