Pruning Random Forests for Prediction on a Budget
We propose to prune a random forest (RF) for resource-constrained prediction. We first construct a RF and then prune it to optimize expected feature cost & accuracy. We pose pruning RFs as a novel 0-1 integer program with linear constraints that encourages feature re-use. We establish total unimodularity of the constraint set to prove that the corresponding LP relaxation solves the original integer program. We then exploit connections to combinatorial optimization and develop an efficient primal-dual algorithm, scalable to large datasets. In contrast to our bottom-up approach, which benefits from good RF initialization, conventional methods are top-down acquiring features based on their utility value and is generally intractable, requiring heuristics. Empirically, our pruning algorithm outperforms existing state-of-the-art resource-constrained algorithms.

This paper proposes a method for pruning a forest of trees taking into account feature costs. Pruning is formulated as a 0-1 integer program with linear constraints and it is shown that this program can be solved exactly through LP relaxation. Experiments are carried out on several datasets.
The idea of taking into account feature costs when pruning tree ensembles is original to the best of my knowledge. The main originality of the proposed approach is the fact that it adopts a bottom-up post-pruning strategy, while most existing approaches are top-down, acting during tree growing. While the authors present this feature as an advantage of their method, actually, I'm not convinced that adopting a bottom-up strategy is a good idea for addressing this problem. Since the algorithm indeed can not modify the existing tree structure (it can only prune it), it should be less efficient in terms of feature cost reduction than top-down methods that can have a direct impact on the features selected at tree nodes. For example, let us assume that two very important features in the dataset carry on the exact same information about the output (i.e, they are redundant). They can both be included in a forest if this latter is built without constraint and BudgetPrune will have no choice but to keep these two variables in the model, even though only one of them is actually necessary.    The optimization problem is nicely formulated, in particular in its coupling of the feature usage variables. The proposed primal-dual optimization problem seems sound and very efficient as it scales linearly with respect to the number of trees (compared to the initial cubic LP relaxation). In the formulation of the (IP) problem, I'm surprised by the fact that the error term in the objective function is the average of the individual errors of the ensemble trees and not the error of the ensemble itself. I guess this is the main reason why the optimization problem can be solved efficiently by the primal-dual algorithm but I think that the authors should discuss and motivate this choice better in the paper. Another  not so well-motivated design choice is the incorporation of the computational cost (average leaf depths) in the objective function. Given that the experiments only focus on assessing the average feature cost, is it necessary to include the computational cost? Doesn't it harm the average feature cost? Don't you need also to introduce an additional coefficient in the objective function to balance the feature acquisition costs and computational cost? Merely summing them seems risky without a proper normalization.    Experiments seem to have been well conducted. They are extensive enough, although the number of datasets (4) could have been higher. They rather convincingly show the superiority of BudgetPrune over competitors (in Figure 2). Experiments in the supplementary material are also very useful to complement the experiments in the main paper.     Some points however need clarification before the results can really be assessed: - The authors propose to use in-bag data to estimate error probability at each node and the validation data for the feature cost variables. It's not clear to me what it means. What is the in-bag data? Is it the learning sample itself or the bootstrap sample of each tree? - It's not clear if all methods make use of the validation data. For the methods that do not need a validation set, it would be more fair to train them on the whole training+validation data. - In the CCP method, what do the authors mean by the sentence "we perform CCP on individual trees to different levels to obtain various points on the accuracy-cost tradeoff curve"? This should be more precisely explained. - The GreedyPrune method is also unclear. What does it mean to remove all nodes corresponding to one feature? Do you prune the whole subtree below each split on this feature? If not, what do you do with the left-right successor nodes of this split? What does it mean to select the pruned RF that "has the lowest training error and average feature cost"? Do you sum these two scores? With a weight? - In the plots in Figure 2,how is the average feature cost measured on the x-axis? Also, how are different tradeoffs obtained for the different methods in Figure 2? With BudgetPrune, I guess that the parameter lambda is changed to obtain a curve but how is it done for the other methods? Is there only one parameter that can change feature cost? - All methods are evaluated only in terms of average feature cost and accuracy while the objective function also includes a term related to the computational costs (the average depth of the leaves reached by the examples). Why are the approaches not evaluated also according to this criterion? Is this term really crucial in the objective function?   The paper is well-written and mostly clear (except for the missing details in the experiments). However, the fact that several experimental details are only present in the supplementary material makes reading section 5 not so easy. I think that some more details could have been given in the main paper.    Minor comments: - There is no reference in the main text to the fact that the supplementary material contain additional (and necessary) details about the problems and settings of algorithms. A link should be given at the beginning of Section 5. - Page 1, I don't understand the meaning of footnote 2. - In the case of uniform cost, alternative competitors to BudgetPrune are feature selection methods that tries to find a small subset of features that are sufficient to predict the output. It would interesting to include feature selection methods in the comparison (for example using Random forests variable importances). - In Section 5.1: "This is a fundamentally combinatorial problem...and we are able to globally optimize cost-accuracy objective". "our proposed formulation possesses 1) elegant theoretical properties".  The BudgetPrune algorithm indeed finds a global optimum but it needs to start from an existing random forests whose training is also a combinatorial problem solved in practice by using relaxations and heuristic with no guarantees of performance in the end. So, I'm not sure it's fair to say that BudgetPrune provides a better solution with stronger theoretical guarantees to the problem than top-down methods.


The Limits of Learning with Missing Data
We study linear regression and classification in a setting where the learning algorithm is allowed to access only a limited number of attributes per example, known as the limited attribute observation model. In this well-studied model, we provide the first lower bounds giving a limit on the precision attainable by any algorithm for several variants of regression, notably linear regression with the absolute loss and the squared loss, as well as for classification with the hinge loss. We complement these lower bounds with a general purpose algorithm that gives an upper bound on the achievable precision limit in the setting of learning with missing data.

The manuscript provides lower bounds for regression (and classification) in the limited attributes observation model. In particular, the lower bounds tighten what is previously known for the squared loss in this setting, and novel lower bounds for absolute loss and hinge loss (in classification) are shown. An algorithm that achieves loss bound up to a certain precision limit is also provided. The results are sufficiently novel and the paper is well-written.
The authors provide lower bounds for regression and classification in the limited attributes observation model. Here, the learning algorithm has access to a subset of attributes, and we want to control the excess loss (over the bayes optimal). The manuscript provides lower bounds in this setting that improve over known bounds, and resolve some open questions in this setting.   The manuscript makes interesting contributions: the provided lower bounds tighten what is previously known for the squared loss in this setting, and for absolute loss and hinge loss (in classification), novel lower bounds are derived. Furthermore, an algorithm that achieves loss bound up to a certain precision limit is also provided; this matches lower-bound up-to a polynomial factor in dimensionality (in case of regression). The results and proof techniques are sufficiently novel and the paper is well-written. I recommend acceptance.  The results in classification setting, albeit, appear less interesting (there is an exponential gap between the lower bound and the guaranteed precision limits of the proposed algorithm for hinge loss). The authors do not comment on Theorem 5 after it is briefly stated in Section 2.1. In particular, the nature of the result in Theorem 5 is different from that in Theorems 1 and 3 (for regression), and seem weaker (existence of some epsilon vs actually quantifying the limits of precision). I would like the authors to comment on this aspect, and perhaps an exposition in the main paper as to the difficulty of stating a stronger result (if that is the case) would be useful.    (I've read the author response.)


Estimating the Size of a Large Network and its Communities from a Random Sample
Most real-world networks are too large to be measured or studied directly and there is substantial interest in estimating global network properties from smaller sub-samples. One of the most important global properties is the number of vertices/nodes in the network. Estimating the number of vertices in a large network is a major challenge in computer science, epidemiology, demography, and intelligence analysis. In this paper we consider a population random graph G = (V, E) from the stochastic block model (SBM) with K communities/blocks. A sample is obtained by randomly choosing a subset W ⊆ V and letting G(W ) be the induced subgraph in G of the vertices in W . In addition to G(W ), we observe the total degree of each sampled vertex and its block membership. Given this partial information, we propose an efficient PopULation Size Estimation algorithm, called PULSE, that accurately estimates the size of the whole population as well as the size of each community. To support our theoretical analysis, we perform an exhaustive set of experiments to study the effects of sample size, K, and SBM model parameters on the accuracy of the estimates. The experimental results also demonstrate that PULSE significantly outperforms a widely-used method called the network scale-up estimator in a wide variety of scenarios.

The article "Estimating the Size of a Large Network and its Communities from a Random Sample" considers the problem of estimating the total number of vertices in a network given the subsample of the network and full degrees of the vertices in the subsample. The authors propose the algorithm which uses the stochastic block model (SBM) approximation of the network to estimate the number of vertices in each block and subsequently in the full network. The properties of the algorithm are demonstrated on a series of examples on model data.
The main idea of the article is to improve the estimation of the size of a network based on considering the SBM model instead of Erdos-Renyi model, which is considered in the baseline algorithm. This is a natural approach, which however leads to significant complications as a resultive likelihood is costly to evaluate due to massive summation. The authors overcome the problem by considering the joint posterior and sampling from this posterior via Gibbs sampling.  The general approach seems fair, however the computational limitations of the algorithm are not considered and it is not evident whether the algorithm achieves the goal of mimicking the respective distribution. I should note, that the theorems provided are stated as one of the main results of work, but in fact don't add significant value to the algorithm and the results. The first theorem states finite sample bias in the basic estimate, however finite sample unbiasness of the proposed algorithm is not showed. The second theorem states the tractability of obtained posterior, however the properties of the obtained algorithm itself are not theoretically accessed.  The experimental part seems to be proper comparison of two algorithms on different variants of model data, however no experiments on real data are considered. Real data is very important as any model for this data is an approximation, which for the SBM case will mean the necessity of choosing some number of blocks K. It is of interest to consider the work of the algorithm on the real data, especially considering that it is quite easy to perform the experiment on any type of real network just by sampling a part of it.  To sum up, I think that the paper presents a good effort in the interesting research direction, however theoretical results seem to be not supportive for the algorithm (while are underlined as important) and experimental part needs significant improvement.


Linear Relaxations for Finding Diverse Elements in Metric Spaces
Choosing a diverse subset of a large collection of points in a metric space is a fundamental problem, with applications in feature selection, recommender systems, web search, data summarization, etc. Various notions of diversity have been proposed, tailored to different applications. The general algorithmic goal is to find a subset of points that maximize diversity, while obeying a cardinality (or more generally, matroid) constraint. The goal of this paper is to develop a novel linear programming (LP) framework that allows us to design approximation algorithms for such problems. We study an objective known as sum-min diversity, which is known to be effective in many applications, and give the first constant factor approximation algorithm. Our LP framework allows us to easily incorporate additional constraints, as well as secondary objectives. We also prove a hardness result for two natural diversity objectives, under the so-called planted clique assumption. Finally, we study the empirical performance of our algorithm on several standard datasets. We first study the approximation quality of the algorithm by comparing with the LP objective. Then, we compare the quality of the solutions produced by our method with other popular diversity maximization algorithms.

The paper considers the problem of selecting M diverse elements of a finite set, given that those are elements of a metric space and therefore a distance between elements is defined. More precisely, the work focuses on the so called 'sum-min diversity', which has attractive practical properties. The problem of finding the M-tuple of diverse points with respect to such measure is known to be NP-hard and the best known algorithm provides an approximation factor of O(1/log n), where n is the size of the set. Authors propose a linear programing approximation of the problem and a probabilistic rounding procedure to obtain a binary vector defining the diverse M-tuple. The expected output of this probabilistic algorithm provides a constant factor approximate solution for the problem.
Although the provided novel algorithm looks impressive both from the theoretical prospective and in the experimental comparison, its substantiation has quite some room for improvement.  The major point is the proof of Theorem 1:  - it is unclear how the proof of the theorem follows from Lemmas 3 and 4, since none of these lemmas is related to the optimal solution of the considered diversity problem. I assume that the missing proposition is the one, which would establish connection between the considered linear program in lines 153-154 (by the way, it is very uncomfortable that the main formulation is not numbered and therefore can not be easily referenced) and the diversity problem. I believe that this connection may have the following format: if the linear program is equipped with integrality constraints (which is, all variables x_{ir}\in {0,1}), the resulting ILP is equivalent to the considered diversity problem. Indeed, the proof of such a proposition is not obvious for me as well.  Without clarification of this point I would assume a major flaw in the proof and, therefore, absence of a theoretical contribution.  == post-rebuttal update==  The authors addressed my main issue - the proof of Theorem 1. Although I agree to R7 about experimental evaluation, I believe the paper can be accepted provided that the mentioned experimental comparison is included into the paper.


Linear Feature Encoding for Reinforcement Learning
Feature construction is of vital importance in reinforcement learning, as the quality of a value function or policy is largely determined by the corresponding features. The recent successes of deep reinforcement learning (RL) only increase the importance of understanding feature construction. Typical deep RL approaches use a linear output layer, which means that deep RL can be interpreted as a feature construction/encoding network followed by linear value function approximation. This paper develops and evaluates a theory of linear feature encoding. We extend theoretical results on feature quality for linear value function approximation from the uncontrolled case to the controlled case. We then develop a supervised linear feature encoding method that is motivated by insights from linear value function approximation theory, as well as empirical successes from deep RL. The resulting encoder is a surprisingly effective method for linear value function approximation using raw images as inputs.

This paper addresses the problem of designing and learning models for planning in reinforcement learning. More specifically, it proposes a linear model comprising of an "encoder", mapping the observation space into a latent space, and a "decoder", mapping the latent space to the predicted next state and reward. The encoder/decoder model draws inspiration from autoencoders (linear) but couples transition dynamics and reward under the same model. The authors use Parr's (2007) Bellman error decomposition to analyze the their model. A factorization approach is chosen for learning such models using the SVD. Experiments finally show that this model can be used in high dimensional settings when learning directly from pixels.
Summary:  The idea of coupling reward and dynamics in an autoencoder-like model is a novel contribution which could benefit our community. I also appreciate that the authors have applied their model on pixel-based observation spaces. However, I find the theory of lines 124 to 136 unnecessary and the fact that it reproduces Parr (2007) line by line is problematic (more on this below). Also, example 1 seems misguided since it simply does adopt the right problem formulation to start with (it seems sufficient to simply start with a Markov chain over state-action pairs).  Detailed comments:  Abstract: l. 4. and sect. 1 l.25: "Typical deep RL [...]" and "It is common"  Is that true ? Beside DQN, what are other examples ? Is it true for TD-Gammon ?  section 1, l. 21: "researchers at Deepmind". I would use a different form. Simply citing the DQN paper would probably be enough.  sect 1. l. 31: The qualifier "novel" seems unnecessary.  sect 1. l. 41: "Aside from the negative one". We don't know yet at this point what this mean. Perhaps you can reference the section in which the negative one is presented.  section 1. l.42 : "Persistent hidden state". This is not clear what you mean here. Perhaps you meant to compare your model-based approach with more of an RNN or PSR-type model. If this is the case, you would have to argue that the proposed model is somehow able to learn a representation which makes the observations as "Markov" as possible, that you can "make state" without state/belief state updating.  section 1. l. 47: "Our results differ from". Please cite the work implied in this sentence. Perhaps Parr (2007), Sutton (2008) ?  section 2. l. 56: This is not any MDP, it's a *Discounted* MDP. Why do you need to define the reward function over $(n \times m) \times 1$ ?  section 2. l. 62: "The policy $\pi$": This is a particular kind of policy, namely a "stochastic" or "randomized" policy.  section 2. l. 65: Sum: "Expected total *sum* of $\gamma$-discounted"  section 2.1 l. 71: "Otherwise, we have [...]. This sentence lacks context. It should probably be mentioned that the Bellman operator is a contraction and that you solve for the fixed point through value iteration.  section 2.3 : "has typically": please cite relevant literature. "Feature selection wrapper": "wrapper" is more of a term that you would use in software engineering. The meaning is not clear in this context.  section 2.3 l. 86: Please explain why it did not scale well.  section 2.3 l. 88: "connect directly to the optimization problem". This isn't clear. Knowing this paper, I think that you would want to say is that PVFs only consider the transition dynamics and not the reward structure.  section 2.3 l. 91: "classical approaches". What are they ? Please cite relevant work.  section 2.3 l. 98: it could be relevant to also mention "Value Pursuit Iteration" by Farahmand and Precup.  section 3. l. 109: "Anecdotally" generally connotes with "amusing", which is not the intended meaning here.  section 3. l. 106-108: I find this sentence confusing because it mixes concepts of "action-conditional" linear models (which is more about the "evaluation problem") with the "control problem" (policy/value iteration). One does not imply the other.  section 3. l. 109. "The following example [...]". I find this sentence confusing and it sounds in a way that contradics Parr (2007). Also, you need to define more formally what you mean by "linear action models" earlier in the section.  Lines 124 to 136 (and section 3.0 as a whole) seem unnecessary. The derivations follow exactly line by line (and using the same notation) that of Parr (2007). It would suffice to say that in order to represent Q-values, we need to consider the augmented Markov chain over state-action pairs. You define the state $\tilde{s}_t = (s_t, a_t)$ and then simply use Parr's results over this chain. The same argument applies when you go from TD(0) to SARSA(0) : see Sutton & Barto for example https://webdocs.cs.ualberta.ca/~sutton/book/ebook/node64.html  section 3. following definition 5: A novel aspect of the proposed approach is that it couples reward and transition dynamics under the same predictor. I think that it's an important design choice for which it would be worth spending more time explaining. You could for example contrasts this approach with one based exclusively on the transitions dynamics (Mahadevan's PVFs for example).  section 3.1, l. 170: Theorem numbering jumps from 3 to 7.  section 4, l. 208: "termination conditions are typical API [...]", please briefly explain why they still hold.  section 4.1 : Please explain why you chose coordinate descent. Also, it would be useful to study the effect of the estimation error in $\tilde{A}$ on the quality of the learned features. Is it stable ?  Lines 220-223 is useful to know but it is also more difficult to clearly see what is the main algorithm proposed in this section. It could perhaps be moved to the appendix.  section 5, l. 246: "Ambitious". Many examples of large scale linear RL can be found in the literature on the Arcade Learning Environment (prior to DQN). For example, see "State of the Art Control of Atari Games Using Shallow Reinforcement Learning" by Liang, Machado & al. for recent work.  section 5, l. 249: I would want to see this claim back up in your experiments: "far greater computational resources". The iterative SVD approach of algorithm 2. with O(n^3) per iteration is non-negligible. It is also not clear that it could benefit from the same computational speedups as GPU-based deep RL methods.  section 5, l. 257: half an hour with or without the exploration phase ?  section 5.1, l. 277: I find this experiment very interesting. However, if the simulator had to be modified, it suggests that the proposed approach cannot cope with the non-Markovianity by building appropriate features.  Have you tried concatenating a number of successive frames (4) as in DQN ?  section 5.1, l. 286: how did you choose $k$ ?  section 5.1: It would be important to remind the reader that you use the planning approach of algorithm 1 in this experiment. You could have also chosen to learn the model using algorithm 2 and plan using any other approach.  section 5.2, l. 321: Why did you have to define this metric rather than reporting the actual return ?  section 5.2, l. 347: What was this threshold ?  How did you choose it ?


A Minimax Approach to Supervised Learning
Given a task of predicting Y from X, a loss function L, and a set of probability distributions Γ on (X, Y ), what is the optimal decision rule minimizing the worstcase expected loss over Γ? In this paper, we address this question by introducing a generalization of the maximum entropy principle. Applying this principle to sets of distributions with marginal on X constrained to be the empirical marginal, we provide a minimax interpretation of the maximum likelihood problem over generalized linear models as well as some popular regularization schemes. For quadratic and logarithmic loss functions we revisit well-known linear and logistic regression models. Moreover, for the 0-1 loss we derive a classifier which we call the minimax SVM. The minimax SVM minimizes the worst-case expected 0-1 loss over the proposed Γ by solving a tractable optimization problem. We perform several numerical experiments to show the power of the minimax SVM in outperforming the SVM.

This paper develops a notion of maximum conditional entropy for arriving at robust Bayes decision rules, where the entropies are generalizations of the usual Shannon entropy, beyond the case of log loss. The formulation is similar to that of Grunwald and Dawid but goes farther (and becomes more relevant for standard learning problems) by allowing for input and label random variables and hence conditional distributions. The first main result shows that (under regularity conditions) the Bayes rule for the distribution that maximizes conditional entropy is also a robust Bayes decision rule (against a set of distributions $\Gamma$). The authors then show an interesting duality result that re-expresses the problem of maximizing conditional entropy as a certain regularized minimization problem that, for exponential families and log loss, corresponds to regularized maximum likelihood. They also connect the suggested procedure to an excess risk bound, showing that with high probability, using their procedure for an empirical distribution guarantees low excess risk versus the hypothesis one obtains when using their procedure (which is still regularized) with oracle access to the true distribution; this result apparently is only for log loss, as per the leadup to Theorem 3. Finally, they show important derivations of key quantities for the method in the case of logistic, 0-1, and quadratic loss, corresponding previously known learning methods or new ones (in the case of minimax SVM). Initial experiments are promising.
\documentclass{article}  \begin{document}    This work represents a very interesting and original generalization of the work of Gr\"unwald and Dawid to the case of learning with inputs $x$. The technical results appear to be correct and the experimental results (which I think are quite preliminary) suggest the minimax SVM might be a good idea. I think the idea of robust Bayes decision rules makes sense and the authors show how under squared loss a connection to the Huber loss emerges. My main comment is that the paper itself is a somewhat difficult read due to terseness at key places, which might limit the impact of the paper. So, the rest of my comments are just geared towards improving the clarity of the paper.  Technically, in every instance where the authors apply Danskin's theorem, it was not really clear what form of Danskin's theorem was being used, and therefore it was difficult to follow the derivation. For instance, the proof of Theorem 2 (in the appendix) is clear up until the end ``Furthermore, applying Danskin's theorem to (9) implies...''. You should provide a citation for the form of Danskin's theorem being used, or just reproduce the theorem in the appendix. Using the form that I found, I could not reproduce (12); I think this comes down to the authors skipping quite a few steps. Also, in equation (24) of the appendix, I could not see how Danskin's theorem give (24), but this might be clarified once you mention the form of Daskins's theorem being used.  I found it difficult to follow the argument from lines 187-194 of the main text, including Corollary 1. It would be useful to explain this better, especially given that you can fit another paragraph in the paper and still stay within 8 pages. In particular, why does the uniform conditional distribution appear in equation (18)? Perhaps you can explain the ``it can be seen that'' with a few sentences. I think the geometric understanding provided is one of the highlights of the paper, so it is unfortunate if the reader cannot fully grasp what you have in mind.   Minor comments:  In equation (10) in the appendix, you are missing a minus sign in front of $\mathbf{A}_i$ both in the middle expression and the rightmost expression. This minus sign does appear in the first line of equation (11), so the mistake apparently was just a typo.  On line 26 of the appendix, you say that ``problem (5) is convex''. But the objective is concave being the entropy (a minimum of a linear function) and indicator are both concave.    \end{document}


Fast and Flexible Monotonic Functions with Ensembles of Lattices
For many machine learning problems, there are some inputs that are known to be positively (or negatively) related to the output, and in such cases training the model to respect that monotonic relationship can provide regularization, and makes the model more interpretable. However, flexible monotonic functions are computationally challenging to learn beyond a few features. We break through this barrier by learning ensembles of monotonic calibrated interpolated look-up tables (lattices). A key contribution is an automated algorithm for selecting feature subsets for the ensemble base models. We demonstrate that compared to random forests, these ensembles produce similar or better accuracy, while providing guaranteed monotonicity consistent with prior knowledge, smaller model size and faster evaluation.

The paper deals with learning monotonic functions. This can be useful for the data, for which there are some inputs that are known to be positively (or negatively) related to the output, and in such cases training the model to respect that monotonic relationship can provide regularization, and makes the model more interpretable. The main problem which is dealt in the paper is that flexible monotonic functions are computationally challenging to learn beyond a few features. The paper proposes to learn ensembles of monotonic calibrated interpolated look-up tables (lattices). A key contribution is an automated algorithm for selecting feature subsets for the ensemble base models. The algorithm is experimentally compared against random forests, and is shown to produce similar or better accuracy, while providing guaranteed monotonicity consistent with prior knowledge, smaller model size and faster evaluation.
The paper is heavily based on the previous work on lattice (and monotone lattice) regression, and is thus not self-contained. It is not even explained in the main part of the paper what is the actual function which a lattice represents (two examples are found in the supplementary materials, but I still found the explanation two brief). I think this should definitely be a part of the main paper, otherwise the reader is forced to check the previous work on this topic to understand what kind model is actually being considered.   The paper barely touches upon the computational complexity issues of optimizing (3), (7), and unnamed eq. in Step 3. While (7) is convex given calibrators being fixed, it is not clear whether it is convex when jointly optimized over calibration and lattice parameters (I doubt it is). Moreover, (3) seems highly complex and combinatorial optimization criterion. The optimization complexity of eq. is Step 3 is unclear to me (note that Step 3 requires knowing ensemble weights \alpha, which are only computed in Step 5; how is this issue solved?). The authors should clearly state which problems are easy to optimize, and which are only solved heuristically, likely returning suboptimal solution.  Given a complex training of lattice ensemble I missed the computational times of training. Since the authors report faster evaluation of their method over random forests at the testing phase, they should also report training time comparing to the competitors.  The experimental results are not very convincing: only around 1% of accuracy improvement is reported comparing to the random forests; and it is not clear if random forest is a good fit for these data sets. It also does not help that all data sets except the first one are proprietary (hard to find other experimental studies on these data sets). It would be beneficial to check at least one more state-of-the-art flexible regression method, such as, e.g., gradient boosting, on these data sets (the unconstrained lattice, as another competitor, is not enough, as it is not particularly known algorithm). For instance, xgboost (an implementation of gradient boosting) easily achieves over 86% accuracy on Adult, see https://jessesw.com/XG-Boost/. RuleFit algorithm of Friedman & Popescu (http://statweb.stanford.edu/~jhf/ftp/RuleFit.pdf) gives 13.6% test error (86.4% accuracy), etc. Apparently, they all beat both random forests and monotone lattices Adding more methods to the comparison would help to resolve the question, whether exploiting monotonicity actually gives an advantage in predictive performance.  Please explain what is novel in the paper comparing to Ref. [2]. Both papers share monotonic lattice regression, feature calibration and some of the optimization criteria (e.g., for joint optimization of calibration and lattice parameters). It looks as if the novel part is the ensemble learning of lattices, together with crystals algorithm to select features for lattice construction. And the parts related to incorporating monotonicity are not.


Batched Gaussian Process Bandit Optimization via Determinantal Point Processes
Gaussian Process bandit optimization has emerged as a powerful tool for optimizing noisy black box functions. One example in machine learning is hyper-parameter optimization where each evaluation of the target function may require training a model which may involve days or even weeks of computation. Most methods for this so-called "Bayesian optimization" only allow sequential exploration of the parameter space. However, it is often desirable to propose batches or sets of parameter values to explore simultaneously, especially when there are large parallel processing facilities at our disposal. Batch methods require modeling the interaction between the different evaluations in the batch, which can be expensive in complex scenarios. In this paper, we propose a new approach for parallelizing Bayesian optimization by modeling the diversity of a batch via Determinantal point processes (DPPs) whose kernels are learned automatically. This allows us to generalize a previous result as well as prove better regret bounds based on DPP sampling. Our experiments on a variety of synthetic and real-world robotics and hyper-parameter optimization tasks indicate that our DPP-based methods, especially those based on DPP sampling, outperform state-of-the-art methods.

The problem of  optimizing an unknown function is considered, where at each time step, one can sample a batch of B points where to evaluate the function. The problem is considered from a Bayesian standpoint, assuming the unknown function is sampled from a Gaussian process.  Borrowing ideas from the rich literature on this topic, the paper proposes a novel algorithm, based on determinantal point processes (DPP). A DPP samples sets of indices proportionally to the determinant of a submatrix formed by this set.  The paper introduces several variants, and shows the equivalence between the actions sampled by the algorithm and other strategies. In Section 3.1, a batch version of EST is considered, and is shown to be equivalent to a batch version of UCB. In Section 3.2,   Another variant based on DPP sampling, as opposed to maximization is considered and analyzed (Theorem 3.4).  Section 4 provides illustrative numerical experiments on a number of problems with both synthetic and real-world datasets.
Overall, this is an interesting algorithmic contribution, supported by both a regret analysis and extensive numerical experiments. The writing of the paper is not very clear and does not help understanding the exact contributions.  The noise is assumed to be exactly Gaussian with known variance \sigma^2.  Although this is a standard assumption in the Bayesian world, this is also a restrictive assumption. It would be interesting to study the case when the sigma^2 is unknown, at least.  In the literature, there is another body of work related to the algorithms DOO, SOO, STOSOO, POO, etc. that is not cited here. I believe these are relevant works and should be discussed, at least briefly.  The contribution should be clarified. First, I think it is important to distinguish better between the existing algorithms from the literature, and  the ones that are introduced specifically here. Then, you should clarify the real benefit of using DPP in the end, both in terms of regret performance and numerical efficiency. It seems to me that the key result is Theorem 3.4; perhaps this can be highlighted.  Theorem 3.4 is interesting, as it shows some possible improvement one can get using DPP sampling. Do you think you can provide a few hints at the proof in the main material?  A natural side question is the computational complexity of the sampling scheme for DPP over other methods, and the effect of possible errors (when using MCMC for instances).   The numerical experiments are convincing and reported in a reproducible way.   L46: reference? L90: reference?


Measuring Neural Net Robustness with Constraints
Despite having high accuracy, neural nets have been shown to be susceptible to adversarial examples, where a small perturbation to an input can cause it to become mislabeled. We propose metrics for measuring the robustness of a neural net and devise a novel algorithm for approximating these metrics based on an encoding of robustness as a linear program. We show how our metrics can be used to evaluate the robustness of deep neural nets with experiments on the MNIST and CIFAR-10 datasets. Our algorithm generates more informative estimates of robustness metrics compared to estimates based on existing algorithms. Furthermore, we show how existing approaches to improving robustness "overfit" to adversarial examples generated using a specific algorithm. Finally, we show that our techniques can be used to additionally improve neural net robustness both according to the metrics that we propose, but also according to previously proposed metrics.

This paper introduces an algorithm to generate adversarial examples for NNs (ie. images with an imperceptible perturbation that are mislabeled by the NN). This algorithm constrains the adversarial example to be in the linear region of the activations of the ReLUs of the NN. Experiments in MNIST show that for the NN called LeNet, the adversarial examples introduced in the paper seem to be more harming than the adversarial examples introduced in previous works (in terms of L_inf norm of the perturbation).
As in [5,20], this paper proposes a new algorithm to generate adverarial examples, and it uses the adversarial examples to re-train the NN. The algorithm to generate the adversarial examples is a bit different from previous works as it adds some additional constraints into to the optimization. I have two concerns regarding the motivation of these constraints:   -Theoretical. The constraints that are added to the algorithm to generate the adversarial perturbation are motivated by: "Since adversarial examples exists because of the linearity in the neural net [5], we restrict our search to the region around the input in which the neural net is linear". Yet, adversarial examples have also been found outside these linear region around the input. A clear case of this is that when the adversarial perturbation is multiplied by a relativelly large constant, the adversarial examples still negativelly affects the NN [5]. Moreover, there are not any theoretical guarantees that by constraining the adversarial examples to be in the aforementioned linear region, the adversarial example has minimum perturbation. There could be adversarial examples with smaller norm outside the search space of the algorithm introduced in the paper.  -Experimental. The conclusions extracted from the results do not seem to be general. The NN architectures are not state-of-the-art and not standard, and are applied in very simple datasets (MNIST and CIFAR). There are no sanity checks that the baseline of [20] has been correctly reproduced. Also, results show marginal improvements in MNIST over the baseline, and in CIFAR, the comparision with the baseline has been omitted.   Other comments:  -The results in CIFAR show that this method may not have much applicability to reduce the adversarial examples, as it takes about 15 seconds to generate one adversarial example using 8 CPUs. This will make it difficult to scale to ImageNet, and create sufficient adversarial examples to re-train the NN.  -The first time that $x_\star$ is introduced, it has been never defined before.  -It has been assumed that L_inf norm is the best measure to analyze the perceptibility of the adversarial perturbation.  -It has been mentioned that "[3] seeks to explain why neural nets may generalize well despite poor robustness properties". A more detailed explanation of [3] seems necessary here. Maybe the motivation of the constraints could be done from [3].


PerforatedCNNs: Acceleration through Elimination of Redundant Convolutions
We propose a novel approach to reduce the computational cost of evaluation of convolutional neural networks, a factor that has hindered their deployment in lowpower devices such as mobile phones. Inspired by the loop perforation technique from source code optimization, we speed up the bottleneck convolutional layers by skipping their evaluation in some of the spatial positions. We propose and analyze several strategies of choosing these positions. We demonstrate that perforation can accelerate modern convolutional networks such as AlexNet and VGG-16 by a factor of 2× -4×. Additionally, we show that perforation is complementary to the recently proposed acceleration method of Zhang et al. [28].

The goal of this paper is to reduce the evaluation time and memory size during evaluation of a CNN. The paper proposes by saving time by only evaluating a subset of convolutional features at selected spatial locations, determined by a “perforation mask”. The non-evaluated spatial locations are then filled in through nearest-neighbor interpolation. The paper proposes a few methods for determining this perforation mask. The paper evaluates the performance of this technique on recent CNN architectures: Network-in-network, AlexNet, and VGG-16.
This paper proposes a method to speed up the evaluation of CNNs. The method skips the evaluation of a subset of spatial locations in the image, and interpolates their values using nearest neighbor values. Methods for accelerating CNNs are of great interest to the community at this time. The approach of the method is well-described. I did have a few concerns with some of the results and parts of the method.  (1) Figure 4(c)(d) shows the increase in error vs speedup factor for several methods. The impact line resulting in greater errors than all of the other methods in the conv3 layer is a surprising result, and some additional analysis would shed more light on the result to the reader.  It may also indicate that the impact is not being computed with the correct metric. A feature which is not computed will be interpolated, rather than zeroed out, before evaluation in the next layer. Why then, is “impact” measured by estimating the effect of zeroing out the feature (feature magnitude times gradient), rather than just the effect of perturbing the feature (simply the gradient)?  (2) The baselines all corrupt the CNN function. The “resize” operation changes the input scale of the image. The “stride” operation reduces the feature map resolution at the layer, so the subsequent layer is effectively performing an a trous/dilated convolution on the feature map at the original resolution, which means that it is evaluating a different function. I believe the true baseline is simply the perforated-CNN with zero-filling instead of interpolation.   (3) Though there is no pooling layer directly after conv3 in AlexNet, the “pooling structure” method for perforation mask selection could reasonably be extended to conv4 and conv3, since layers conv3-conv5 share the same spatial locations.  (4) For VGG, “pooling structure” will have a tie in all spatial locations. Since kernel&stride are equal in the network, no features are in overlapping regions. This should be mentioned, either in the page 3 line 123 or page 7 line 225.  (5) The barycentric interpolation is likely to be of interest to the reader. I would recommend moving this material from the appendix into the paper, with an analysis of its classification performance, memory, and evaluation time.


Linear dynamical neural population models through nonlinear embeddings
A body of recent work in modeling neural activity focuses on recovering lowdimensional latent features that capture the statistical structure of large-scale neural populations. Most such approaches have focused on linear generative models, where inference is computationally tractable. Here, we propose fLDS, a general class of nonlinear generative models that permits the firing rate of each neuron to vary as an arbitrary smooth function of a latent, linear dynamical state. This extra flexibility allows the model to capture a richer set of neural variability than a purely linear model, but retains an easily visualizable low-dimensional latent space. To fit this class of non-conjugate models we propose a variational inference scheme, along with a novel approximate posterior capable of capturing rich temporal correlations across time. We show that our techniques permit inference in a wide class of generative models.We also show in application to two neural datasets that, compared to state-of-the-art neural population models, fLDS captures a much larger proportion of neural variability with a small number of latent dimensions, providing superior predictive performance and interpretability.

proposed novel nonlinear population model and corresponding inference algorithm.
the idea/paper is great, i'm a huge fan. i was, however, confused about a few things. below, i have some general comments, followed by more specifics.   GENERAL  1. accuracy vs. sample size: in statistics, "efficiency" is key (accuracy as a function of sample size).  i want to understand the efficiency for this method for both simulations as well as the real data. ie, please plot error vs sample size for all the methods in each scenario.  2. dimensionality vs. kernel: it is well know that in many cases complexity can be "pushed around", either by keeping more dimensions, or by adding a nonlinear kernel.  i want to understand better how many dimensions the linear model needs to get the accuracy of the nonlinear one.  ie, please plot error vs latent dimensionality for both simulations and real data using both your nonlinear stuff and the linear (inhomogeneous) stuff.  3. baselines: a great feature of simulations is that one can compute bayes optimal performance, to provide an "upper bound".  we can also compute naive performance (eg, a homogeneous poisson model), which provide a null "lower bound". in the real data example, you plotted lower bound.   in both simulation settings, for all experiments, please provide lower and upper bounds.  4. notation: i couldn't quite follow it.   SPECIFICS  1. notation: i found it obtuse, perhaps there is no better way.  but, for example, why is P a function of \ell instead of only \lambda, which is itself a function of \ell?   also, why are \ell and f() separated?  it is because f can be a function of each neuron, and \ell is not?  and if \lambda = \ell, i'm not sure why \lambda exists too.  it was also a bit unclear what role "i" plays: "the i^th element of a deterministic rate function"? but also i \in {1,...,n}.  so i assumed i indexed neuron, so what does it mean to say it is the i^th element of a rate function? it indexed the square bracket, which i guess means that it indexed \ell?  so, i think i would have written:   x_rti | z_rt ~ P( f_i(z_rt))  in any case, please clarify your choices or greatly simplify notation. if the obtuseness is useful for connecting to other stuff,  maybe introduce it as simple as possible first, and then connect it to the other ones?  2. i also didn't understand  "we represent f_\psi through a parametric neural network model" what is a "nonparametric" neural network model, such that you are distinguishing? maybe the word "parametric" in not necessary? please remove or clarify  3.  \theta = (...).  i was a bit confused which of those parameters index a neuron, and which index the whole population.   the dimensionality of \theta was also a bit unclear to me, i realize it is a function of m & n (and the neural network).  but what happened to say, c & d in the PLDS model? are they "eaten" by \psi, meaning you don't even represent them? please clarify which parameters are neuron specific, and the dimensionality of \theta (possibly different dimensionalities for the different special cases).  4.  i believe that PfLDS is a generalization of fLDS, insofar as PfLDS has a nonlinear function of z, and fLDS has a linear one. in other words, the rates are:  exp[ c*z+d ] for fLDS, and exp[ f_\psi(x) ] for PfLDS.  is that correct? and the same kind of thing for GCfLDS? please clarify the text around lines 104-107.   5. "we append the noise model to the acroynm"  i think you mean "prepend"? "append" means put on the end. and acronym is spelled wrong.  6. table 1: i was a bit confused about a few things here.  A) are these numbers good? they are simulations, so you could use the true parameters and estimate the upper bound,  and use some null/dumb model (like constant rate) to provide a lower bound.  i just have no intuition for what PLL or time should be in any of these cases.  B) assuming my comment #4 above is correct, then the fact that PfLDS does nearly identical to fLDS and the same is true for GCfLDS, suggests that f_\psi(.) might have learned a *linear* function (ie, gotten it correct). you don't show it, though you do show something similar for the next simulations.  please show the estimated rate functions (maybe in appendix if you don't have room).  C) you mention the number of trials, neurons, and time bins, as well as the architecture of f_\psi, but not the firing rates or counts.  the estimation error is closely linked to spike count, ie, if your network only fired very sparsely, all estimates would be poor, and your nonlinear estimator would be even worse than the linear ones in many cases.  so, at a minimum, please clarify some summary statistics for the number of spikes per time (maybe per neuron, or something like that).   much better: please conduct an experiment where you compute accuracy as a function of # of spikes in training trials (or some other appropriate measure).  i would expect that the linear estimators outperform the nonlinear ones until spike count is sufficiently high, but i have no intuition whatsoever in terms of how high it must be, in particular under various model misspecification scenarios that you have already described.   the same experiment could be applied to the grid cell simulation.  7. "A paired t-test for PLL is significant (p < 106)."  a t-test?  is PLL really gaussian? i would have thought a wilcoxon test would be much more meaningful.  8. a major claim of the abstract and paper is: "We also show in application to two neural datasets that, compared to state-of-the-art neural population models, fLDS captures a much larger proportion of neural variability with a small number of latent dimensions"  the results you provide indicate that for sufficiently large sample size, upon conditioning on latent dimensionality, your method wins. this is not particularly surprising, because yours is explicitly a nonlinear generalization of the others, so given enough data, it should.  but, this improvement comes at a cost (2 costs). a) it takes more time to fit b) the understanding of each neuron now requires a nonlinear function, rather than a linear one.  a natural question i have therefore is: how does performance change with the number of latent dimensions.  in other words, above i asked for experiments where you conditioned on latent dimension, and change sample size.  i'd also like to see experiments where you condition on sample size, and change the number of latent dimensions.  for example, if adding a single latent dimension to the linear models recovers the additional variance that you obtain from your nonlinear one,  it is not obvious which approach would be preferred.  i suspect that this will not be the case, nonetheless, experimental demonstration would be nice.  9. "cleaner" and "more structured" are words you used to compare the estimated latent states.  these are a bit informal.  i wonder if you can quantify those concepts in a meaningful way?  10. "negative predictive log likelihood" why did you switch from PLL to NLL?  seems like staying with one or the other throughout would be advantageous.  11. "We find 229 that the highly nonlinear generative models used by PfLDS lead to lower-dimensional and hence 230 more interpretable latent-variable representations."  is 1D + nonlinear more interpretable than 2D+linear?   i think this is debatable.  in particular, the highly nonlinear generative models estimated by PfLDS have a highly nonlinear estimate amid the representation, which may or may not help interpretability.  the latent representations are perhaps more interpretable.  this is related to my previous comment.  12. Figure 2: why do (d) and (e) compare to a *homogeneous* Poisson process, rather than PLDS?  i think adding PLDS to this figure would satisfy my desire for understanding the tradeoff between dimensions and nonlinearity in real data (though not in simulation).  13. how should one choose the dimensionality (and neural network architecture) when using your approach? presumably, both depend on sample size.  some guidance would be appreciated.  - jv


Incremental Boosting Convolutional Neural Network for Facial Action Unit Recognition
Recognizing facial action units (AUs) from spontaneous facial expressions is still a challenging problem. Most recently, CNNs have shown promise on facial AU recognition. However, the learned CNNs are often overfitted and do not generalize well to unseen subjects due to limited AU-coded training images. We proposed a novel Incremental Boosting CNN (IB-CNN) to integrate boosting into the CNN via an incremental boosting layer that selects discriminative neurons from the lower layer and is incrementally updated on successive mini-batches. In addition, a novel loss function that accounts for errors from both the incremental boosted classifier and individual weak classifiers was proposed to fine-tune the IB-CNN. Experimental results on four benchmark AU databases have demonstrated that the IB-CNN yields significant improvement over the traditional CNN and the boosting CNN without incremental learning, as well as outperforming the state-of-the-art CNN-based methods in AU recognition. The improvement is more impressive for the AUs that have the lowest frequencies in the databases.

The paper presents and validates a new method to improve classification of facial action units (AUs) from single frames. The challenges and applications of this problem are discussed. The presented method is based on a traditional convolutional neural network (CNN) with a boosting algorithm on top of fully connected features (B-CNN). The authors suggest an incremental B-CNN (IB-CNN), in which the weights of boosting classifiers are trained in a soft way, better accumulating information from all batches. The theoretical formulations and description of the method are presented. In the experimental part, the classification results for two quite popular datasets are presented, compared with baseline methods (CNN and B-CNN) and with previous works, improvements are noticeable. Additional experiments analyzing properties of the developed method are also presented.
Some answers in the rebuttal appeared useful for me, so I can keep the same scores.  “Technical quality” The overall quality of experiments is good. - Ideally, it would be also great to present results on the CK+ database to compare this submission with [21] (which is very related) and to better compare it to [28]. - Line 249: “Data analysis of the parameter η:” The interpretation and analysis of the result in this experiment is missing. Maybe it is not necessary to present results of this experiment? - It would be interesting to see how many active neurons (i.e., with α_j > 0) were in each case in the experiment corresponding to Fig. 5. Is the proposed IB-CNN more robust to the number of input neurons because the number of active neurons is always the same after training? - For some AUs the results are worse than using a CNN or a B-CNN on both datasets? Do the authors have any explanation? - Also, how the authors chose β (beta) in (2) and (8)?  “Novelty/originality” The core idea of incrementally updating weights of classifiers (Eq. (5)) seems to be novel. However, little is said about the work on “Facial Expression Recognition via a Boosted Deep Belief Network” (ref [21] in the submission), which seems to more related than other works discussed. For instance, definition of h(x_ij , λ_j) in (1) of the submission is analogous to (7) in [21], whereas (2) is analogous to (8) in [21], and (4) is very similar to (9) in [21]. So, [21] should be better credited.  “Potential impact or usefulness” The paper presents results for facial action classification. Experimentally, the proposed method is shown to be useful for this task only. But the method is quite general and can be potentially useful in any task where the number of (positive) training samples is small, i.e. can be applied to other (image, video) datasets. As the authors say in lines 288-289: “The IB-CNN is a general machine learning method and can be adopted by other learning tasks, especially those with limited and insufficient training data”. Ideally, this conclusion should be supported in the paper by testing their method on other databases, not necessary facial expressions, to extend their impact. For instance, most of video datasets (KTH, HMDB51) with human actions have a relatively small number of samples per class. To sum up, the paper could have a greater impact (i.e., oral level) if it were more general (from the experimental perspective), something like “Incremental Boosting Convolutional Neural Network for Limited Training Data”. However, admittedly the methods particularly suited for facial action recognition can be indeed very useful in practice.  “Clarity and presentation” Overall, the paper is well written, except for the following issues which can be fixed in a short time. - There must be more space between paragraphs, headings, tables and figures. So, the authors need to remove something less significant (e.g., "Data analysis of the parameter η") from the final version of the paper. Headings to sections, subsections and paragraphs should have consistent case. Please, consult the NIPS style file. - For Tables 1, 2 three digits after decimal look not good, i.e., consider “30.4±6.5” instead of “0.304±0.065”. Then also replace “F1” with “F1x100”. This will free some space, which can be used to add columns describing what are AU2, 12, etc., and, more interestingly, their frequencies in the datasets. For instance, the authors say in lines 287-288: “Furthermore, the IB-CNN is more effective in recognizing infrequent AUs with insufficient training data” and in lines  279-280: “To deal with insufficient positive samples in a mini-batch.” In Tables 1, 2 it would be great to provide some numbers relating to the frequencies of occurrence to support these statements. - The paragraph starting at line 83 is a little bit confusing. The authors say that “A few approaches developed new objective functions to improve recognition performance...”. Then, in the same paragraph they say that “Hinton et al. [23] utilized a dropout ...”, which is not really related to the objective functions nor to making decision (dropout is not used during inference, so it is still the inner product). Consider polishing this paragraph. It’s probably better to say “A few works developed improved training procedures...”.  - Fig. 1 has description for (a) and (b), but there is no (a) and (b) on the figure.  - The authors are suggested to follow punctuation rules in expressions, which are part of sentences (in fact, all expressions (1)-(10) in the paper miss commas and dots). - Lines 142: “However, the information captured previously, e.g., the weights and thresholds of the active neurons, is discarded for a new batch.” And related line 170: “Compared to the B-CNN, the IB-CNN exploits the information from all mini-batches.” Although, Fig. 2 compares a B-CNN and an IB-CNN, in text these sentences may sound confusing. Could the authors specify more concretely what is discarded and what is not, i.e. which weights and thresholds? Otherwise, it sounds like all information from the previous batch is discarded and it is not clear then how the network can be trained.  - Can the authors solve the issue with references to make them hyperlinked?


A scalable end-to-end Gaussian process adapter for irregularly sampled time series classification
We present a general framework for classification of sparse and irregularly-sampled time series. The properties of such time series can result in substantial uncertainty about the values of the underlying temporal processes, while making the data difficult to deal with using standard classification methods that assume fixeddimensional feature spaces. To address these challenges, we propose an uncertaintyaware classification framework based on a special computational layer we refer to as the Gaussian process adapter that can connect irregularly sampled time series data to any black-box classifier learnable using gradient descent. We show how to scale up the required computations based on combining the structured kernel interpolation framework and the Lanczos approximation method, and how to discriminatively train the Gaussian process adapter in combination with a number of classifiers end-to-end using backpropagation.

The paper builds a method of classification of irregularly sampled time series. GP regression is applied to the time series, and an efficient method is proposed to draw samples from the posterior functions, to propagate through the classifier. To do this, a regular set of points is defined on which all the samples will be interpolated (with uncertainty), the classifier is trained on the function values at those points, and the expected loss is computed by Monte Carlo over the GPR posteriors.
Section 1 The standard of writing in this work is good, it was a pleasure to read the paper, and I understood nearly everything on the first pass. I'm still not entirely clear what the MEG method is, though, but I understand it uses Fourier features and a one-time estimate of the kernel parameters (i.e. it's not 'end-to-end').  Section 2 is standard GP background. Fine.   Section 3. line 96. Naming your "framework" is rather overkill, no? You're propagating uncertainty in the inputs through to a classifier model. I'm sure similar things have been done for e.g. missing data: you might view your problem as a missing data problem, with GP imputation, if so inclined.  It would be nice if you could outline what MEG does, and how it differs from your approach. My understanding is that the main difference between this paper and MEG is that you propagate samples through, and MEG just propagates the mean, is that right?  Section 4 is the most technical part of the paper. In section 4.1, the authors recap the SKI framework of Wilson and Nickish. In section 4.2, they show how do compute approximate square-root-matrix vector multiplication using the Lanczos method. I suspect that this procedure may be straightforward to those more versed in numerical linear algebra, but I think it's a neat cross-over with ML, and it does depend a little on combination with the SKI idea to get fast matrix-vector operations inside the Lanczos loop.  In section 5, the authors reiterate that they are doing end-to-end learning, and so will attempt to learn the GP parameters alongside the parameters of the classifier. This means backpropagating through the Lanczos operations.  Section 6 does not do a very good job of explaining the difference between this work and MEG. At this point, I'm left with the burning question: "is this worth it"?  Section 7.  Section 7.1 is well done: it attempt to answer the question about the accuracy of the sampling-approximation.  Figure 1 : caption should explain what the colors mean, very confusing otherwise.  In section 7.2, the method is demonstrated on a single dataset. Disappointingly, the irregular-sampling of the dataset is artificial i.e. data were sub-sampled irregularly. The experiment does attempt to address my main concern: "is it really worth training everything together", and table 1 purports that this is so.  The paper is is rather let down by a lack of scientific investigation as to _why_ this must be so. What were the different kernel parameters that were learned using the two methods? Are the GPs smoother under the end-to-end framework? Does the noise get interpreted differently? Maybe we could see a plot of the different imputed time series?!  The paper is also let down by failing to provide a motivating use-case. If it was difficult to find an irregularly sampled dataset, what's the point of the paper? For this reason, I've marked the paper down for impact.


Avoiding Imposters and Delinquents: Adversarial Crowdsourcing and Peer Prediction
We consider a crowdsourcing model in which n workers are asked to rate the quality of n items previously generated by other workers. An unknown set of αn workers generate reliable ratings, while the remaining workers may behave arbitrarily and possibly adversarially. The manager of the experiment can also manually evaluate the quality of a small number of items, and wishes to curate together almost all of the high-quality items with at most an fraction of low-quality items. Perhaps surprisingly, we show that this is possible with an amount of work required of the manager, and each worker, that does not scale with n: the dataset can be curated withÕ 1 βα 3 4 ratings per worker, andÕ 1 β 2 ratings by the manager, where β is the fraction of high-quality items. Our results extend to the more general setting of peer prediction, including peer grading in online classrooms.

This paper considers the problem of aggregating the work of many crowdsourcing workers, each of whom is capable of providing ratings on a set of m available items. The goal is to have the workers provide ratings for only a small fraction of the items, and still recover the "top K" items (with limited error) despite the potential for adversarial workers who can provide false or damaging ratings to the system. The authors provide a set of algorithms in how to assign workers to tasks, and then from their responses how to estimate the highest-quality items. The main result of the paper is  an associated bound that specifies how many ratings per worker are required to get a good estimate of the top group in terms of the error rate of the workers, the number of "top items" required, and the fraction of poor workers.
 My overall evaluation of this work is that I find it quite interesting, but with some caveats and concerns. Here are the high-level points:  1. The goal of the paper, which is to design an algorithm to find highly-desired items in a large pool, seems very natural to me. I am also quite satisified with the worker model, which I think is reasonably realistic: some fraction of the workers provide accurate and high-quality ratings with few errors, whereas another fraction of the workers provide poor or potentially-harmful ratings, and the algorithm is able to obtain a small number of "gold standard ratings". Within this setting, the authors provide a very straightforward solution using a "matrix completion"-like algorithm, and they combine a number of interesting ideas (matrix concentration, nuclear-norm regularization, etc.) to obtain their result. The natural question to ask is "how many worker ratings and gold-standard ratings are needed?" and the bound proven by the authors provides a reasonably sufficient and clear answer.  2. One of the downsides of the paper is that it seems to be trying to do a lot in 8 pages. The paper proposes one main algorithm, but includes about 3-4 other subroutines, each of which involves some significant complexity and detail. There is an application of randomized rounding, the authors need to apply a Lagrangian duality argument, there is the use of a matrix optimization with a trace-norm constraint, etc. There is a lot of notation as well, and keeping it all straight while reading is quite a challenge. So one major objection is that the authors simply tried to do too much here, and maybe went a little overboard without describing in suitable detail all of the pieces.  3. A bigger concern is that the paper is sloppy in a number of places, and one has the impression that it was written in a hurry without a careful reading. Here are just a few (big) examples: -- In the definition of the setting, the authors refer to "when i \in [m] and j \in [m]" which suggests that the #workers and #items are the same (which is suggested in the intro as well). But later it's clear that the authors probably meant "i \in [n]" -- The description in Figure 1 says "indicated by \tilde{r}" but I don't see that character anywhere in the figure (and I wanted to understand what it means) -- The authors refer to "rating it ourself" -- who is "ourself"??! This was a very confusing way of describing what I can only assume to be a "gold standard oracle". -- I tried to look at references in the main body of the document to see what papers were being cited and... I found nothing. Where is Moitra et al? Agarwal et al? Mannor/Tsitsiklis? These missing references were very suspicious to me, and it makes it very hard to evaluate the novelty. -- This seems to be a highly-relevant paper which doesn't appear to have been cited: *Who Moderates the Moderators? Crowdsourcing Abuse Detection in User-Generated Content* A Ghosh, S Kale, P McAfee  I would lean towards acceptance for this paper, as many of the issues above can be fixed with a good revision. But indeed some doubts still linger.


Leveraging Sparsity for Efficient Submodular Data Summarization
The facility location problem is widely used for summarizing large datasets and has additional applications in sensor placement, image retrieval, and clustering. One difficulty of this problem is that submodular optimization algorithms require the calculation of pairwise benefits for all items in the dataset. This is infeasible for large problems, so recent work proposed to only calculate nearest neighbor benefits. One limitation is that several strong assumptions were invoked to obtain provable approximation guarantees. In this paper we establish that these extra assumptions are not necessary-solving the sparsified problem will be almost optimal under the standard assumptions of the problem. We then analyze a different method of sparsification that is a better model for methods such as Locality Sensitive Hashing to accelerate the nearest neighbor computations and extend the use of the problem to a broader family of similarities. We validate our approach by demonstrating that it rapidly generates interpretable summaries.

This paper studies a facility-location problem formulated as a submodular optimization problem. In machine learning, this problem has previously been shown to apply in the context of dataset summarization/k-medians clustering. Despite the submodularity of the objective function, solving the problem with the standard greedy algorithm is prohibitive because the complexity is k*n*m where k is the cardinality constraint and n*m is the size of the "benefit matrix". A previously suggested approach is to first sparsify the benefit matrix and then apply an optimization algorithm on the sparsified matrix.  This paper refines the results from [36] by giving theoretical guarantees on the approximation ratio obtained by this method as a function of the chosen sparsification level. Two methods of sparsification are analyzed:  * top t: only the top t items (most similar) are kept in each row of the benefit matrix  * threshold based: only the entries of benefit matrix above the chosen threshold are kept.  Finally experiments show how the suggested approach outperforms (in terms of the quality of the solution) the (non sparsified) greedy algorithm for the same running time. Or equivalently, reaching a given solution quality takes much longer if no sparsification is used.
The problem is interesting and well-motivated. While I haven't read paper [36] upon which this paper claims to build upon, I understand that a complete theoretical understanding of the sparsification-based approach was still missing and this paper is a good attempt at completing the picture.   I think having precise theoretical guarantees is very valuable, especially since they come with almost matching lower bound (up to logarithmic factors).  My only concern is about section 3.1 (threshold-based sparsification): I think this section should have a more precise discussion of the following two facts:  * the stated theoretical guarantee is data dependent. Is there any way to obtain a data-independent bound under extra assumptions? For example, assume that the benefit matrix comes from a distribution, then given threshold x, the sparsification is "good" with high probability. The benefit of such a result would be to give some guidance on how to choose the threshold, or what is a good threshold (which the paper only briefly discusses).  * it is also not clear how the LSH-based approach combines with the threshold-based approach. It would be nice to see an "end-to-end" theorem/proposition of the following flavor: assume the matrix is sparsified using LSH, then with high probability, we get approximation of...  Finally, the experiments are interesting and convincing. However, while the conclusion of section 4.1 is clear, the results of section 4.2 are hard to interpret: there is no metric to quantify the quality of the solution and it seems that the reader is simply invited to "eyeball" the results, as a consequence I am not sure that this section is necessary.


Multimodal Residual Learning for Visual QA
Deep neural networks continue to advance the state-of-the-art of image recognition tasks with various methods. However, applications of these methods to multimodality remain limited. We present Multimodal Residual Networks (MRN) for the multimodal residual learning of visual question-answering, which extends the idea of the deep residual learning. Unlike the deep residual learning, MRN effectively learns the joint representation from vision and language information. The main idea is to use element-wise multiplication for the joint residual mappings exploiting the residual learning of the attentional models in recent studies. Various alternative models introduced by multimodality are explored based on our study. We achieve the state-of-the-art results on the Visual QA dataset for both Open-Ended and Multiple-Choice tasks. Moreover, we introduce a novel method to visualize the attention effect of the joint representations for each learning block using back-propagation algorithm, even though the visual features are collapsed without spatial information.

The authors proposed a new network architecture to extend the deep residual learning to multi-modal inputs, applied to a visual question-answering task.  Unlike some previous work where explicit attention network mechanism was utilized, the proposed method follows a previous work using element-wise multiplication between question and visual feature vectors for the joint residual function representation, seen as an implicit attention mechanism. The authors experimented with multiple variants of the architecture and settled on a relatively simple learning block in a 3-block layered network. The combination of deep residual learning and implicit attention were shown to be effective on the Visual QA data set and out-performed other state-of-the-art results. To visualize the attention effect of multi-modal residual learning, the authors proposed a technique to display the difference between visual input and the joint residual mapping with back-propagation for each learning block. Examples from visual QA task intuitively demonstrated the implicit attention effect.
The authors successfully built upon two effective ideas, the deep residual learning and element-wise multiplication for implicit attention, and created a solution for general multi-modal tasks. Experiments were carefully run to select an optimal architecture and hyper-parameters for the targeted Visual QA task.  The results appeared to be superb, compared to previous studies with various deep learning techniques. It would be helpful if the authors can present additional comparison with existing techniques in terms of model parameter size, as well as amount of data required for learning.  It would also be interesting to separately assess the value of residual learning and implicit attention on the Visual QA task, to help understand which aspect is the most critical. The proposed visualization method was intuitively appealing and the examples demonstrated its effectiveness in explaining the implicit attention mechanism. It would be helpful if the authors can include additional explanation on the differences between the three images at each block layer, and perhaps provide an intuitive explanation why three layer appears to be optimal for this task.


An urn model for majority voting in classification ensembles
In this work we analyze the class prediction of parallel randomized ensembles by majority voting as an urn model. For a given test instance, the ensemble can be viewed as an urn of marbles of different colors. A marble represents an individual classifier. Its color represents the class label prediction of the corresponding classifier. The sequential querying of classifiers in the ensemble can be seen as draws without replacement from the urn. An analysis of this classical urn model based on the hypergeometric distribution makes it possible to estimate the confidence on the outcome of majority voting when only a fraction of the individual predictions is known. These estimates can be used to speed up the prediction by the ensemble. Specifically, the aggregation of votes can be halted when the confidence in the final prediction is sufficiently high. If one assumes a uniform prior for the distribution of possible votes the analysis is shown to be equivalent to a previous one based on Dirichlet distributions. The advantage of the current approach is that prior knowledge on the possible vote outcomes can be readily incorporated in a Bayesian framework. We show how incorporating this type of problem-specific knowledge into the statistical analysis of majority voting leads to faster classification by the ensemble and allows us to estimate the expected average speed-up beforehand.

In this paper, the authors describe a new algorithm for dynamic ensemble pruning for classification tasks. The main idea of the algorithm is to stop querying more classifiers of an ensemble if the probability that the current classification result for one instance will coincide with the result obtained by querying all classifiers of that ensemble reaches a preset threshold. This allows to trade off classification accuracy with classification speed. In contrast to previous approaches, this algorithm models the ensemble voting process as urn problem, where each classifier is represented as one marble in the urn and its classification result is modeled as the color of this marble. Querying a subset of classifiers can then be seen as a draw without replacement from this urn. This model makes it possible for each dataset to take into account the prior distribution of class biases of the different classifiers. In their paper, the authors show that their solution is equivalent to the Statistical Instance-Based Algorithm (SIBA) when a uniform prior distribution of those class biases is assumed. Furthermore, they propose to use Monte Carlo estimation to obtain non-parametric non-uniform prior distributions and compare their algorithm with SIBA. The authors report that the disagreement rate of their algorithm is closer to the preset threshold than SIBA’s, which leads to a significant speed up, since less classifiers need to be queried.
           Technical quality:           In my opinion, the proofs provided for proposition 2 and 3 are correct. The performed experiments seem suitable to compare the proposed approach with SIBA with respect to disagreement rate and classification speed. One particular nice thing about the performed experiments is that the authors also compare their approach with a more simple dynamic pruning algorithm, which stops querying classifiers if their vote would not change the result. However, it would be nice if the authors would have evaluated the reported measurements using statistical measures or tests. For instance, the authors argue that Table 1 shows that their approach produces disagreement rates closer to the specified one than SIBA. This could be easier illustrated by at least reporting the mean disagreement rate over all datasets. Also, by performing statistical analysis the authors could potentially show significant differences between the disagreement rates of both algorithms.                      Novelty:           This paper improves on SIBA in that it allows to adjust to the class bias of an ensemble, instead of assuming a uniform distribution. It is novel in the way the authors model the classification using an ensemble of classifiers as drawn without replacement from an urn of marbles and derive from this model a sound way to determine the confidence that a queried subset of classifiers will agree with the complete ensemble.                      Potential impact:           I believe that this work can be interesting for many people, since it provides a sound way to trade off voting accuracy and speed in an ensemble of classifiers, which can be useful when dealing with large amounts of instances. It has also a broad applicability, since this algorithm is not limited to work with random forest, but could also be applied to an ensemble of different classifiers.                      Clarity and presentation:           Overall, the presentation of the paper is good and the work is clearly described and explained. Especially the main intuition behind the proposed algorithm is well explained and the shift from an urn of marbles to an ensemble of classifiers is done nicely.                      The mathematical notation as well as the proofs in section 2 are very clear and rather easy to follow. It is helpful that the authors state the Chu-Vandermonde Identity, upper negation and the symmetry property of the binomial, which they are using to prove proposition 2 and 3. However, there appear to be two minor editing mistakes in the proofs. In the equation under line 124, it is unclear how the authors arrived at the expression on the left-hand side. In my opinion, it would have been enough to copy the right-hand side of the equation under line 123, since applying upper negation to this expression yields the right-hand side of the equation under line 124. Additionally, there is a plus sign instead of a minus sign in the upper bound of the second sum symbol in the equation under line 138. The current upper bound is $T - \sum_{i=1}^{l-2} T_{i}^{*} + t_{l}$, but it should be $T - \sum_{i=1}^{l-2} T_{i}^{*} - t_{l}$ instead.                      One issue in the presentation of the paper are the plots. While the plot of the vote distribution is good, the plots depicting the disagreement rates should be revised. According to the values on the x-axis and the description in the paper the x-axis should be named “1 - alpha” instead of “alpha”. Additionally, it is impossible to distinguish the different lines in these plots in a black and white printout. Therefore, I would recommend to use different line types to make it more clear which line belongs to which label without relying on different marker types and colors.                      Throughout the paper, the authors refer to the used sampling technique as “Mortecarlo sampling”. However, I believe this should be changed to “Monte Carlo sampling” or properly referenced as different sampling technique. Other minor spelling issues are:           line 7: “to” missing after “possible”           line 36: “on” instead of “of”           line 71: repetition of the word “marbles”           line 98: “than” instead of “that”           line 108: “infinite” instead of “infinity”            line 131: “by” missing after “simplified”           line 255, 258, 263: “a” instead of “an”


On statistical learning via the lens of compression
This work continues the study of the relationship between sample compression schemes and statistical learning, which has been mostly investigated within the framework of binary classification. The central theme of this work is establishing equivalences between learnability and compressibility, and utilizing these equivalences in the study of statistical learning theory. We begin with the setting of multiclass categorization (zero/one loss). We prove that in this case learnability is equivalent to compression of logarithmic sample size, and that uniform convergence implies compression of constant size. We then consider Vapnik's general learning setting: we show that in order to extend the compressibility-learnability equivalence to this case, it is necessary to consider an approximate variant of compression. Finally, we provide some applications of the compressibility-learnability equivalences.

This paper studies learning with general label spaces, under either the 0-1 loss or in some cases more general losses, and establishes relationships between several quantities of interest in learning theory: namely, realizable-case PAC sample complexity, agnostic PAC sample complexity, the size of a sample compression scheme (in both realizable-case and agnostic-case variants), and the size of an approximate sample compression scheme (again, in both variants). Under the 0-1 loss, they show that the leading complexity factor in all of these quantities can be bounded by the others up to certain logarithmic factors.  Under general losses, some of these relations break down, but they still find a relation between realizable-case sample complexity and realizable-case approximate sample compression, and between agnostic-case sample complexity and agnostic-case approximate sample compression.  The paper also includes a very nice separation result, arguing that for losses other than 0/1 loss (in fact, even for 3-valued losses), realizable-case learnability does not imply agnostic learnability. Specifically, they introduce an interesting construction of a label space and loss function, such that realizable-case learning is trivial, but there are distributions in the agnostic case for which there exists a hypothesis of risk 1/2, but it is not possible to identify a classifier of risk less than 3/4 from a finite data set. They establish the negative result by combining their equivalence of approximate compressability and learnability with some combinatorial arguments on hypergraphs (of size a very very large infinity); they also sketch a much simpler construction not requiring such large infinities.  Given the setup, it seems a direct proof of the negative result (not going through the compression equivalence) is also possible, but making use of the equivalence to approximate compressability offers a nice simplification and ties the result in with the theme of the rest of the paper.
Most of the results established in the paper would, in the special case of binary classification, trivially follow from the known upper and lower bounds on sample complexity based on the VC dimension. However, the results were not previously known for multiclass learning, and other general loss functions.  The results for the 0-1 loss are not particularly surprising, but it is good to know that, for instance, in multiclass classification with the 0-1 loss, the complexity measure in the agnostic sample complexity is the same as that in the realizable-case (up to log factors, but no extra factors such as log(|Y|) not present in the realizable-case sample complexity).  They also prove a tighter lower bound than previously known for the sample complexity of uniform convergence for multiclass classification in Theorem 3.6.  The techniques used in the proofs are mostly straightforward or have appeared in other related contexts previously.  They seem primarily to be pointing out ways to combine known results and argument to reach new conclusions.  The separation result in section 4.3 may be the exception, as I do not believe I have seen this kind of construction before this paper.  There are also some novel aspects in the proof of the lower bound in Theorem 3.6.  Even if most of the arguments are  straightforward, the results add up to a fairly complete picture of  how these quantities are related in general.  Furthermore, as  demonstrated in the separation result in section 4.3, the results may be useful for simplifying proofs of learnability / non-learnability, by allowing one to focus on existence / non-existence of finite approximate compression schemes.  The paper is generally well-written and easy to read.  I did notice several grammatical mistakes, so a careful read-through to correct these would improve readability.  minor comments:  I think the upper bound in Theorem 3.6 can be improved, removing the log(1/epsilon) factor, simply by using the tail bound of Talagrand (based on the chaining technique) in place of the bound of Vapnik & Chervonenkis; this is the argument that appeared in the proof of Theorem 5 of Daniely, Sabato, Ben-David & Shalev-Shwartz (2015).  In the second proof of Theorem 4.4 (last paragraph of the full paper), the fonts for X and Y need to be made consistent.  Also, the range of the hypothesis class H should have Y_i in the union, rather than X_i. Finally, since most readers will find this second proof much more accessible than the first, it seems important to include enough details so that they can easily follow the argument.  In particular, instead of merely claiming "one can verify" points (i) and (ii), it would be best to include these arguments explicitly.


Improved Techniques for Training GANs
We present a variety of new architectural features and training procedures that we apply to the generative adversarial networks (GANs) framework. Using our new techniques, we achieve state-of-the-art results in semi-supervised classification on MNIST, CIFAR-10 and SVHN. The generated images are of high quality as confirmed by a visual Turing test: our model generates MNIST samples that humans cannot distinguish from real data, and CIFAR-10 samples that yield a human error rate of 21.3%. We also present ImageNet samples with unprecedented resolution and show that our methods enable the model to learn recognizable features of ImageNet classes.
Another of our proposed techniques, minibatch features, is based in part on ideas used for batch normalization [12], while our proposed virtual batch normalization is a direct extension of batch normalization.
One of the primary goals of this work is to improve the effectiveness of generative adversarial networks for semi-supervised learning (improving the performance of a supervised task, in this case, classification, by learning on additional unlabeled examples). Like many deep generative models, GANs have previously been applied to semi-supervised learning [13,14], and our work can be seen as a continuation and refinement of this effort. In concurrent work, Odena [15] proposes to extend GANs to predict image labels like we do in Section 5, but without our feature matching extension (Section 3.1) which we found to be critical for obtaining state-of-the-art performance.

This paper presents a collection of practical techniques for improving the training of GAN models and the quality of the samples from the learned models.  Most of the techniques are motivated heuristically, but seems to help from the experiment results.  A new application of the GAN models to semi-supervised learning is also proposed, and the proposed approach achieves great results on MNIST, CIFAR-10 and SVHN semi-supervised classification.
This paper presents a collection of techniques for improving GAN training.  Most of them are motivated heuristically, and the results are evaluated empirically.  The collection of these techniques, are effective and GAN models trained with these techniques are able to generate very good quality samples on MNIST, CIFAR-10 and SVHN.  I found the semi-supervised learning approach very interesting, and I think it should make the samples look more like examples from the classes, thus improving perceptual sample quality.  This is a natural result of optimizing the loss presented in section 5.  A big part of the improvement of sample quality may actually come from this semi-supervised learning.  However this is not verified very thoroughly.  In Table 3, the “Our methods” vs “-L” comparison seems to be verifying the use of semi-supervised learning vs not, if I understand the meaning of “L” correctly.  However, the sample quality of the “-L” model seems rather bad, and a reasonably well trained standard GAN model should do a little better, at least the obvious artifacts can be removed.  On the other hand, the experiment results show that good generative model of images do not help that much for learning classifiers in semi-supervised learning, while good classifier models learned in a semi-supervised way do not help training good generative models.  So clearly something not well understood is going on, and our intuitions might be incorrect or incomplete.  On a separate note, having achieved state-of-the-art semi-supervised learning results is nice, but from the paper it is hard to see if the methods are directly comparable.  It would be good to have some more careful control over the experiment setup, for example by making sure the classifier networks having the same architecture as previous methods, and then demonstrating the benefit of using the proposed semi-supervised training procedure.  Otherwise the picture is mixed.  The proposed Inception score metric makes intuitive sense.  However, a bad generative model that generates nothing like any of the classes, thus having p(y|x=G(z)) distributing equally across all classes, seems able to fool this metric.  It probably makes sense to measure the quality of models in similar nature using this metric, but if comparing models of different nature, for example GANs vs VAEs the results may not be very comparable as the behavior of this metric is not well understood.  The formulation of the minibatch discrimination features is a bit arbitrary for me.  I wonder if there are any reason behind actually choosing such a formulation.  Presumably there are other ways to introduce minibatch features.  At last, the paper proposes the techniques with the goal of making GAN training more stable.  But none of the experiments actually show that the GAN training is more “stable” as a result of these techniques.  The results mostly focus on quality of samples and semi-supervised learning results, but do not touch any measure of “stability”.  Maybe it would be better to change the phrasing a little bit?  Or maybe it would be more interesting to show some results demonstrating that these techniques actually improve “stability” with some understandable measure, even qualitative ones.  Overall I like the results of this paper, and some of the proposed techniques are novel, the semi-supervised learning approach is also interesting.  There are a lot of unanswered questions but some of it may be left to future work.