Bilevel Learning of the Group Lasso Structure
Regression with group-sparsity penalty plays a central role in high-dimensional prediction problems. However, most existing methods require the group structure to be known a priori. In practice, this may be a too strong assumption, potentially hampering the effectiveness of the regularization method. To circumvent this issue, we present a method to estimate the group structure by means of a continuous bilevel optimization problem where the data is split into training and validation sets. Our approach relies on an approximation scheme where the lower level problem is replaced by a smooth dual forward-backward algorithm with Bregman distances. We provide guarantees regarding the convergence of the approximate procedure to the exact problem and demonstrate the well behaviour of the proposed method on synthetic experiments. Finally, a preliminary application to genes expression data is tackled with the purpose of unveiling functional groups.

The paper addresses the group lasso problem where the groups have to be learned (rather than being pre-supplied). This problem is cast as a (continuous) bilevel optimisation problem and the optimisation problem addressed by an algorithm devised by the authors. Experiments on synthetic data are given and support the usefulness of the presented method.  Groups are discrete features so the authors have to relax the problem by removing the integrality constraint on group membership indicator variables. We are not given an idea of how much is "lost" by relaxing, but given the difficulty of the original problem I have no objection to solving a relaxation of the original problem.  The paper is well-written (apart from many small errors - see below), the problem worth solving, the algorithm reasonable and the experimental results are positive. Using validation error to learn the groups is nice and straightforward. The empirical results given are also well chosen, particuarly the comparison with oracle GL. So I have difficulty in finding substantive criticisms of the paper. It's a typical NIPS submission: optimise to solve some machine learning problem.  SMALL PROBLEMS  21: "a priori structure of the data". I found this confusing: what is a priori can not be found in the data.  25: we are only told later what \theta is so it is confusing here.  100: arbitrary -> arbitrarily  111: iterates -> iterations  135: the solve -> to solve  212: as less informative -> as uninformative  246,247: change to "...  Q sufficiently permits reaching"  247: stress out -> stress  250: "Since a group ..." This sentence needs rewriting.  Refs: Fix capitalisation: e.g. Lipschitz, Bregman  AFTER DISCUSSION / FEEDBACK  I remain positive about the paper!


Constructing Deep Neural Networks by Bayesian Network Structure Learning
We introduce a principled approach for unsupervised structure learning of deep neural networks. We propose a new interpretation for depth and inter-layer connectivity where conditional independencies in the input distribution are encoded hierarchically in the network structure. Thus, the depth of the network is determined inherently. The proposed method casts the problem of neural network structure learning as a problem of Bayesian network structure learning. Then, instead of directly learning the discriminative structure, it learns a generative graph, constructs its stochastic inverse, and then constructs a discriminative graph. We prove that conditional-dependency relations among the latent variables in the generative graph are preserved in the class-conditional discriminative graph. We demonstrate on image classification benchmarks that the deepest layers (convolutional and dense) of common networks can be replaced by significantly smaller learned structures, while maintaining classification accuracy-state-of-the-art on tested benchmarks. Our structure learning algorithm requires a small computational cost and runs efficiently on a standard desktop CPU.

The presented method learns a structure of a deep ANN by first learning a BN and then constructing the ANN from this BN.  The authors state that they "propose a new interpretation for depth and inter-layer connectivity in deep neural networks". Neurons in deep layers represent low-order conditional independencies (ie small conditioning set) and those in 'early' (non-deep) layers represent high-order CI relationships. These are all CI relations in the "X" ie the input vector of (observed) random variables.  Perhaps I am missing something here but I could not find an argument as to why this is a principled way to build deep ANNs with good performance. After all, when we actually use an ANN to make a prediction, all the X input variables will be observed, so it is not obvious to me why we should care about CI relations between the components of X. So we are using unsupervised learning to help supervised learning. Nothing wrong with that, but I think we need more about how the former helps the latter in this case.  The actual process of geneating the ANN structure is clearly described. Since the ultimate goal of the BN learning is somewhat unusual so is the BN learning algorithm - essentially we add latent variables to represent the (estimated) CI relations in the input.  Although I can't see why this is a good method for making ANN structure, perhaps it just is, so the empirical evaluation is very important. 5 image classification and 7 other benchmarks are used. Not too surprisingly the empirical results are positive: the learned (more sparse) structures do better than fully connected ones, particularly with smaller training sets. The authors also (in Section 4.2) compare to other methods: getting comparable accuracies to NAS but with smaller networks.  I think this is promising work and reasonable results have been produced but I judge that for publication in NIPS we need a fuller account of why/when this method should work.  SPECIFIC POINTS  Using a threshold for the CI tests set by a validation set is a good approach (certainly better than many constraint-based approaches where some arbitrary threshold is set and then CI test results on finite data are treated as 100% reliable). However, the authors do not address the issue of the inevitable errors in CI test results, despite flagging up this issue on lines 45-46.  line 255: smaller number of -> fewer  AFTER READING AUTHOR FEEDBACK  The authors have provided a better explanation of the rationale of their approach, which I hope will be included in any future version, and I have upped my rating appropriately.


Distributed Stochastic Optimization via Adaptive SGD
Stochastic convex optimization algorithms are the most popular way to train machine learning models on large-scale data. Scaling up the training process of these models is crucial, but the most popular algorithm, Stochastic Gradient Descent (SGD), is a serial method that is surprisingly hard to parallelize. In this paper, we propose an efficient distributed stochastic optimization method by combining adaptivity with variance reduction techniques. Our analysis yields a linear speedup in the number of machines, constant memory footprint, and only a logarithmic number of communication rounds. Critically, our approach is a black-box reduction that parallelizes any serial online learning algorithm, streamlining prior analysis and allowing us to leverage the significant progress that has been made in designing adaptive algorithms. In particular, we achieve optimal convergence rates without any prior knowledge of smoothness parameters, yielding a more robust algorithm that reduces the need for hyperparameter tuning. We implement our algorithm in the Spark distributed framework and exhibit dramatic performance gains on large-scale logistic regression problems.

=========== after rebuttal I carefully read the authors’ response. Some of my criticism has been addressed, yet I think that some critical flaws are still in the paper and I am not confident that these would be fixable in a final version within this submission round.  I believe that the authors do not make a strong case for their algorithm, since they do not appropriately compare with the state of the art. Eg, they do not compare to any of the algorithms mentioned below: https://arxiv.org/pdf/1511.01942.pdf and https://arxiv.org/pdf/1603.06861.pdf The authors mention in their experimental section: “Implementing our algorithm in the Spark environment is fairly straightforward. SVRG OL switches between two phases: a batch gradient computation phase and a serial SGD phase. “ As I mentioned in my review, this is very very similar -modulo the varying stepsize- to the algorithms above. At the very least the authors should have compared with the above.  Moreover, the authors *do not* support a key claim of the paper, ie that the proposed algorithm achieves a linear speedup? (this claim is made in the 4th sentence of their abstract, and repeated through out their text) There is no single experiment that compares the serial implementation of the algorithm with its distributed version, hence no scaling/speedup result can be inferred.   Since they addressed some of my criticisms, but not the major points above, I will update my score from 3 to 4.  ===============original review The authors present SVRG OL, an adaptive SVRG algorithm (not quite sure why it is refered as SGD), that is designed to get linear speedups on parallel architectures.  The theoretical contribution of the paper is that the algorithm requires log number of comm rounds to achieve an epsilon accurate result.  Although the problem setup is interesting, there are several issues with the paper that I list below:  - First of all, the presented algorithm is *identical* to  Algorithm 1 here:https://arxiv.org/pdf/1412.6606.pdf  Algorithm 1 here https://arxiv.org/pdf/1511.01942.pdf  and CheapSVRG here: https://arxiv.org/pdf/1603.06861.pdf It is unclear to me what is algorithmically novel between the above and the presented SVRG OL algorithm.  - This is certainly subjective, but I do not understand the usefulness of online optimization results in the context of distributed setups. In most dist. setups that are of interest, all data is available for batch processing. In this case convergence rates are indeed important, but online regret bounds don’t shed a particular insight on how to implement these algorithms.   - The authors mention “Unfortunately, minibatch-SGD obtains a communication complexity that scales as sqrt{N} (or N^1/4 for accelerated variants). In modern problems when N is extremely large, this overhead is prohibitively large.” This is absolutely not true, as every single large scale ML problem that trains DNNs uses backprop and minibatch SGD implementations. So although the # comm rounds is high, it is not prohibitive.   - The authors mention “prior analyses require specific settings for ⌘ that incorporate L and fail to converge with incorrect settings, requiring the user to manually tune ⌘ to obtain the desired performance. “  That is true, but in practice one almost never computes L or the str. cvx constant, as that can be as expensive as solving the problem. For non-convex problems that actually *never* happens (eg pre computing L).  - The authors implement the algorithm in spark. All the data sets tested fit in a single machine, and can most probably be run much faster with a simple python implementation of SVRG. I will not be convinced that the implementations presented are relevant, unless the authors have a comparison with a single machine, SVRG non-Spark baseline.  - Although the authors make a claim on linear speedups, no speedup curves are presented in the paper.  - minor: Although the authors mention they need log number of comm rounds, in Table 1 they present constants and omit the log factors. There is enough space to list the details in this table.   Overall, this paper deals with an interesting problem. however, the presented algorithm is not novel, the implementation and experiments might not be as relevant, and comparisons with serial SVRG are missing. I do not think the paper should be accepted to NIPS.


Quadratic Decomposable Submodular Function Minimization
We introduce a new convex optimization problem, termed quadratic decomposable submodular function minimization. The problem is closely related to decomposable submodular function minimization and arises in many learning on graphs and hypergraphs settings, such as graph-based semi-supervised learning and PageRank. We approach the problem via a new dual strategy and describe an objective that may be optimized via random coordinate descent (RCD) methods and projections onto cones. We also establish the linear convergence rate of the RCD algorithm and develop efficient projection algorithms with provable performance guarantees. Numerical experiments in semi-supervised learning on hypergraphs confirm the efficiency of the proposed algorithm and demonstrate the significant improvements in prediction accuracy with respect to state-of-the-art methods.

In this paper the authors consider the problem of minimizing sums of squares of Lovasz extensions. This is a convex problem and the authors derive its dual and propose several techniques for solving it that have a linear convergence rate. The consider problems over graphs - transductive learning and pagerank - and I am not aware of other problem classes that can be cast into this framework. What I find slightly confusing is why the authors posed the problem in terms of submodularity and Lovasz extensions, when, unless I'm mistaken, their theory and algorithms work for any polyhedral support function, not necessarily over base polyhedra. If I'm mistaken, it would be nice to point out what is the barrier to applying their methods beyond submodular polytopes. Moreover, unlike for submodular functions, there are no results about the quality of the rounded solution if we treat the problem as a continuous relaxation of the discrete problem.  In the dual the authors obtain a dual containing cones derived from the base polytope and they develop algorithms for projecting onto them. They use Frank-Wolfe and a variant of it (Fujishige's min-norm algorithm is essentially a variant of fully corrective Frank-Wolfe). What is interesting is that they use FW which is classically defined only for compact domains and apply it to cones. They keep the same 1/k rate, and it would be quite useful if their analysis can be extended to other families of cones "generated" by polytopes.  I unfortunately cannot comment much about the quality of the obtained solutions in the experimental section as I am not very familiar with the recent literature on transductive learning. I am interested why they did not compare against label propagation, which is a widely used method.  Questions and remarks:  1) The Lovasz extension is not necessarily positive. I guess you assume F to map to [0, oo) instead of R.  2) Perhaps you should point out in line 126 that you are using the conjugate representation of the quadratic.  3) What would happen if you introduce linear terms in the objective? How would your algorithms change?  4) Why do you present the coordinate descent method in section 4, when you obtain the best results using AP?  5) Why not also plot the value of the function you are minimizing in the experimental section, which is more important than the duality gap.  6) It would be interesting to see how does accuracy change as a function of cpu time. I am not sure if having gaps of the order of 1e-10 makes a big difference, a the problems are defined using noisy data.  7) If I'm looking at it right, in Figure 1 it is the case that gaps of 1e-8 and 1e-9 can have a quite different classification performance (3% vs 9% error). Is the top figure not representative of the gaps you obtain for the lower table, or is something else going on?  8) What is the submodular function corresponding to (10)? It is not in your original table due to the presence of sqrt(w_ii) and sqrt(w_jj) in the denominator.  9) How does the accuracy look like for the real data? You're only showing duality gaps, and not comparing the accuracy of the existing work.   ----  The rebuttal did address the questions that I had. To conclude, while I can not evaluate the impact on the transductive learning problems, I think this paper has several interest results on submodular optimization and the Frank-Wolfe algorithm that could be interesting to parts of the NIPS community. Hence, I am leaning towards accepting this submission.


Regularization Learning Networks: Deep Learning for Tabular Datasets
Despite their impressive performance, Deep Neural Networks (DNNs) typically underperform Gradient Boosting Trees (GBTs) on many tabular-dataset learning tasks. We propose that applying a different regularization coefficient to each weight might boost the performance of DNNs by allowing them to make more use of the more relevant inputs. However, this will lead to an intractable number of hyperparameters. Here, we introduce Regularization Learning Networks (RLNs), which overcome this challenge by introducing an efficient hyperparameter tuning scheme which minimizes a new Counterfactual Loss. Our results show that RLNs significantly improve DNNs on tabular datasets, and achieve comparable results to GBTs, with the best performance achieved with an ensemble that combines GBTs and RLNs. RLNs produce extremely sparse networks, eliminating up to 99.8% of the network edges and 82% of the input features, thus providing more interpretable models and reveal the importance that the network assigns to different inputs. RLNs could efficiently learn a single network in datasets that comprise both tabular and unstructured data, such as in the setting of medical imaging accompanied by electronic health records. An open source implementation of RLN can be found at https://github.com/irashavitt/regularization_ learning_networks.

Summary This work develops Regularization Learning Networks (RLN), an approach to learn varying levels of regularization for different network weights. Regularization levels for (D)NN are usually  a global hyper-parameter, optimized via CV for the whole network. The author point out that figuring different levels or regularization for different features is appealing especially for data where features are of different “type” with some possibly highly important while others not and the transition between those is more striking than common in say structured images. Good examples are smaller tabular data of expression, EHR where DL commonly underperforms.  The CV approach without a direct target function to optimize does not scale for optimizing each weight separately. Instead, the authors develop an elegant solution in the paper, the counterfactual loss (L_cf). The SGD now has two contributions: the gradient for the loss and the gradient for the regularization, as a result of evaluating the delta from moving the weight i to a new point W_{t+1,i} on a new batch Z_{t+1} as a derivative of the hyperparameter for penalty \Lamba_i.   The authors evaluate the performance of their method on a dataset of microbiome from patients with several different target values to predict (Age, BMI, etc.). Their RLN greatly improves compared to DNN with standard “global” regularization (optimized by CV), but GBT is still better. They also try ensembles of learners. As expected these give improved results, with the best predictor composed of a combination of GBT and RLN. Finally, they evaluate the level or regularization on the features during learning (Fig7) and the features importance compared to a DNN (Fig8), showing learned models are extremely sparse.  Pros: The paper addresses an important and challenging problem. The solution seems novel, simple/elegant, yet efficient. Paper is generally clearly written. The solution appears to significantly improve performance compared to standard DNN regularization. It was nice to see the authors also make an effort to understand “what is going on” in terms of regularization (Fig7) and interpretation applying matrix product  (p. 8 top paragraph, Fig 8).  Cons:  Overall, we did not find any major flaws though several improvements can definitely be made.   First, we note that at the end of the day on all tasks tested GBT was still the better model. Only an ensemble of GBT+ RLN gave a very slight improvement. Thus, on all tasks tested, there is no real benefit in the proposed method.  The above is related to the fact that while the authors rightfully motivate their work with a general hypothesis about (un)structured/tabular data etc. they only use a single dataset with no synthetic testing to explore behavior. Why? We would like to see the final including a more elaborate evaluation of different datasets which the authors are already aware of (e.g. line 21). This could include different synthetic settings. For example: What happens if the features are informative but (highly) correlated? How does the amount of data affect the new framework?  Even on the dataset used the effect of the RLN specific hyperparameters are not explored (they are listed in the supp). Similarly, the analysis of feature importance and how to decide on those important ones was lacking. Why are 5 features selected? What is the criteria for significance? Do they make sense? There are many other ways to extract meaning from DNN (e.g. IG) which are at least worth discussing.   It is also important that the authors make code/analysis available - it could be highly useful *if* indeed it proves to give a boost (pun intended) in performance in other domains.  Other comments:  In the introduction and related work the authors keep interchanging the (undefined) terms (un)structured data, (non) distributed representation, and tabular data. Though the general intention is clear this adds unnecessary confusion.  The authors can do a better job introducing, deriving the main idea (p.3) and giving the intuition behind it. As is, it took a few back and forth to understand things.  Many of the figures are informative yet not so user-friendly and their caption limited (Fig1, 5, 7, 8). Consider plotting differently or adding info.


Frequency-Domain Dynamic Pruning for Convolutional Neural Networks
Deep convolutional neural networks have demonstrated their powerfulness in a variety of applications. However, the storage and computational requirements have largely restricted their further extensions on mobile devices. Recently, pruning of unimportant parameters has been used for both network compression and acceleration. Considering that there are spatial redundancy within most filters in a CNN, we propose a frequency-domain dynamic pruning scheme to exploit the spatial correlations. The frequency-domain coefficients are pruned dynamically in each iteration and different frequency bands are pruned discriminatively, given their different importance on accuracy. Experimental results demonstrate that the proposed scheme can outperform previous spatial-domain counterparts by a large margin. Specifically, it can achieve a compression ratio of 8.4× and a theoretical inference speed-up of 9.2× for ResNet-110, while the accuracy is even better than the reference model on CIFAR-10.

[Response to rebuttal: Thanks to the authors for addressing my questions during rebuttal. My only major issue has been addressed and the same is true for my minor questions and issues, except for (5), which I do not consider crucial, particularly given that the authors only have one page for their response. Since most of my issues were regarding question that I had or minor detail that should be added to the paper, I have raised my confidence of reproducibility to 3. ]  The paper introduces a novel method for parameter-pruning in convolutional neural networks that operates in the frequency domain. The latter is a natural domain to determine parameter-importance for convolutional filters – most filters of a trained neural network are smooth and thus have high energy (i.e. importance) in their low-frequency components, which means that high-frequency components can often be pruned without much loss in task-performance. An additional advantage of the method is that pruning is not performed as a single post-training step, but parameters can be pruned and re-introduced during training in a continuous fashion, which has been shown to be beneficial in previous pruning schemes. The method is evaluated on three different image classification tasks (with a separate network architecture each) and outperforms the methods it is compared against.  The topic of neural network compression is timely and of high practical significance for deep learning applications. The paper is well written, although another pass to fix a few smaller glitches is recommended. The current state-of-the art and related work is nicely discussed in the introduction, though there are some recent methods that should be included and also compared against (see my comments below). In terms of originality, the method builds on two ideas which have been proposed recently: going to the frequency domain for neural network parameter pruning (references [14, 15] in the paper) and dynamic pruning and re-introduction of parameters during training (reference [18] in the paper). The paper combines both ideas in a novel and original way that makes a lot of sense and is demonstrated to work well empirically. The experimental section is satisfying, though not outstanding, except that comparison against some recent state-of-the-art methods is currently missing. I think this is a solid paper, which could be further improved to a great paper and therefore I recommend acceptance of the paper. My minor comments below are aimed at helping the authors to further improve the quality of the paper.  Major: While the Introduction generally gives a good overview over state-of-the-art approaches in network compression, there are a few methods that should be mentioned and if applicable the methods should also be compared against in the experimental section as these are spatial-domain pruning methods that perform on-par with the proposed method (luckily, most of them report results for LeNet-5 on MNIST, so there is no strict need to run additional experiments). Most of the methods use the idea of inducing sparsity during training via learning a posterior-distribution over weights under a sparsity-inducing prior. This paradigm has been remarkably successful and at least some of the methods have the advantage that they prune whole neurons/feature-maps, which leads to speed-ups without requiring special hardware. Please briefly discuss and compare against the following methods (I do not expect the authors to run additional experiments, all papers report results on LeNet-5, additional results and comparisons are a bonus of course): Weight pruning: [1, 2] Neuron/filter pruning: [3, 4, 5]  [1] Ullrich, Meeds, Welling, Soft weight-sharing for neural network compression, ICLR 2017 [2] Mochanov, Ashukha, Verov, Variational Dropout sparsifies neural networks, ICML 2017 [3] Louizos, Ullrich, Welling, Bayesian compression for deep learning, NIPS 2017 [4] Neklyudov, Molchanov, Ashukha, Vetrov, Structured Bayesian pruning via log-normal multiplicative noise, NIPS 2017 [5] Louizos, Welling, Kingma, Learning sparse neural networks through L0 regularization, ICLR 2018  Minor: (1) Sec. 2.3: In FDNP, the importance of a parameter is given by the frequency-coefficient (the energy of the frequency component). I find it natural to prune frequency components with small coefficients. However, in BA-FDNP the paper argues that frequency components have different magnitudes and thus proposes to bin frequency components and apply different pruning thresholds to each bin. Empirically this seems to work even better than FDNP, but I struggle a bit with the argument because this seems to go against the original argument that the frequency coefficient determines parameter importance. Do I misunderstand something or are you essentially saying with BA-FDNP that “you want to keep a certain number of parameters from each frequency band, even though potentially all the coefficients of that band have a very small magnitude compared to other bands”? (2) It would be nice to show a histogram of typical frequency components of a trained network and a pruned network. Perhaps this would also clarify the questions of my previous comment (1). Additionally it would also be interesting to see a few examples of pruned filters which have been transformed back into the spatial domain and compare them against the original, unpruned version. (3) Question regarding the training procedure: what is the loss function in Eq. (4) used in your experiments, standard cross-entropy loss? Why does it only involve a single layer though? In the pseudo-code (Appendix 2) it looks like you compute the loss for each layer and then perform a gradient-update step on the spatial-domain weights for each layer. I would have expected that you need to do the full forward pass through all layers first and then compute the loss and back-propagate through all layers accordingly. Did I miss something here? (4) I greatly appreciate that the authors are clear about the reported speed-ups being of a theoretical nature. Please also indicate in the abstract (line 11) that the speed-up is a theoretical one. (5) Line 143 – 146 and Eq(8): You mention that you evaluate several methods for setting the compression rate of the individual frequency bands. Could you please add some more details on alternatives that you have tried or propose to try (perhaps in the appendix?) and give some arguments on why you finally chose the function in Eq(8)? (6) Line 27: I think the reference you have here ([5] in the paper) only has binary weights but not activations, the one you are referring to is from the same main author, but the title is: Binarized Neural Networks: Training deep neural networks with weights and activations constrained to +1 or -1. (7) Reproducibility of experimental results: for all experiments please report: which gammas did you use in the experiments where you set them manually (line 197)? Do you use a learning-rate schedule, if so which one? What initialization method did you use, which optimizer with which parameters? Did you do any pre-processing / augmentation of the data? (8) Line 227: You mention that you prune “the other fully-connected layer in spatial domain” – with which method? Reference [18] from the paper? (9) Line 246: Same question as before: which method do you use for “spatial-domain dynamic pruning”? (10) Can you comment on whether a pre-trained network is strictly necessary for your method or whether there is also the potential to train from scratch?


Revisiting Decomposable Submodular Function Minimization with Incidence Relations
We introduce a new approach to decomposable submodular function minimization (DSFM) that exploits incidence relations. Incidence relations describe which variables effectively influence the component functions, and when properly utilized, they allow for improving the convergence rates of DSFM solvers. Our main results include the precise parametrization of the DSFM problem based on incidence relations, the development of new scalable alternative projections and parallel coordinate descent methods and an accompanying rigorous analysis of their convergence rates.

The authors consider the problem of submodular minimization under factorization and incidence assumptions. Specifically, they assume that the function decomposes into terms that depend only on a subset of the variables. They consider the minimum norm problem and two algorithms - alternating minimization (AP) and coordinate descent (CD). In both cases, they improve the old bounds in a pretty natural way - they replace NR by the sum of the neighbourhoods of the components. AP under incidence constraints has been also studied by  Scalable Variational Inference in Log-supermodular Models , Josip Djolonga, Andreas Krause, ICML 2015,  who also extend the results from Nishihara et al, and arrive at exactly the same problem as in this paper (they prove only under the assumption of a regular graph though). The fact that one scales by the inverse of the number of components (i.e. the skewed norms) is not surprising, as similar problems also appear in classical dual decomposition. Their randomized CD algorithm is novel, as they showed that one can not easily extend the results from Ene et al. I find that the intuition behind their approach is not well explained and the discussion could certainly be improved.  I am not sure about the applicability of these models though. For example, in the experimental section they split into 1065 components, but this is necessarily as only two can suffice -- all vertical and all horizontal edges. Similarly, the superpixels typically do not overlap -- if one does several layers they will overlap, but this is typically on the order of 2 to 3. I think if they implement it that way, there won't be any benefits when compared to the classical AP and RCD methods. Moreover, they only implement AP, while Jegelka et al. show that a reflection based approaches obtains much faster convergence. Finally, they measure certified duality gap vs # of iterations - I think the x-axis should be wall-clock time, as the comparison is unfair because different methods do varying number of projections per iteration.  Questions ---  * In the experiments for the superpixel potentials you do not have to use divide and conquer, as a single sort + isotonic regression would suffice, as explained in  Lim, Cong Han, and Stephen J. Wright. "Efficient Bregman projections onto the permutahedron and related polytopes." AISTATS 2016.  * l73, 74 - Maybe note that { i | x_i > 0 } and { i | x_i >= 0 } are both in the lattice of minimizers.  * l121. Can't you say that I(w) consists of R copies of w? It can be a bit confusing in the way it is stated, especially with the variable r appearing twice.  * l234. Not clear why convergence is guaranteed. Aren't you simply showing that you minimize an upper bound? More discussion could certainly help in this section.  * Def 3.8 Maybe say "We say that P is an alpha-proper for some \alpha\in(0,1) if ....". A better name might be alpha-sparse?  * Example 3.1. - Perhaps note that this is exactly a cut function on a chain graph?  * Spell out maybe in more detail why the inequality l* < ... holds? What is the y^* from Def 3.6 for this y?  * l230: unclear if you mean h_{r,i} is an indicator vector on coordinate {r, i} or if for all r\in C only y_{r,i} is nonzero, or i h_{r,*} for r\in C are the only non-zero blocks.  * You can cheaply improve the certificates using isotonic regression, as explained in Bach, [p. 281 ,"Primal candidates from dual candidates."].   Post rebuttal --- In Djolonga et al., they analyze the exactly same problem, even though the motivation is through L-Field. They specifically show that the problems are equivalent and use the same weighted projections, despite proving only the regular case. The authors did not respond to how the results would look like if the natural decomposition is done for the images. I expect their improvements to significantly diminish for this setting. I think the paper is interesting and the setting they consider is natural, although I do not think the practical benefits are well supported by the experiments, as the instances chosen have very natural and simple decompositions.


Predict Responsibly: Improving Fairness and Accuracy by Learning to Defer
In many machine learning applications, there are multiple decision-makers involved, both automated and human. The interaction between these agents often goes unaddressed in algorithmic development. In this work, we explore a simple version of this interaction with a two-stage framework containing an automated model and an external decision-maker. The model can choose to say PASS, and pass the decision downstream, as explored in rejection learning. We extend this concept by proposing learning to defer, which generalizes rejection learning by considering the effect of other agents in the decision-making process. We propose a learning algorithm which accounts for potential biases held by external decision-makers in a system. Experiments demonstrate that learning to defer can make systems not only more accurate but also less biased. Even when working with inconsistent or biased users, we show that deferring models still greatly improve the accuracy and/or fairness of the entire system.
Our main contribution is the formulation of adaptive rejection learning, which we call learning to defer, where the model works adaptively with the decision-maker. We provide theoretical and experimental evidence that learning to defer improves upon the standard rejection learning paradigm, if models are intended to work as part of a larger system. We show that embedding a deferring model in a pipeline can improve the accuracy and fairness of the system as a whole. Experimentally, we simulate three scenarios where our model can defer judgment to external decision makers, echoing realistic situations where downstream decision makers are inconsistent, biased, or have access to side information. Our experimental results show that in each scenario, learning to defer allows models to work with users to make fairer, more responsible decisions.  Figure 1: A larger decision system containing an automated model. When the model predicts, the system outputs the model's prediction; when the model says PASS, the system outputs the decisionmaker's (DM's) prediction. Standard rejection learning considers the model stage, in isolation, as the system output, while learning-to-defer optimizes the model over the system output.

Rebuttal: Thanks for the rebuttal. I appreciate the scenarios you detailed about my concerns of limited applicability due to "overfitting" to a DM. I did not give detecting common trends as well as crowd sourcing enough thought during reading (maybe make these examples more explicit) and agree that they appear quite practical. Given that I liked the approach to begin with, I changed my score to a 6.   This paper formulates a two-stage decision making process, where one decision maker (an automated model) can either make the decision right away, or defer it to a second decision maker (typically human and with access to additional information, but potentially biased). The  main extension beyond rejection learning is that the automated model has access and adapts to the (human) decision maker's output during training, which allows not only to exploit the decision makers strengths (additional information), but also make up for its weaknesses (bias against subgroups, uncertainty on certain subgroups), in applications for fair decision making.  The paper is clearly structured, well written and very well motivated. Except for minor confusions about some of the math, I could easily follow and enjoyed reading the paper. As far as I know, the framework and particularly the application to fairness is novel. I believe the general idea of incorporating and adjusting to human decision makers as first class citizens of the pipeline is important for the advancement of fairness in machine learning. However, the framework still seems to encompass a rather minimal technical contribution in the sense that both a strong theoretical analysis and exhaustive empirical evaluation are lacking.  Moreover, I am concerned about the real world applicability of the approach, as it mostly seems to concern situations with a rather specific (but unknown) behavior of the decision maker, which typically does not transfer across DMs, needs to be known during training. I have trouble thinking of situations where sufficient training data, both ground truth and the DMs predictions, are available simultaneously. While the authors do a good job evaluating various aspects of their method (one question about this in the detailed comments), those are only two rather simplistic synthetic scenarios.  Because of the limited technical and experimental contribution, I heavy-heartedly tend to vote for rejection of the submission, even though I am a big fan of the motivation and approach.  Detailed Comments  - I like the setup description in Section 2.1. It is easy to follow and clearly describes the technical idea of the paper. - I have trouble understanding (the proof of) the Theorem (following line 104). You show that eq (6) and eq (7) are equal for appropriately chosen $\gamma_{defer}$. However, (7) is not the original deferring loss from eq (3). Shouldn't the result be that learning to defer and rejection learning are equivalent if for the (assumed to be) constant DM loss, $\alpha$ happens to be equal to $\gamma_{reject}$? In the theorem it sounds as if they were equivalent independent of the parameter choices for $\gamma_{reject}$ and $\alpha$. The main takeaway, namely that there is a one-to-one correspondence between rejection learning with cost $\gamma_{reject}$ and learning to defer with a DM with constant loss $\alpha$, is still true. Is there a specific reason why the authors decided to present the theorem and proof in this way? - The authors highlight various practical scenarios in which learning to defer is preferable and detail how it is expected to behave. However, this practicability seems to be heavily impaired by the strong assumptions necessary to train such model, i.e., availability of ground truth and DM's decisions for each DM of interest, where each is expected to have their own specific biases/uncertainties/behaviors during training. - What does it mean for the predictions \hat{Y} to follow an (independent?) Bernoulli equation (12) and line 197? How is p chosen, and where does it enter? Could you improve clarity by explicitly stating w.r.t. what the expectations in the first line in (12) are taken (i.e., where does p enter explicitly?) Shouldn't the expectation be over the distribution of \hat{Y} induced by the (training) distribution over X? - In line 210: The impossibility results only hold for (arguably) non-trivial scenarios. - When predicting the Charlson Index, why does it make sense to treat age as a sensitive attribute? Isn't age a strong and "fair" indicator in this scenario? Or is this merely for illustration of the method? - In scenario 2 (line 252), does $\alpha_{fair}$ refer to the one in eq (11)? Eq. (11) is the joint objective for learning the model (prediction and deferral) given a fixed DM? That would mean that the autodmated model is encouraged to provide unfair predictions. However, my intuition for this scenario is that the (blackbox) DM provides unfair decisions and the model's task is to correct for it. I understand that the (later fixed) DM is first also trained (semi synthetic approach). Supposedly, unfairness is encouraged only when training DM as a pre-stage to learning the model? I encourage the authors to draw the distinction between first training/simulating the DM (and the corresponding assumptions/parameters) and then training the model (and the corresponding assumptions/parameters) more clearly. - The comparison between the deferring and the rejecting model is not quite fair. The rejecting model receives a fixed cost for rejecting and thus does not need access to DM during training. This already highlights that it cannot exploit specific aspects (e.g., additional information) of the DM. On the other hand, while the deferring model can adaptively pass on those examples to DM, on which the DM performs better, this requires access to DM's predictions during training. Since DMs typically have unique/special characteristics that could vary greatly from one DM to the next, this seems to be a strong impairment for training a deferring model (for each DM individually) in practice?   While the adaptivity of learning to defer unsurprisingly constitutes an advantage over rejection learning, it comes at the (potentially large) cost of relying on more data. Hence, instead of simply showing its superiority over rejection learning, one should perhaps evaluate this tradeoff? - Nitpicking: I find "above/below diagonal" (add a thin gray diagonal to the plot) easier to interpret than "above/below 45 degree", which sounds like a local property (e.g., not the case where the red line saturates and has "0 degrees"). - Is the slight trend of the rejecting model on the COMPAS dataset in Figure 4 to defer less on the reliable group a property of the dataset? Since rejection learning is non-adaptive, it is blind to the properties of DM, i.e., one would expect it to defer equally on both groups if there is no bias in the data (greater variance in outcomes for different groups, or class imbalance resulting in higher uncertainty for one group). - In lines 306-307 the authors argue that deferring classifiers have higher overall accuracy at a given minimum subgroup accuracy (MSA). Does that mean that at the same error rate for the subgroup with the largest error rate (minimum accuracy), the error rate on the other subgroups is on average smaller (higher overall accuracy)? This would mean that the differences in error rates between subgroups are larger for the deferring classifier, i.e., less evenly distributed, which would mean that the deferring classifier is less fair? - Please update the references to point to the conference/journal versions of the papers (instead of arxiv versions) where applicable.  Typos  line 10: learning to defer ca*n* make systems... line 97: first "the" should be removed End of line 5 of the caption of Figure 3: Fig. 3a (instead of Figs. 3a) line 356: This reference seems incomplete?


Domain Adaptation by Using Causal Inference to Predict Invariant Conditional Distributions
An important goal common to domain adaptation and causal inference is to make accurate predictions when the distributions for the source (or training) domain(s) and target (or test) domain(s) differ. In many cases, these different distributions can be modeled as different contexts of a single underlying system, in which each distribution corresponds to a different perturbation of the system, or in causal terms, an intervention. We focus on a class of such causal domain adaptation problems, where data for one or more source domains are given, and the task is to predict the distribution of a certain target variable from measurements of other variables in one or more target domains. We propose an approach for solving these problems that exploits causal inference and does not rely on prior knowledge of the causal graph, the type of interventions or the intervention targets. We demonstrate our approach by evaluating a possible implementation on simulated and real world data.

Summary: this paper proposes a new approach to domain adaptation that relies on the identification of a separating feature set, conditional on which the distribution of a variable of interest is invariant under a certain intervention. The problem that the authors set out to solve is a vey ambitious and difficult one. I think that they are partially successful in their attempt, although I have a number of serious reservations. Detailed comments follow:   1) I found the paper generally very well written and clear. I come from a `potential outcomes’ school of causal inference, and my knowledge of DAGs is not extensive — even so, the authors made a reasonable effort to introduce all needed concepts and I was able to follow the details of the exposition. I believe this article to be accessible to most researchers who have a solid background in causal inference, even if their knowledge of DAGs is limited.  2) I appreciate the fact that the authors (mostly) avoided making grandiose claims about the reach of their method. They acknowledge some of its limitations openly. I also greatly appreciate the innate conservativeness of their algorithm (if we accept their assumption) which may refuse to make predictions if it cannot find a separating feature set.  3) If we accept the assumptions made by the authors, then although the basic idea is not groundbreaking (the authors acknowledge as much in footnote 4 on page 5), their contribution to identifying separating sets of features is original and interesting.  3) My first serious criticism is with the assumptions being made. First of all, I think that more space should have been devoted to discussing the assumptions. I understand that 10 pages doesn’t leave a lot of space, but the entire premise of this work rests on these assumptions. The Markov and Faithfulness assumptions being standard, it’s ok to not expand on them too much (although it never hurts to discuss an assumption). Assumptions 1 and 2 are another matter entirely — neither is standard and both are crucial to the rest of the paper. While I am roughly ok with Assumption 1, I have serious reservations about Assumption 2. Specifically, Assumption 2(ii) and Assumption 2(iii) seem to be assuming away most of the problem. At a minimum, the authors should provide (more than one) concrete example where both assumptions are credible. My main contention is that 2(ii) and 2(iii) assume a strong prior knowledge of the causal mechanisms in the system, and I cannot envision a scenario in which an individual with such knowledge would still have a need for domain adaptation.  4) My second serious criticism is with the practical aspect of this method. The authors acknowledge the computational complexity of the approach, but don’t discuss the details of this limitation. Concretely, how many variables can this method handle? Both examples in Section 3 deal with very few variables, far fewer than what would be required in most practical scenarios I can think of. The authors state in Section 4 that “more work remains to be done on the implementation side […] scaling up to more variables”. Do you have a sense for what the algorithmic complexity of the problem is? How likely is it that any significant scaling improvements (say handling ~50 variables) will occur soon? Do you have concrete ideas about how to get there?   Additional minor comments:  1) The first sentence of Example 1 is awkward. It’s a subordinate clause that isn’t attached to a main clause. 2) Same comment for the first sentence of Task 1. 3) I found Example 1 to be unclear. You should make it clear earlier that the example assumes $P(Y | X_1, C_1 = 0) = P(Y | X_1, C_1=1)$ instead of writing it at the end. 4) The labels of the plots in Figure 3 are way too small.


Dimensionally Tight Bounds for Second-Order Hamiltonian Monte Carlo
Hamiltonian Monte Carlo (HMC) is a widely deployed method to sample from highdimensional distributions in Statistics and Machine learning. HMC is known to run very efficiently in practice and its popular second-order "leapfrog" implementation has long been conjectured to run in d 1 4 gradient evaluations. Here we show that this conjecture is true when sampling from strongly log-concave target distributions that satisfy a weak third-order regularity property associated with the input data. Our regularity condition is weaker than the Lipschitz Hessian property and allows us to show faster convergence bounds for a much larger class of distributions than would be possible with the usual Lipschitz Hessian constant alone. Important distributions that satisfy our regularity condition include posterior distributions used in Bayesian logistic regression for which the data satisfies an "incoherence" property. Our result compares favorably with the best available bounds for the class of strongly log-concave distributions, which grow like d 1 2 gradient evaluations with the dimension. Moreover, our simulations on synthetic data suggest that, when our regularity condition is satisfied, leapfrog HMC performs better than its competitors -both in terms of accuracy and in terms of the number of gradient evaluations it requires.

This paper looks at the "running time" of HMC.   It seems one of the prime contributions of this work is a relaxation of a condition given in prior work (which this reviewer has not read) to accommodate a more general regularity structure on the target. Namely, Assumption 1 in the present work is a kind of Lipschitz assumption on the Hessian but based on some apparently weaker norm. Under this condition, and importantly strongly concave targets, the authors look at the number of "gradient evaluations" required in HMC.   I think this is interesting from the theoretical point of view, and perhaps useful - albeit the nature of the problems addressed are not so interesting alone and thus the practical impact of this work is limited I would guess.  I have some concerns and I don't think the paper is really ready for publication in NIPS. These are minor in some sense.  Firstly, the idea of "run time" from the start is pretty vague. It seems it is only in Theorem 1 that running time is defined explicitly as "the number gradient evaluations". Even this is perhaps not completely transparent.  The Algorithm 1 has a fixed running time determined by i_{max}, T, \eta, and it is independent of d in terms of the number of gradient evaluations. So it seems the notation between this and, for example, Theorem 1, inconsistent.   Line 107 - why do I have only p_1 and p_2, I am confused?  What is the "^\prime" symbol, surely it is used for multiple purposes or something is wrong here?  What is the norm used in the Theorem 1 to measure X_{i_{max}} and Y? Perhaps this is mentioned somewhere but I missed it. Same question in the definition of a warm start and in the Section 5. This seems rather critical? And then what is a "high probability" in Theorem 1. I don't really follow this theorem in a rigorous sense nor its proof outline - I think there are some problems with a lack of defining an appropriate distance here.  I didn't read the supplementary material.  Personally, for this reviewer, the formatting with a lot of "in-line" notation made the paper a little hard and frustrating to read. But this is certainly not a criticism for rejection - rather just a comment that may aid the authors in presenting a revision.  In summary, I think there is a lot of potential with this result and it would be of theoretical interest to people working in this particular direction in this area. However, from my reading I didn't get the feeling it was polished enough and perhaps suitable enough for NIPS although I am happy to be corrected.   ----  Added after author rebuttal:  I am happy with the changes. I don't think you need the "_2" subscript on the norm if you define it - previously you had the Euclidean norm with "_2" in some places and someplace without. For notation, there are a lot of places I don't see the notation as necessary or even understandable. For example, the \dagger in Algorithm 1 seems completely unnecessary. As does the \tilde in the warm start definition. I don't follow the \tilde{O} notation - I guess this is to distinguish it from O^\star which only captures the dimension terms, but then just plain O( ) notation is also used so the  \tilde{O} is unknown to me (perhaps its separate but standard notation I should know?). Anyway, these are just minor issues really; but I saw a lot of probably unnecessary notation which could probably be worked on in later versions (e.g. the superscript notation later on in the leapfrog integrator section (7 in the arXiv) is a bit much.  My main concern is the length and the suitability for NIPS in terms of acceptance. I like the idea of the result. But unadjusted HMC is not really a practically chosen algorithm. And the length of the supplementary material - which I tried in the rebuttal period to follow but had to stop due to time constraints - is too long. I don't believe supplementary material's purpose is to contain the bulk of an article, which is the case when the main contribution of the article is a theorem and the supplementary contains the proof. So while I think the paper has merit, I think it needs to benefit from a proper review of the entirety of the technical details in a more suitable venue (e.g. some probability journal). I note, for example, similar studies on unadjusted Langevin appear in AAP for example.


Optimization of Smooth Functions with Noisy Observations: Local Minimax Rates
We consider the problem of global optimization of an unknown non-convex smooth function with noisy zeroth-order feedback. We propose a local minimax framework to study the fundamental difficulty of optimizing smooth functions with adaptive function evaluations. We show that for functions with fast growth around their global minima, carefully designed optimization algorithms can identify a near global minimizer with many fewer queries than worst-case global minimax theory predicts. For the special case of strongly convex and smooth functions, our implied convergence rates match the ones developed for zeroth-order convex optimization problems. On the other hand, we show that in the worst case no algorithm can converge faster than the minimax rate of estimating an unknown functions in 8 -norm. Finally, we show that non-adaptive algorithms, although optimal in a global minimax sense, do not attain the optimal local minimax rate.

This paper studies local minimax bounds for noisy optimization of smooth functions. In the traditional setting of global optimization, the algorithmic goal is to design an adaptive procedure to find an approximate (in this case, global) optimum of an unknown function f, given access to noisy evaluations  at feasible points. This notion of complexity might be too rough to understand the difficulty of the problem, so it is proposed to study the local minimax complexity, when the algorithm may have additional information on how close is the objective to a fixed function f_0. In practice, the algorithm doesn't need to know f_0, but this parameterization serves as an instance-dependent notion of complexity. This paper additionally considers how the regularity of the function (given by its Holder continuity) improves the complexity, as it is well-known that for Lipschitz functions adaptivity does not help.   The main contributions of the paper can be summarized as follows: 1. An adaptive algorithm whose optimization error decreases at a rate R_n(f_0)=\tildeO(n^{-\alpha/[2\alpha+d-\alpha\beta]}), where \alpha is the Holder exponent and beta is a parameter related to the volume growth of level sets of f_0. This algorithm does not need to know f_0 in advance.  2. A local minimax lower bound matching the previous upper bound up to polylogs, for the case when even f_0 is known in advance. 3. A lower bound of Omega(n^{-\alpha/[2\alpha+d]}) for nonadaptive algorithms, exposing a substantial gap between adaptive and nonadaptive methods. 4. For the case when f_0 is strongly convex they show a local minimax rate of \tilde O(1/\sqrt{n}), and a stricly stronger lower bound when f_0 is merely convex.  The paper is quite interesting, and looks mathematically solid. I didn't have the chance to carefully verify all proofs, but they seem correct.  Detailed comments:  1. The setup of the paper seems comparable to the one of approximately convex optimization (see e.g.  https://papers.nips.cc/paper/5841-information-theoretic-lower-bounds-for-convex-optimization-with-erroneous-oracles or https://papers.nips.cc/paper/6576-algorithms-and-matching-lower-bounds-for-approximately-convex-optimization.pdf). This setting is analog to the one proposed in this paper, but without the extra noise in function evaluations. In particular, they show a phase transition of epsilon_n(f_0) (for some f_0) where no polynomial number of adaptive queries can reach an approximate solution.  Can the results of the paper be translated to this zero noise setting?  -------------- After Rebuttal --------------  I want to thank the authors for clarifying the differences w.r.t. approximate convex optimization. It is very interesting how these subtleties change complexity bounds. As some other reviewer suggested, it is important that the authors also include the dimension dependence in the minimax rates.


How Many Samples are Needed to Estimate a Convolutional Neural Network?
A widespread folklore for explaining the success of Convolutional Neural Networks (CNNs) is that CNNs use a more compact representation than the Fullyconnected Neural Network (FNN) and thus require fewer training samples to accurately estimate their parameters. We initiate the study of rigorously characterizing the sample complexity of estimating CNNs. We show that for an m-dimensional convolutional filter with linear activation acting on a d-dimensional input, the sample complexity of achieving population prediction error of ✏ is r Opm{✏ 2 q 2 , whereas the sample-complexity for its FNN counterpart is lower bounded by ⌦pd{✏ 2 q samples. Since, in typical settings m ! d, this result demonstrates the advantage of using a CNN. We further consider the sample complexity of estimating a onehidden-layer CNN with linear activation where both the m-dimensional convolutional filter and the r-dimensional output weights are unknown. For this model, we show that the sample complexity is r O`pm`rq{✏ 2˘w hen the ratio between the stride size and the filter size is a constant. For both models, we also present lower bounds showing our sample complexities are tight up to logarithmic factors. Our main tools for deriving these results are a localized empirical process analysis and a new lemma characterizing the convolutional structure. We believe that these tools may inspire further developments in understanding CNNs.

The authors consider the number of samples needed to achieve an error epsilon in the context  of learning an m-dimensional convolutional filter as well as one followed by a linear projection. This is motivated by a desire to rigorously understand the empirical success of CNNs.  This paper seems technically correct, yet I believe the setting is very far from real CNNs to the point where it’s not clear if the results will be impactful. The authors only consider a linear convolution layer, which corresponds to a wiener filtering-like operation according to their model, for removing noise for estimating the label. My concern is the motivation, the novelty and the assumptions.  For the assumptions, I am not sure that the non degeneracy assumption (A3) is that easy to constraint. Just  whitening looks complex, for example in the case of image: either one could use a basis a priori such as a wavelet basis but then this operation is quite complex wrt the tools used in this paper; a second option would be to whiten directly the signal that might be approximative somehow for very large signals  For novelty, the authors relate their work to relevant and very recent work on understanding and characterizing deep learning, however I am surprised there is no discussions of any classic signal processing/adaptive filters, the linear restriction in the model and the squared loss gives me a sense that there is a literature from signal processing not considered. In general the lack of literature before 2015 seems odd. I'd like the authors to relate their work to classic signal processing which considers in detail learning linear filters. Additionally I would suggest a relationship to the classic results such as (Barron, 1994) on one-hidden-layer networks.  Regarding the motivation, while I agree the results is interesting, since as the authors admit the result is quite distant from the non-linear activation case and  it is unclear to me if this result will be useful.


Efficient Algorithms for Non-convex Isotonic Regression through Submodular Optimization
We consider the minimization of submodular functions subject to ordering constraints. We show that this potentially non-convex optimization problem can be cast as a convex optimization problem on a space of uni-dimensional measures, with ordering constraints corresponding to first-order stochastic dominance. We propose new discretization schemes that lead to simple and efficient algorithms based on zero-th, first, or higher order oracles; these algorithms also lead to improvements without isotonic constraints. Finally, our experiments show that non-convex loss functions can be much more robust to outliers for isotonic regression, while still being solvable in polynomial time.

This paper considers the problem of continuous submodular minimization with box constraints and extra isotonic constraints, motivated by nonconvex isotonic regression. The paper gives an elegant algorithm for solving these constrained problems, by leveraging isotonic structure that is already present in the algorithms for continuous submodular minimization with box constraints from [1]. Beyond generalizing the approach of [1] to these constrained problems, the paper further shows that an extra smoothness condition on the objective yields better dependence the suboptimality criterion, for free, than thought before. Higher order smoothness can yield even better dependence. Finally, the paper shows the practical relevance of these techniques via a seemingly simple robust isotonic regression problem where natural convex formulations fail. A different (student t) model succeeds, but the objective is nonconvex; heuristics give poor performance, while the techniques from the paper solve the problem optimally and yield good performance on the regression task.  Overall I think the paper has clean, well-executed theory and a motivating practical problem. Continuous submodular optimization is an important topic relevant to the NIPS community, as it presents a class of tractable nonconvex problems of use in machine learning applications (like nonconvex isotonic regression). Within this area of work, the paper is original and significant: while continuous submodular _maximization_ has seen much activity and many positive results in recent years, continuous submodular _minimization_ (this paper) seems more elusive. (beyond this paper I can only think of Bach '18 [1] and Staib and Jegelka '17, versus a multitude of maximization papers). This paper grants to us a new class of tractable constrained nonconvex problems.  The paper is also clearly written and was a joy to read.  I do have a few minor concerns/questions -- these are all practical considerations as I think the math is clean and not fluffy: - what is the wall-clock runtime like? - is it obvious that majorization-minimization is the most competitive alternative? (e.g. vs running projected gradient descent from random initialization) - I find the practical benefit of extra Taylor expansion terms in the approximation somewhat suspect -- it seems like you have to do a lot of extra work (both algorithmically and just by having to query q different derivative oracles) to get the improvement in epsilon dependence. It seems the fancy discretization was not really even used in the experiments (in "Non-separable problems" the authors compare the quality of approximations, but they seem to just use the usual discretization for the main regression experiments and just note that smooth objectives automatically get better approximation). Thus I am skeptical that "the new discretization algorithms are faster than the naive one." However the observation that smoothness implies better speed for free is still important.  Overall, I think this is a solid paper with significant theoretical and algorithmic contribution, and it is also well-written. Some aspects of the experiments could be strengthened, but still I think this is a clear accept.   minor comments: - the SDP relaxation is based on [15] which should be cited in the main text (it is cited in the appendix) - the point that \hat H is not technically submodular but we can still approximately solve probably should be stated more formally


Cooperative Holistic Scene Understanding: Unifying 3D Object, Layout, and Camera Pose Estimation
Holistic 3D indoor scene understanding refers to jointly recovering the i) object bounding boxes, ii) room layout, and iii) camera pose, all in 3D. The existing methods either are ineffective or only tackle the problem partially. In this paper, we propose an end-to-end model that simultaneously solves all three tasks in realtime given only a single RGB image. The essence of the proposed method is to improve the prediction by i) parametrizing the targets (e.g., 3D boxes) instead of directly estimating the targets, and ii) cooperative training across different modules in contrast to training these modules individually. Specifically, we parametrize the 3D object bounding boxes by the predictions from several modules, i.e., 3D camera pose and object attributes. The proposed method provides two major advantages: i) The parametrization helps maintain the consistency between the 2D image and the 3D world, thus largely reducing the prediction variances in 3D coordinates. ii) Constraints can be imposed on the parametrization to train different modules simultaneously. We call these constraints "cooperative losses" as they enable the joint training and inference. We employ three cooperative losses for 3D bounding boxes, 2D projections, and physical constraints to estimate a geometrically consistent and physically plausible 3D scene. Experiments on the SUN RGB-D dataset shows that the proposed method significantly outperforms prior approaches on 3D object detection, 3D layout estimation, 3D camera pose estimation, and holistic scene understanding.

I have read the rebuttal and I am satisfied with the authors responses. I am happy accepting the paper.  ----------- This paper proposes to take an image as input and output: (a) camera pose (b) room layout as a 3D box, and (c) for every object, the 3D box. The method consists of a global network that predicts (a,b) and a local network to predict (c). In addition to the standard losses, the paper proposes physical losses that entangle the global and local predictions by using the predicted camera pose in the 3D bounding box loss and a 2D projected bounding box, as well as a loss that ensures that each object is inside the room.  I like many aspects of the submission but I am concerned that the experiments do not clearly support the claims of the paper.  + The parametrization of the 3D box is clever and, while it is simple geometry, I do not think I have seen it before. It quite cleanly refactors the estimation of the coordinates so that it's easy to actually get the box to align with the 2D box.  + Forcing the two networks to work together makes a lot of sense and the particular formulation of how the two should work together is clean: factoring it so that there's only one camera pose so that the objects are not independently implicitly predicting a camera pose is quite nice. Similarly, the non-intersection loss is quite nice.   Experimentally, the paper is tricky to evaluate -- there isn't a lot of work on this task, so it's difficult to do meaningful comparisons. 3DGP and Hedau et al.'s VP estimator are very much out of date, leaving IM2CAD and ablations. My overall concern with the experiments is that I'm not convinced the paper clearly demonstrates the benefits of the system.  -It's not clear that the "individual" vs "cooperative" result is explained by the cooperative estimation: estimating 3D pose from a cropped 2D bounding box is a difficult problem, and so it's not surprising that the setting that can see the full image outperforms the network that can only see cropped bounding boxes. Indeed this was observed in Tulsiani et al. CVPR 2018, in which not providing the context of the full image for estimating object pose led to performance losses.  -The comparison to IM2CAD is difficult to assess. It's clear that the paper improves on it by a little bit in terms of holistic understanding and by a lot in terms of the layout estimation, but it's unclear where that performance gain comes from. The base architectures change from VGG16 -> Resnet34 for object detection (and maybe also the object detection pipeline) and VGG16 + hand-crafted optimization -> Direct estimation via resnet34 for layout estimation.   -The ablative analysis does not clearly support the importance of most of the proposed constraints. For example, S3 ablation (without L_PHY), the drop without L_PHY is not particularly big, mainly <0.5% and sometimes not including L_PHY improves things (precision, for instance). This could easily be explained by plenty of other effects like the natural variance of training runs. Similarly, in the S2 ablation, getting rid of the 2D projection loss improves things dramatically.   Smaller stuff that needs to be fixed but which does not impact my review a lot:  -The very first sentence is misciting Oliva 2005. Oliva 2005 has nothing to do with getting full, metric 3D out of an image and indeed, comments "Common to these representations is their holistic nature: the structure of the scene is inferred, with no need to represent the shape or meaning of the objects". -SoftNMS is not a detector. SoftNMS is a post-detection heuristic used to improve the output of a detector. The paper should identify which detector was used. - Missing citations. The paper is missing an important line of work from the 2010-2014 era of joint optimization of objects and layout, e.g., Lee, et al. NIPS 2010; Schwing et al. Box in the Box ICCV 2013. (which is notably not slow) -Paragraph discussing the ablative analysis S1/S2/S3 has the order of S2/S3 flipped.


Learning Hierarchical Semantic Image Manipulation through Structured Representations
Understanding, reasoning, and manipulating semantic concepts of images have been a fundamental research problem for decades. Previous work mainly focused on direct manipulation on natural image manifold through color strokes, keypoints, textures, and holes-to-fill. In this work, we present a novel hierarchical framework for semantic image manipulation. Key to our hierarchical framework is that we employ structured semantic layout as our intermediate representation for manipulation. Initialized with coarse-level bounding boxes, our structure generator first creates pixel-wise semantic layout capturing the object shape, object-object interactions, and object-scene relations. Then our image generator fills in the pixel-level textures guided by the semantic layout. Such framework allows a user to manipulate images at object-level by adding, removing, and moving one bounding box at a time. Experimental evaluations demonstrate the advantages of the hierarchical manipulation framework over existing image generation and context hole-filing models, both qualitatively and quantitatively. Benefits of the hierarchical framework are further demonstrated in applications such as semantic object manipulation, interactive image editing, and data-driven image manipulation.

In this paper a new method for image manipulation is proposed. The proposed method incorporates a hierarchical framework and provides both interactive and automatic semantic object-level image manipulation. In the interactive manipulation setting, the user can select a bounding box where image editing for adding and removing objects will be applied. The proposed network architecture consists of a foreground output stream which produces the predictions on binary object mask and a background output stream for producing per-pixel label maps. As the result, the proposed image manipulation method generates output image by filling in the pixel-level textures guided by the semantic layout. The manipulation operations are only ‘insertion’ and ‘deletion’ and other operations such as change of color and style are not supported.   The paper is well-written and clear; Figures and qualitative examples convey the idea. As to investigate the effectiveness of the single-stream vs two-stream using ground truth or predicted layout ablation study is conducted.   The performance of the proposed method is compared with prior works of [21] and [32] and human study is conducted via AMT. Based on the results summarized in Table 2, the proposed method has higher SSIM and obtains higher human evaluation compared to the prior works and baseline. However, as for the the segmentation accuracy the prior work of [32] obtains higher accuracy.   While the proposed method is interesting, the evaluation is only conducted on outdoor street images and only two selected examples are shown in the indoor scenes in supplemental. In light of the high quality performance of the previous works on image generation such as [4] and [41], image generation in the context of indoor scenes is more subject to evaluation as such indoor scenes involve more visual diversity and contain larger set of object categories for manipulation.  Most of the examples shown in the paper demonstrate the quality of adding or removing cars and person in street scenes which have high biases in terms of pose and shape. The question that remains is that how much object diversity (in terms of both category and appearance differences) such image manipulation technique can handle.


Beyond Grids: Learning Graph Representations for Visual Recognition
We propose learning graph representations from 2D feature maps for visual recognition. Our method draws inspiration from region based recognition, and learns to transform a 2D image into a graph structure. The vertices of the graph define clusters of pixels ("regions"), and the edges measure the similarity between these clusters in a feature space. Our method further learns to propagate information across all vertices on the graph, and is able to project the learned graph representation back into 2D grids. Our graph representation facilitates reasoning beyond regular grids and can capture long range dependencies among regions. We demonstrate that our model can be trained from end-to-end, and is easily integrated into existing networks. Finally, we evaluate our method on three challenging recognition tasks: semantic segmentation, object detection and object instance segmentation. For all tasks, our method outperforms state-of-the-art methods.

The paper proposes to learn graph representations from visual data via graph convolutional unit (GCU). It transforms a 2D feature maps extracted from a neural network to a sample-dependent graph, where pixels with similar features form a vertex and edges measure affinity of vertices in a feature space. Then graph convolutions are applied to pass information along the edges of the graph and update the vertex features. Finally, the updated vertex features are projected back to 2D grids based on the pixel-to-vertex assignment. GCU can be integrated into existing networks allowing end-to-end training and capturing long-range dependencies among regions (vertices). The approach is evaluated on ADE20k for semantic segmentation task building upon PSPNet with ResNet-50/100 backbone and on MS COCO for object detection and instance segmentation tasks using Mask RCNN architecture. The method consistently improves over considered baselines (PSPNet, Mask RCNN) across different tasks.  Pros:  - A novel model has been presented for learning graph representations from 2D data. - The proposed method seems to be effective and efficient, allows end-to-end learning and can be integrated into existing networks.  Cons:  - The number of vertices has to be pre-specified. For efficiency reasons the number of vertices is kept small (set up to max 32 in the experiments). This results in very coarse, overly smooth outputs and missed finer details/objects (lines 240, 294), and will not be suitable for the applications that require pixel-accurate predictions. - No ablation study is provided. The paper lacks analysis of contributions of different steps to the performance, e.g. how much does initialization by clustering and regularization described in sec. 3.3. affect the final results? Also many design choices are not justified/analyzed in the paper, e.g. number of GCUs and number of vertices. Why were different number of vertices chosen for different task? How does the number of GCUs / number of vertices influence the results?  - The learnt groupings of pixels does not form meaningful semantic regions.  Minor comments/typos:  - Table 3,4 should be changed to Figure 3,4 - Line 293: mode -> modes  Overall, the paper presents a novel approach for learning graph representations from 2D data, which seems to be effective and flexible. However, it lacks the analysis and ablation study of different components. This work does not provide any insights on the design choices and the reader would have troubles transferring the proposed method to other tasks or new architectures. In addition, the proposed approach has a shortcoming that the number of vertices has to be pre-specified which negatively influences the quality of output predictions.


Using Trusted Data to Train Deep Networks on Labels Corrupted by Severe Noise
The growing importance of massive datasets used for deep learning makes robustness to label noise a critical property for classifiers to have. Sources of label noise include automatic labeling, non-expert labeling, and label corruption by data poisoning adversaries. Numerous previous works assume that no source of labels can be trusted. We relax this assumption and assume that a small subset of the training data is trusted. This enables substantial label corruption robustness performance gains. In addition, particularly severe label noise can be combated by using a set of trusted data with clean labels. We utilize trusted data by proposing a loss correction technique that utilizes trusted examples in a data-efficient manner to mitigate the effects of label noise on deep neural network classifiers. Across vision and natural language processing tasks, we experiment with various label noises at several strengths, and show that our method significantly outperforms existing methods.

The submission describes a training methodology when you are given a dataset partitioned into a trusted and untrusted dataset. In the abstract and introduction, the authors justify the need for such a method and give a brief overview of the related work. They call their method ‘Gold Loss Correction’ (GLC).  The algorithm describes approximating the probability of an untrusted label given an input by training some classifier with softmax output and then using this model alongside the trusted dataset to construct a matrix of corruption probabilities. They do this by approximating the corruption probabilities by simply measuring the likelihood a model will classify some class from the partition of the dataset that is without doubt of some true class. Ofcourse this is non-normalized, so they mention trying both a normalized (The corruption matrix becomes a confusion matrix) and the non-normalized version.  In either case they use the matrix to weight the output of a newly trained classifier to consider this likelihood. The results show that the normalized version performs slightly worse, and so GLC requires no normalization.  They compare this approach to other Training procedures that account for untrusted data, but only one other that utilizes a trusted set. The comparison to Forward Loss Correction has this inherent bias because it does not utilize the important information that is the foundation of this work. They attempt to remedy this by adjusting the Forward Loss Corrections corruption matrix with the identity on the trusted component of each dataset. This ended up not assisting, which they hypothesize is due to variance introduced in their setup. This specific differential doesn’t seem clear, and I feel further investigation into why the adjustment had little effect may shed light onto strengths and weaknesses of their algorithm.  The results show a strong advantage to using GLC over the other algorithms in most cases (except for one against distillation in a niche setting) as well as it being robust and data-efficient. Though it should be mentioned that GLC works on a baseline assumption that the trusted and untrusted labels are conditionally independent on x. They backup this assumption by saying most systems that induce noise usually do not know the true label which is valid, but it would be interesting to see the comparative results in cases where this assumption is broken in examples like worst-case adversarial settings.  The paper is well-written, and they describe both the related work and their procedures in detail. The concept underlying their innovative training methodology is not purely original because they are using the same foundation as in Patrini et al 2017, but then originally extend it to utilize trusted component of the dataset.


Scaling Gaussian Process Regression with Derivatives
Gaussian processes (GPs) with derivatives are useful in many applications, including Bayesian optimization, implicit surface reconstruction, and terrain reconstruction. Fitting a GP to function values and derivatives at n points in d dimensions requires linear solves and log determinants with an n(d + 1) × n(d + 1) positive definite matrix -leading to prohibitive O(n 3 d 3 ) computations for standard direct methods. We propose iterative solvers using fast O(nd) matrix-vector multiplications (MVMs), together with pivoted Cholesky preconditioning that cuts the iterations to convergence by several orders of magnitude, allowing for fast kernel learning and prediction. Our approaches, together with dimensionality reduction, enables Bayesian optimization with derivatives to scale to high-dimensional problems and large evaluation budgets.

The paper investigates the application of the recent SKI and SKIP GP methodologies in the case where additional gradient information is available. They derive the necessary mathematical background, discuss numerical strategies to make the computations robust efficient and efficient and evaluate extensively on established synthetic and real benchmark problems.  A weakness of the paper is the somewhat entangled discussion of dimensionality reduction strategies. D-SKI scales with 6^d, which is approximately 60 million for d=10 and roughly half a trillion for d=15. These numbers demonstrate that D-SKIP will not be optional for many problems typically considered with BO. From the presented experiments, it is unclear to me how the "online dimension reduction" would influence the BO algorithm. Additionally, I find the description in lines 205-211 too vague to be able to understand the experimental evaluation. I will not consider this a problem for the submission as long as the authors hold up their promise of providing the experiment code. (As a minor side note: you might want to remind the reader of reference [25] at this point in the text.)  Nevertheless, a strength of the paper is its holistic description and evaluation of the proposed method. The background and the newly extended analysis seem straight forward, so the authors make good use of the page limit to discuss important practical strategies like preconditioning and provide a principled evaluation. Their basic experiments are run on common test functions which could in principle be used for any desired mathematical analysis. I agree with their rationale on terrain and surface reconstruction and the experiments are convincing.  In terms of significance, I am hesitant to provide a clear judgement. Personally, I have not worked on surface reconstruction before, so I don't know how big this application area is. In terms of originality, this seems to be a clear continuation from early work on SKI and SKIP. As a consequence, I am not surprised by this work, but I can imagine that providing this detailed derivation (and, possibly, the code) was still a highly non-trivial issue, in particular with respect to the pre-conditioning evaluation and experiments.  In conclusion: in my opinion, this is a solid piece of research, whose lack of ground-braking novelty is made up by the good presentation and experimental evaluation.  ---- Post-rebuttal update: Thanks for the authors for the clarifications. I very much appreciated your comment regarding the computational complexity of D-SKIP and I encourage the authors to put this information into the paper as well. I am willing to follow your arguments regarding significance and novelty, and I will support this view in the discussion. I have not updated my score of the overall review as this score already reflected my ignorance in terms of significance "in dubio pro reo".


Temporal alignment and latent Gaussian process factor inference in population spike trains
We introduce a novel scalable approach to identifying common latent structure in neural population spike-trains, which allows for variability both in the trajectory and in the rate of progression of the underlying computation. Our approach is based on shared latent Gaussian processes (GPs) which are combined linearly, as in the Gaussian Process Factor Analysis (GPFA) algorithm. We extend GPFA to handle unbinned spike-train data by incorporating a continuous time point-process likelihood model, achieving scalability with a sparse variational approximation. Shared variability is separated into terms that express condition dependence, as well as trial-to-trial variation in trajectories. Finally, we introduce a nested GP formulation to capture variability in the rate of evolution along the trajectory. We show that the new method learns to recover latent trajectories in synthetic data, and can accurately identify the trial-to-trial timing of movement-related parameters from motor cortical data without any supervision.

-- Paper Summary   This paper tackles the problem of identifying patterns in spiking population activity of neurons by considering latent structure and trajectories. This is a particularly challenging problem due to potential time misalignment between multiple trials. Additionally, the various time series are generally assumed to be independent while there are also issues with tractability due to linear algebraic operations involving a TxT kernel matrix. This work tackles the latter issue by introducing the sparse variational approximation within the GPFA model, and addresses shared latent structure across trials (or even partitions of trials) by exploiting a nested GP architecture.   -- Originality + Significance  The variational sparse approximation to Gaussian process inference has become the go-to approach for solving tractability issues with GPs without compromising on performance. Although the approximation in itself has been used and applied extensively in other papers, to the best of my knowledge, this paper is the first in applying it to the GPFA model in particular.  In combination with the point process likelihood being considered, the resulting model is sufficiently different from existing approximate models based on sparse VI while also incorporating additional elements (explored in Section 4 of the paper).  A recent paper discussing simultaneous modelling of data and temporal alignment, ‘Gaussian Process Latent Variable Alignment Learning (Kazlauskaite et al)’, seems to be related in terms of scope. In fact, the generalisation to GPLVMs is listed in the concluding remarks of the paper under review. Although there is a clear distinction between the two papers, particularly with regards to the likelihood considered, any similarity to this work should be duly discussed.  It might be interesting to draw a connection to papers such as ‘Collaborative Multi-output Gaussian processes (Nguyen & Bonilla)’, where inducing points are shared across multiple latent functions. It is not clear whether the inducing points considered in this paper are unique to each latent function or shared between them, although I believe it’s the former. If not, this could possibly contribute towards further improving scalability.   -- Quality  The model is clearly detailed in Sections 3 and 4. As highlighted earlier, the sparse variational approximation is fairly standard; however, the authors do not limit themselves to the base approximation, but also consider several extensions which make the paper more complete and well-rounded.   Once all aspects of the model are described, I encourage the authors to summarise the overall complexity of the model with regards to how many trials, conditions, time points etc are considered in a given experiment.   -- Writing and Clarity  The paper is very well-written and a pleasure to read - I did not spot any typos while reading it. The figures are well-captioned and easy to interpret. Corresponding discussion of the figures in the text is also appropriate.  Some minor comments: - While the ‘macaque’ species might be well-known in the biology domain, I would suggest adding a footnote or small parenthesis explaining the term first. It is currently introduced in L57 without much context. - Similarly, it would be good to explain what is ‘innovations noise’ (L198)  - ‘Where’ in L121 could be changed to lower case  The content provided in the supplementary material complements the main text with more detailed proofs and figures. I found the balance between the two to be just right.   -- Overall Recommendation  This is a well-written paper with clearly established goals that are both individually and jointly investigated in this paper.  The contributions themselves are fairly modest and application-specific; nonetheless, state-of-the-art results are obtained in the problem setting being investigated, while the majority of the modelling choices are sufficiently general to be of interest to the broader NIPS community.  ------------  -- Post-review  Thank you for replying to my queries, in particular for expanding on the relation of this work to the indicated papers and detailing the overall complexity of the method. I believe that incorporating this discussion into the main paper (along with other minor fixes) will only elevate what is already a very good piece of work!