Decoupling "when to update" from "how to update"
Deep learning requires data. A useful approach to obtain data is to be creative and mine data from various sources, that were created for different purposes. Unfortunately, this approach often leads to noisy labels. In this paper, we propose a meta algorithm for tackling the noisy labels problem. The key idea is to decouple "when to update" from "how to update". We demonstrate the effectiveness of our algorithm by mining data for gender classification by combining the Labeled Faces in the Wild (LFW) face recognition dataset with a textual genderizing service, which leads to a noisy dataset. While our approach is very simple to implement, it leads to state-of-the-art results. We analyze some convergence properties of the proposed algorithm.

Summary
The paper proposes a meta algorithm for training any binary classifier in a manner that is robust to label noise. A model trained with noisy labels will overfit them trained for too long. Instead, one can train two models at the same time, initialized at random, and update by disagreement: the updates are performed only when the two models' prediction differ, a sign that they are still learning from the genuine signal in the data (not the noise); and instead, defensively, if the models agree on their predictions and the respective ground truth label is different, they should not perform an update, because this is a sign of potential label noise. A key element is the random initialization of the models, since the assumption is that the two should not give the same prediction unless they are close to converge; this fits well with deep neural networks, the target of this work.
The paper provides a proof of convergence in the case of linear models (updated with perceptron algorithm and in the realizable case) and a proof that the optimal model cannot be reach in general, unless we resort to restrictive distributional assumptions (this is nice since it also shows a theoretical limitation of the meta-algorithm). The method works well in practice in avoiding to overfit labels noise, as shown by experiments with deep neural networks applied to gender classification on the LFW dataset, and additional tests with linear models and on MNIST.
Comments
The paper is very well written: the method is introduced intuitively, posed in the context of the relevant literature, proven to be sound in a simple theoretical setting and shown to be effective on a simple experimental set up, in realistic scenario with noise. Additionally, this work stands out from the large set of papers on the topic because of its simplicity and the potential of use in conjunction with others methods.
Proofs are easy to follow and seem flawless. Experimental results are promising on simple scenarios but will need future work to investigate the robustness and effectiveness on at scale and in multi-class  --- although I don't consider this a major issue because the paper is well balanced between theory and practice.
Also, do you find any interesting relation with boosting algorithms? In particular I am referring to "The strenght of weak learnability" by R. Schapire, that introduced a first form (although impractical) of boosting of weak classifier. The algorithm presented back then uses a form of "update by disagreement" for boosting, essentially training a third model only on data points classified the same way by the former two models.
Minors
25 -> better
107 -> probabilistic
183 -> algorithm


Lower bounds on the robustness to adversarial perturbations
The input-output mappings learned by state-of-the-art neural networks are significantly discontinuous. It is possible to cause a neural network used for image recognition to misclassify its input by applying very specific, hardly perceptible perturbations to the input, called adversarial perturbations. Many hypotheses have been proposed to explain the existence of these peculiar samples as well as several methods to mitigate them, but a proven explanation remains elusive. In this work, we take steps towards a formal characterization of adversarial perturbations by deriving lower bounds on the magnitudes of perturbations necessary to change the classification of neural networks. The proposed bounds can be computed efficiently, requiring time at most linear in the number of parameters and hyperparameters of the model for any given sample. This makes them suitable for use in model selection, when one wishes to find out which of several proposed classifiers is most robust to adversarial perturbations. They may also be used as a basis for developing techniques to increase the robustness of classifiers, since they enjoy the theoretical guarantee that no adversarial perturbation could possibly be any smaller than the quantities provided by the bounds. We experimentally verify the bounds on the MNIST and CIFAR-10 data sets and find no violations. Additionally, the experimental results suggest that very small adversarial perturbations may occur with non-zero probability on natural samples.

This paper introduces lower bounds on the minimum adversarial perturbations that can be efficiently computed through layer-wise composition.
The idea and the approach is timely, and addresses one of the most pressing problems in Machine Learning. That being said, my main criticism is that the bounds are too loose: the minimum adversarials found through FGSM are several orders of magnitude larger then the estimated lower bounds. That might have two reasons: for one the lower bounds per layer might not be tight enough, or the adversarials found with FGSM are simply to large and not a good approximation for the real minimum adversarials perturbation. To test the second point, I’d encourage the authors to use better adversarial attacks like LBFGS or DeepFool (e.g. using the recently released Python package Foolbox which implements many different adversarial attacks). Also, the histograms in Figure 2&3 are difficult to compare. A histogram with the per-sample ratio between adversarial perturbation and lower bound would be more enlightening (especially once the bounds get tighter).


Scalable Model Selection for Belief Networks
We propose a scalable algorithm for model selection in sigmoid belief networks (SBNs), based on the factorized asymptotic Bayesian (FAB) framework. We derive the corresponding generalized factorized information criterion (gFIC) for the SBN, which is proven to be statistically consistent with the marginal log-likelihood. To capture the dependencies within hidden variables in SBNs, a recognition network is employed to model the variational distribution. The resulting algorithm, which we call FABIA, can simultaneously execute both model selection and inference by maximizing the lower bound of gFIC. On both synthetic and real data, our experiments suggest that FABIA, when compared to state-of-the-art algorithms for learning SBNs, (i) produces a more concise model, thus enabling faster testing; (ii) improves predictive performance; (iii) accelerates convergence; and (iv) prevents overfitting.

The authors propose a variational Bayes method for model selection in sigmoid belief networks. The method can eliminate nodes in the hidden layers of multilayer networks. The derivation of the criterion appears technically solid and a fair amount of experimental support for the good performance of the method is provided.
I have to say I'm no expert in this area, and I hope other reviewers can comment on the level of novelty.
detailed comments:
- p. 1, l. 21: "Model selection is here the task of selecting the number of layers [...]": I got the impression that the proposed algorithm only eliminates individual nodes, not entire layers. Please clarify.
- p. 2, ll. 76--77: Are there really no weights on the first layer? You forgot to define b.
- p. 5, ll. 173--175: If nodes with expected proportion of 1's very small can be eliminated, why doesn't the same hold for nodes with expected proportion of 0's equally small? "When the expectation is not exact, such as in the top layers, [...]": Please clarify. How can we tell, when the expectation is not exact? And do you really mean 'exact' or just 'accurate', etc. What is the precise rule to decide this.
- p. 6, ll. 226--227: Please explain what 10-5^2 and 25-15 means (is there a typo, or why would you write 5^2 instead of simply 25?).
- p. 6, l. 239: "Our performance metric is the variational lower bound of the test log-likelihood." Why use a variational bound as a performance metric? Sounds like variational techniques are an approach to derive a criterion, but using them also to measure performance sounds questionable. Should the evaluation be based on a score that is independent of the chosen approach and the made assumptions?
- references: please provide proper bibliographic information, not just "JMLR" or "NIPS". Add volume, page numbers, etc.


Do Deep Neural Networks Suffer from Crowding?
Crowding is a visual effect suffered by humans, in which an object that can be recognized in isolation can no longer be recognized when other objects, called flankers, are placed close to it. In this work, we study the effect of crowding in artificial Deep Neural Networks (DNNs) for object recognition. We analyze both deep convolutional neural networks (DCNNs) as well as an extension of DCNNs that are multi-scale and that change the receptive field size of the convolution filters with their position in the image. The latter networks, that we call eccentricitydependent, have been proposed for modeling the feedforward path of the primate visual cortex. Our results reveal that the eccentricity-dependent model, trained on target objects in isolation, can recognize such targets in the presence of flankers, if the targets are near the center of the image, whereas DCNNs cannot. Also, for all tested networks, when trained on targets in isolation, we find that recognition accuracy of the networks decreases the closer the flankers are to the target and the more flankers there are. We find that visual similarity between the target and flankers also plays a role and that pooling in early layers of the network leads to more crowding. Additionally, we show that incorporating flankers into the images of the training set for learning the DNNs does not lead to robustness against configurations not seen at training.

This paper studies if crowding, a visual effect suffered by human visual systems, happens to deep neural network as well. The paper systematically analyzes the performance difference when (1) clutter/flankers is present; (2) the similarity and proximity to the target; (3) when different architectures of the network is used.
Pros:
There are very few papers to study if various visual perceptual phenomenon exists in deep neural nets, or in vision algorithms in general. This paper studies the effect of crowding in DNN/DCNN image classification problem, and presents some interesting results which seems to suggest similar effect exists in DNN because of pooling layers merges nearby responses. And this is related to the theories of crowding in humans, which is also interesting. The paper also suggests what we should not do prematurely pooling if when designing architectures. In my opinion such papers should be encouraged.
Cons:
My main criticism to the paper is that it solely studies crowding in the context of image classification. However, if crowding is studied in the context of object detection, where the task is localize the object and recognize its category, the effect may be significantly lessened. For example, R-CNN proposes high saliency regions where the object might be and perform classification on that masked region. Because targets are usually centered in such proposed region and background clutters are excluded from the proposed region, the accuracy can potentially be much higher.
After all, the extent to which crowding is present in DNN depends a lot on the chosen architecture. And the architecture in this paper is very primitive compare to what researchers consider state-of-the-art these days, and the accuracy of the MNIST tasks reported by the paper are way lower than what most researchers would expect from a practical system. For example, [1] performs digit OCR which has much more clutters but with very high accuracy. It is not obvious architectures like that also suffer from crowding.
Suggestion:
The paper is overall easy to follow. I feel the experimental setup can be more clear if some more example images (non-cropped, like the ones in the Fig1 of supplementary material).
Overall, this paper has an interesting topic and is a nice read. The conclusions are not too strong because it uses simplistic architecture/datasets. But I think it is nonetheless this is a helpful paper to (re-)generate some interest on drawing relation between theories of human visual recognition and neural nets.
[1] Goodfellow, Ian J., et al. "Multi-digit number recognition from street view imagery using deep convolutional neural networks." arXiv preprint arXiv:1312.6082 (2013).


Lookahead Bayesian Optimization with Inequality Constraints
We consider the task of optimizing an objective function subject to inequality constraints when both the objective and the constraints are expensive to evaluate. Bayesian optimization (BO) is a popular way to tackle optimization problems with expensive objective function evaluations, but has mostly been applied to unconstrained problems. Several BO approaches have been proposed to address expensive constraints but are limited to greedy strategies maximizing immediate reward. To address this limitation, we propose a lookahead approach that selects the next evaluation in order to maximize the long-term feasible reduction of the objective function. We present numerical experiments demonstrating the performance improvements of such a lookahead approach compared to several greedy BO algorithms, including constrained expected improvement (EIC) and predictive entropy search with constraint (PESC).

This paper seems a continuation of last year: Bayesian optimization with a finite budget... where the authors have added new elements to deal with inequality constraints. The method uses a approximation of a lookahead strategy by dynamic programming. For the constrained case, the authors propose an heuristic that combines the EIc criterion for all the steps except for the last one were the mean function is used. The authors claim that the mean function has an exploitative behaviour, although it has been previously shown that it might be misleading [A].
      A considerably amount of the text, including Figure 1, can be mostly found in [16]. Although it is nice to have an self-contained paper as much as possible, that space could be used to explain better the selection of the acquisition heuristic and present alternatives. For example, the experiments should show the result a single-step posterior mean acquisition function, which will correspond to h=0 as a baseline for each of the functions/experiments.
      Concerning the experiments, there are some details that should be improved or addressed in the paper:
      - Why P2 does not include h=3?
      - How is it possible that EIc for P2 goes upwards around n=25?
      - The plots only shows median values without error bars. I can understand that for that, for such number of repetitions, the error bars might be small, but that should be adressed. Furthermore, a common problem of lookahead methods is that, while most of the time outperform greedy methods, they can also fail catastrophically for selecting the wrong path. This would results in some lookahead trials resulting in poor results. Using the average instead of the median would show also the robustness of the method.
      - The number of iterations is fairly small for the standard in Bayesian optimization. In fact, it can bee seen that most methods have not reach convergence at that point. This is specially problematic in P1 and P4 where the plots seems to cross exactly at the end of experiments,
      - It is unclear why the performance decreases after h=2. If possible, the authors should provide an intuition behind that results. Furthermore, the fact that for most of the experiments h=1 is the optimal strategy seems to indicate that the whole dynamic programing and heuristics might be excessive for CBO, or that more complex and higher-dimensional problems are need to illustrate the benefits of the strategy.

      [A] Jones, Donald R. "A taxonomy of global optimization methods based on response surfaces." Journal of global optimization 21.4 (2001): 345-383.


Decomposable Submodular Function Minimization Discrete and Continuous
This paper investigates connections between discrete and continuous approaches for decomposable submodular function minimization. We provide improved running time estimates for the state-of-the-art continuous algorithms for the problem using combinatorial arguments. We also provide a systematic experimental comparison of the two types of methods, based on a clear distinction between level-0 and level-1 algorithms.

This paper studies the problem of minimizing a decomposable submodular function. Submodular minimization is a well studied and important problem in machine learning for which there exist algorithms to solve the problem exactly. However, the running time of these algorithms is a high polynomial and they are thus oftentimes not practical. To get around this issue, submodular functions that can be decomposed and written as a sum of submodular functions over a much smaller support (DSFM) are often considered as they often appear in practice.
This paper improves the analysis of the fastest algorithms for DSFM by a factor equal to the number of functions in the decomposition. It also provides an experimental framework based on a distinction between “level 0” algorithms, which are subroutines for quadratic minimization, and “level 1” algorithms which minimize the function using level 0 as a black box. These allow a more meaningful comparison where the same level 0 algorithms are used to compare different algorithms. These experiments show a tradeoff between the discrete algorithms that require more calls to the level 0 subroutines and gradient methods with weaker requirements for level 0 but more computation for level 1.
The analysis is complex and relies on both discrete and continuous optimization techniques to meaningfully improve the running time of an important problem where the computational complexity is expensive. The experiments also highlight an interesting tradeoff which suggests that different algorithms should be used in different contexts for the running time of DSFM.
A weakness of the paper is that the writing is very dense and sometimes hard to follow. It would have been nice to have more discussion on the parameters kappa* and l* and the precise bounds in terms of these parameters. It would have also been nice to have some comparison on the theoretical bounds between RCDM, ACDM, and IBFS.


Streaming Weak Submodularity: Interpreting Neural Networks on the Fly
In many machine learning applications, it is important to explain the predictions of a black-box classifier. For example, why does a deep neural network assign an image to a particular class? We cast interpretability of black-box classifiers as a combinatorial maximization problem and propose an efficient streaming algorithm to solve it subject to cardinality constraints. By extending ideas from Badanidiyuru et al. [2014], we provide a constant factor approximation guarantee for our algorithm in the case of random stream order and a weakly submodular objective function. This is the first such theoretical guarantee for this general class of functions, and we also show that no such algorithm exists for a worst case stream order. Our algorithm obtains similar explanations of Inception V3 predictions 10 times faster than the state-of-the-art LIME framework of Ribeiro et al. [2016].

This paper proposes a new approach STREAK for maximizing weakly submodular functions. The idea is to collect several outputs of the Threshold Greedy algorithm, where the selection is based on a given threshold. The theoretical results of the Threshold Greedy algorithm and STREAK are verified sequentially. STREAK is also used to provide interpretable explanations for neural-networks and the empirical studies are given.
This is an interesting work. The streaming algorithm is novel and the analyses are elaborate. The problem constructed to prove the ineffectiveness of randomized streaming algorithms is ingenious. The experiments also show the superiority of STREAK. However, how to choose the proper \epsilon in STEAK? Figure 2(a) shows that by varying \epsilon, the algorithm can achieve a gradual tradeoff between speed and performance, but the result is trivial because with the increase of \epsilon, the time and space complexity will both increase and lead to better performance.
I have checked all the proofs, and believe that most of them are correct. However, there are also some places which are not clear.
1. In the proof of Lemma A.1, the choice of appropriate arrival order is unclear and how to derive the inequality at the end of the proof?
2. In the proof of Lemma A.9, what is the meaning of the first paragraph? It seems to have nothing to do with the proof in the second paragraph.
3. In the proof of Theorem 5.6, why m=0 implies I=\ emptyset?


Selective Classification for Deep Neural Networks
Selective classification techniques (also known as reject option) have not yet been considered in the context of deep neural networks (DNNs). These techniques can potentially significantly improve DNNs prediction performance by trading-off coverage. In this paper we propose a method to construct a selective classifier given a trained neural network. Our method allows a user to set a desired risk level. At test time, the classifier rejects instances as needed, to grant the desired risk (with high probability). Empirical results over CIFAR and ImageNet convincingly demonstrate the viability of our method, which opens up possibilities to operate DNNs in mission-critical applications. For example, using our method an unprecedented 2% error in top-5 ImageNet classification can be guaranteed with probability 99.9%, and almost 60% test coverage.

The paper addresses the problem of constructing a classifier with the reject
option that has a desired classification risk and, at the same time, minimizes the
probability the "reject option". The authors consider the case when the
classifiers and an associate confidence function are both known and the task is
to determine a threshold on the confidence that determines whether the
classifier prediction is used or rejected. The authors propose an algorithm
finding the threshold and they provide a statistical guarantees for the method.
Comments:
- The authors should provide an exact definition of the task that they attempt
to solve by their algorithm. The definition on line 86-88 describes rather the
ultimate goal while the algorithm proposed in the paper solves a simpler
problem: given $(f,\kappa)$ find a threshold $\theta$ defining $g$ in equation
(3) such that (2) holds and the coverage is maximal.
- It seems that for a certain values of the input arguments (\delta,r^*,S_m,...)
the Algorithm 1 will always return a trivial solution. By trivial solution I
mean that the condition on line 10 of the Algorithm 1 is never satisfied and
thus all examples will be at the end in the "reject region". It seems to me that
for $\hat{r}=0$ (zero trn error) the bound B^* solving equation (4) can be
determined analytically as
   $B^* = 1-(\delta/log_2(m))^{1/m}$.
Hence, if we set the desired risk $r^*$ less than the number $B^* =
1-(\delta/log_2(m))^{1/m}$ then the Algorithm 1 will always return a trivial
solution. For example, if we set the confidence $\delta=0.001$ (as in the
experiments) and the number of training examples is $m=500$ then the minimal
bound is $B^*=0.0180$ (1.8%). In turn, setting the desired risk $r^* < 0.018$
will always produce a trivial solution whatever data are used. I think this
issue needs to be clarified by the authors.
- The experiments should contain a comparison to a simple baseline that anyone
would try as the first place. Namely, one can find the threshold directly using
the empirical risk $\hat{r}_i$ instead of the sophisticated bound B^*. One would
assume that the danger of over-fitting is low (especially for 5000 examples used
in experiments) taking into account the simple hypothesis space (i.e. "threshold
rules"). Without the comparing to baseline it is hard to judge the practical
benefits of the proposed method.

- I'm missing a discussion of the difficulties connected to solving the
numerical problem (4). E.g. which numerical method is suitable and whether there
are numerical issues when evaluating the combinatorial coefficient for large m
and j.
Typos:
- line 80: (f,g)
- line 116: B^*(\hat{r},\delta,S_m)
- line 221: "mageNet"


Accelerated First-order Methods for Geodesically Convex Optimization on Riemannian Manifolds
In this paper, we propose an accelerated first-order method for geodesically convex optimization, which is the generalization of the standard Nesterov's accelerated method from Euclidean space to nonlinear Riemannian space. We first derive two equations and obtain two nonlinear operators for geodesically convex optimization instead of the linear extrapolation step in Euclidean space. In particular, we analyze the global convergence properties of our accelerated method for geodesically strongly-convex problems, which show that our method improves the convergence rate from
Moreover, our method also improves the global convergence rate on geodesically general convex problems from O(1/k) to O(1/k 2 ). Finally, we give a specific iterative scheme for matrix Karcher mean problems, and validate our theoretical results with experiments.

Summary of the Paper
====================
The paper considers a geodesic generalization of Nesterov's accelerated gradient descent (AGD) algorithm for Riemannian spaces. Two versions are presented: geodesic convex case and geodesic strongly convex smooth case. The proposed algorithms are instantiated for Karcher mean problems, and are shown to outperform two previous algorithms (RGD, RSGD), which address the same setting, with randomized data.
Evaluation
==========
From theoretical point of view, finding a proper generalization for the momentum term (so as to be able to implement AGD) which maintains the same convergence rate for any Riemannian space is novel and very interesting. From practical point of view, it is not altogether clear when the overall running time reduces. Indeed, although this algorithm requires significantly smaller number of iterations, implementing the momentum term can be very costly (as opposed to Euclidean spaces). That said, the wide range of settings to which this algorithm potentially applies makes it appealing as a general mechanism, and may encourage further development in this direction. The paper is well-written and relatively easy-to-follow.

General Comments
================
- The differential geometrical notions and and other definitions used intensively throughout this paper may not be familiar for the typical NIPS reader. I would suggest making the definition section more tight and clean. In particular, the following are not seemed to be defined in the text: star-concave, star-convex, grad f(x), intrinsic inner-product, diameter of X, conic geometric optimization, retractions.
- Equations 4 and 5 are given without any intuition as how should one derive them. They seem to be placed somewhat out of the blue, and I feel like the nice figure provided by the authors, which could potentially explain them, is not addressed appropriately in the text.

Minor Comments
==============
L52 - redundant 'The'
L52+L101 - There is a great line of work which tries to give a more satisfying interpretation for AGD. Stating that the proximal interpretation as the main interpretation seems to me somewhat misleading.
L59 - Can you elaborate more on the computational complexity required for implementing the exponent function and the nonlinear momentum operator S.
L70 - why linearization of gradient-like updates are contributions by themselves.
L72 - classes?
L90 + L93 - The sentence 'we denote' is not clear.
L113 - the sentence 'In addition, different..' is a bit unclear..
L126 - besides the constraint on alpha, what other considerations are needed to be taken in order to set its value optimally?
L139 + L144 - Exp seems to be written in the wrong font.
L151 - The wording used in Lemma 3 is a bit unclear.
L155 - in Thm 1, consider restating the value of beta.
L165 - How does the upper bound depend on D? Also, here again, how is alpha should be set?
L177 - maybe geometrically -> geodesically?
L180 - In section 5, not sure I understand how the definition of S is instantiated for this case.
L181 - 'For the accelerated scheme in (4)' do you mean algorithm 1?
L183 - Y_k or y_k?
L193 - 'with the geometry' is not clear.
L201 - Redundant 'The'. Also, (2) should be (3)?
L219 - Can you explain more rigorously (or provide relevant pointers) why RGD is a good proxy for all the algorithm stated above?
Page 8 - Consider providing the actual time as well, as this benchmark does not take into account the per-iteration cost.
L225 - Why is C defined explicitly?


Phase Transitions in the Pooled Data Problem
In this paper, we study the pooled data problem of identifying the labels associated with a large collection of items, based on a sequence of pooled tests revealing the counts of each label within the pool. In the noiseless setting, we identify an exact asymptotic threshold on the required number of tests with optimal decoding, and prove a phase transition between complete success and complete failure. In addition, we present a novel noisy variation of the problem, and provide an information-theoretic framework for characterizing the required number of tests for general random noise models. Our results reveal that noise can make the problem considerably more difficult, with strict increases in the scaling laws even at low noise levels. Finally, we demonstrate similar behavior in an approximate recovery setting, where a given number of errors is allowed in the decoded labels.

In the pooled data problem there are p items, each of which is assigned 1 of d possible labels. The goal is to recover the vector of labels from n "pooled tests." In each of these tests, a "pool" (subset of the items) is chosen and we observe how many items in the pool have each label (but not which items have which). A random prior on the label vector is used: the proportion with which each label occurs in the population is fixed, and the true labelling is chosen independently among all labellings that exactly obey these proportions. This paper considers the non-adaptive case where all the pools (subsets) must be chosen in advance. The focus is mainly on the case of Bernoulli random pools: the incidence matrix of pools versus items has iid Bernoulli(q) entries for some fixed 0 < q < 1. The objective is exact recovery of the label vector, although extensions to approximate recovery are considered in the appendix.
This paper proves information-theoretic lower bounds for the pooled data problem, both in the noiseless setting described above and in a very general noisy model.
In the noiseless setting with Bernoulli random pools, they prove the following sharp phase transition. The regime is where the number of samples is n = c*p/log(p) for a constant c, with p tending to infinity. They find the exact constant C such that if c > C then exact recovery is possible with success probability tending to 1, and if c < C then any procedure will fail with probability tending to 1. The constant C depends on the number of labels and the proportions of each label (but not on the constant q which determines the pool sizes). The upper bound was known previously; this paper proves the tight lower bound (i.e. the impossibility result), improving upon a loose bound in prior work.
The sharp information-theoretic threshold above is not known to be achieved by any efficient (i.e. polynomial time) algorithm. Prior work (citation [4] in the paper) suggests that efficient algorithms require a different threshold (in which n scales proportionally to p).
The key technical insight used in this paper to prove the sharp lower bound is the following "genie argument." Let G be some subset of {1,..,d} (the label values), and imagine revealing the labels of all the items that have labels not in G. Now apply a more standard lower bound to this setting, and optimize G in order to get the strongest possible bound.
The other contribution of this paper is to the noisy setting, which has not previously been studied. This paper defines a very general noise model where each observation is corrupted via an arbitrary noisy channel. They give a lower bound for exact recovery in this setting, and specialize it to certain concrete cases (e.g. Gaussian noise). The proof of the lower bound is based on a variant of the above genie argument, combined with standard lower bounds based on Fano's inequality.
Unfortunately there are no existing upper bounds for the noisy setting, so it is hard to judge the strength of this lower bound. The authors do, however, argue for its efficacy in a few different ways. First, it recovers the sharp results for the noiseless case and also recovers known bounds for the "group testing" problem, which is essentially the special case of 2 labels: "defective" and "non-defective" (and you want to identify the defective items). Furthermore, the lower bound for Gaussian noise is strong enough to show an asymptotic separation between the noiseless and noisy cases: while in the noiseless case it is necessary and sufficient for n to be of order p/log(p), the noisy case requires n to be at least p*log(p).
In the conclusion, the authors pose the open question of finding upper bounds for the noisy setting and provide some justification as to why this seem difficult using existing techniques.
I think this is a good paper and I recommend its acceptance. The exposition is good. The sharp lower bound for the noiseless case is a very nice result, and the proof technique seems to be novel. The lower bound for the noisy case is also a nice contribution; although it is hard to judge whether or not it should be tight (since no upper bound is given), it is certainly nontrivial, using the genie argument to surpass standard methods.
Specific comment:
- I think there is a typo in Table 2: the heading "exact recovery" should be "noiseless." (If I understand correctly, the entire table pertains to exact recovery.)


A Greedy Approach for Budgeted Maximum Inner Product Search
Maximum Inner Product Search (MIPS) is an important task in many machine learning applications such as the prediction phase of low-rank matrix factorization models and deep learning models. Recently, there has been substantial research on how to perform MIPS in sub-linear time, but most of the existing work does not have the flexibility to control the trade-off between search efficiency and search quality. In this paper, we study the important problem of MIPS with a computational budget. By carefully studying the problem structure of MIPS, we develop a novel Greedy-MIPS algorithm, which can handle budgeted MIPS by design. While simple and intuitive, Greedy-MIPS yields surprisingly superior performance compared to state-of-the-art approaches. As a specific example, on a candidate set containing half a million vectors of dimension 200, Greedy-MIPS runs 200x faster than the naive approach while yielding search results with the top-5 precision greater than 75%.

The aim of the paper is to propose a new greedy approach for Maximum Inner Product Search problem: given a candidate vector, retrieve a set of vectors with maximum inner product to the query vector. This is a crucial step in several machine learning and data mining algorithms, and the state of the art methods work in sub-linear time recently. The originality of the paper is to study the MIPS problem under a computational budget. The proposed approach achieves
better balance between search efficiency and quality of the retrieved vectors, and  does not require a nearest neighbor search phase, as commonly done by state of the art approaches. The authors claim impressive runtime results (their algorithm is 200x faster than the naive approach), and a top-5 precision  greater than 75%.

	  The paper is very dense (the space between two lines seems smaller than the one in the template). However, the paper is well-written and the procedure is well-explained. The proposed method seems also quite original, and comes with theoretical guarantees. The technical results seem sound.

	  Some remarks:

	  Figure 1 should be placed at the top of P.5, it is a bit difficult to follow without the later explanations.

	  The bound used in P.4 needs to be more studied, in order to find, for instance, some properties (or better, an approximation). This bound is a key point of this procedure, and it is used at the beginning.

	  P.5 "visit (j,t) entries of Z": (j,t) is a cell in the matrix Z, however you consider this notation as a number. Maybe "j \times t" entries?

	  The reviewer would be interested to have access to the source code of the algorithm and the data, so as he can reproduce the expriments?


Convergence rates of a partition based Bayesian multivariate density estimation method
We study a class of non-parametric density estimators under Bayesian settings. The estimators are obtained by adaptively partitioning the sample space. Under a suitable prior, we analyze the concentration rate of the posterior distribution, and demonstrate that the rate does not directly depend on the dimension of the problem in several special cases. Another advantage of this class of Bayesian density estimators is that it can adapt to the unknown smoothness of the true density function, thus achieving the optimal convergence rate without artificial conditions on the density. We also validate the theoretical results on a variety of simulated data sets.

Note: Below, I use [#M] for references in the main paper and [#S] for references in the supplement, since these are indexed differently.
Summary: This paper proposes and analyzes a Bayesian approach to nonparametric density estimation. The proposed method is based on approximation by piecewise-constant functions over a binary partitioning of the unit cube, using a prior that decays with the size of the partition. The posterior distribution of the density is shown to concentrate around the true density f_0, at a rate depending on the smoothness r of f_0, a measure in terms of how well f_0 can be approximated by piecewise-constant functions over binary partitionings. Interestingly, the method automatically adapts to unknown r, and r can be related to more standard measures of smoothness, such as Holder continuity, bounded variation, and decay rate of Haar basis coefficients. As corollaries, posterior concentration rates are shown for each of these cases; in the Holder continuous case, this rate is minimax optimal.
Major Comments:
The theoretical results of this paper appear quite strong to me. In particular, the results on adaptivity (to unknown effective dimension and smoothness) are quite striking, especially given that this seems an intrinsic part of the estimator design, rather than an additional step (as in, e.g., Lepski's method). Unfortunately, I'm not too familiar with Bayesian or partition-based approaches to nonparametric density estimation, and my main issue is that the relation of this work to previous work isn't very clear. Looking through the supplement, it appears that most of proofs are based on the results of [9M] and [14M], so there is clearly some established closely related literature. I'd likely raise my score if this relation could be clarified, especially regarding the following two questions:
1) Is the prior introduced in Section 2.2 novel? If so, how does it differ from similar prior work (if such exists), and, if not, which aspects, precisely, of the paper are novel?
2) Much of the proof of Theorem 3.1 appears to depend on results from prior work (e.g., Lemmas A.2, A.4, B.1, and B.2, and Theorem A.3 are from [4S] and [6S]). Corrolaries 3.2, 3.3, and 3.4 all follow by combining Theorem 3.1 with results from [4S] and [1S] that relate the respective smoothness condition to smoothness in terms of approximability by piecewise-constant functions on binary partitions. Thus, much of this work appears to be a re-working of previous work, while Lemma B.4 and the remainder of Theorem 3.1 appear to be the main contributions. My question is: at a high level, what were the main limitations of the previous results/proof techniques that had to be overcome to prove Theorem 3.1, and, if it's possible to summarize, what were the main innovations required to overcome these?
Minor Comments:
Lines 36-44: It would help to mention here that rates are measured in Hellinger divergence.
Line 40-41: (the minimax rate for one-dimensional Hölder continuous function is (n/log n)^{−\beta/(2\beta+1)}): If I understand correctly, the log factors stem from the fact that \beta is unknown (I usually think of the minimax rate as n^{−\beta/(2\beta+1)}, when \beta is treated as known). If this is correct, it would help to mention this here.
Line 41: small typo: "continuous function" should be "continuous functions"
Lines 60-72: There's an issue with the current notation: As written, \Omega_2,...,\Omega_I aren't well-defined partition by this recursive procedure. If we split \Omega_j at step i, then \Omega_j should be removed from the list of partitions and replaced by two smaller partitions. I think I understand what is meant (Figure 1 is quite clear), but I don't see a great way to explain this with concise mathematical notation - the options I see are (a) describing the process as a tree with a set at each node, and then taking all the leaves of the tree or (b) using a pseudocode notation where the definition of \Omega_j can change over the course of the recursive procedure.
Line 185: I believe the minimax rate for the bounded variation class is of order n^(-1/3) (see, e.g., Birgé, Lucien. "Estimating a density under order restrictions: Nonasymptotic minimax risk." The Annals of Statistics (1987): 995-1012.) Perhaps this is worth mentioning?
Lines 157-175: Section 3.1 considers the case of a weak-\ell^p constraint on the Haar basis coefficients of the density. The paper calls this a spatial sparsity constraint. I feel this is misleading, since the sparsity assumption is over the Haar basis coefficients, rather than over spatial coordinates (as in [1M]). As a simple example, the uniform distribution is extremely sparse in the Haar basis, but is in no way spatially concentrated. I believe this is actually a smoothness assumption, since Haar basis coefficients can be thought of as identifying the discontinuities of a piecewise constant approximation function. Indeed, the weak-\ell^p constraint on the Haar coefficients is roughly equivalent to a bound on the Besov norm of the density (see Theorem 5.1 of Donoho, David L. "De-noising by soft-thresholding." IEEE transactions on information theory 41.3 (1995): 613-627.)
Line 246: small typo: "based adaptive partitioning" should be "based on adaptive partitioning".
Line 253: Citation [1M] is missing the paper title. I believe the intended citation is "Abramovich, F., Benjamini, Y., Donoho, D. L., & Johnstone, I. M. (2006). Special invited lecture: adapting to unknown sparsity by controlling the false discovery rate. The Annals of Statistics, 584-653."


Translation Synchronization via Truncated Least Squares
In this paper, we introduce a robust algorithm, TranSync, for the 1D translation synchronization problem, in which the aim is to recover the global coordinates of a set of nodes from noisy measurements of relative coordinates along an observation graph. The basic idea of TranSync is to apply truncated least squares, where the solution at each step is used to gradually prune out noisy measurements. We analyze TranSync under both deterministic and randomized noisy models, demonstrating its robustness and stability. Experimental results on synthetic and real datasets show that TranSync is superior to state-of-the-art convex formulations in terms of both efficiency and accuracy.

In this paper the authors describe a robust and scalable algorithm,TranSync, for solving the 1D translation synchronization problem. The algorithm is quite simple, to solve a truncated least squares at each iteration, and then the computational efficiency is superior to state-of-the-art methods for solving the linear programming formulation. On the other hand, the analyze TranSync under both deterministic and randomized noise models, demonstrating its robustness and stability. In particular, when the pair-wise measurement is biased, TranSync can still achieve sub-constant recovery rate, while the linear programming approach can tolerate no more
than 50% of the measurements being biased. The paper is very readable and
the proofs of main theorems are clear and appear correct. However, it will be good if the authors can provide more intuition behind the theorem. The numerical experiments are complete and clear.


Solid Harmonic Wavelet Scattering: Predicting Quantum Molecular Energy from Invariant Descriptors of 3D Electronic Densities
We introduce a solid harmonic wavelet scattering representation, invariant to rigid motion and stable to deformations, for regression and classification of 2D and 3D signals. Solid harmonic wavelets are computed by multiplying solid harmonic functions with Gaussian windows dilated at different scales. Invariant scattering coefficients are obtained by cascading such wavelet transforms with the complex modulus nonlinearity. We study an application of solid harmonic scattering invariants to the estimation of quantum molecular energies, which are also invariant to rigid motion and stable with respect to deformations. A multilinear regression over scattering invariants provides close to state of the art results over small and large databases of organic molecules.

The paper presents the solid harmonic scattering, which creates a rotation invariant representation of 2D and 3D structures. The paper presents the details of the proposed transformation and derives its properties. The solid harmonic scattering is then used to predict the energy of molecules given the positions of individual atoms and their charges. A permutation invariant embedding of a molecule is first computed from positions and charges, and then the scattering transform is applied to obtain a rotation and translation invariance representation. This representation is used to predict the total energy with a linear regressor and a neural net with multiplicative gates. Experiments in the GDB7-12 dataset are performed and the results are competitive with other machine learning based approaches.
The problem of energy prediction is important, and the proposed transformation is interesting. The introduction makes the case of learning from data using the right operators (such as convolutions for images), and motivates the exploration of special operators to analyze other types of data, such as molecule structures. The authors implemented the solution in GPUs to accelerate computations. The formulation of the method seems interesting, but the following questions need to be answered to frame this work with the current research in machine learning:
* Although an elegant design, the proposed module does not have learnable parameters, and thus the contribution to machine learning appears limited. Convolutional filters have the capacity to adapt weights for each problem, while the proposed transform seems to be fixed. Even though it has connections to neural networks, the static nature of the solution goes in the opposite direction of designing machines that learn.
* The transform seems to be very specific for molecule representation. It would be interesting to see applications in other domains that would benefit from this transform. If the value of this work has more impact in the quantum physics community, perhaps NIPS is the wrong venue to discuss its merit?


Clustering Stable Instances of Euclidean k-means
The Euclidean k-means problem is arguably the most widely-studied clustering problem in machine learning. While the k-means objective is NP-hard in the worst-case, practitioners have enjoyed remarkable success in applying heuristics like Lloyd's algorithm for this problem. To address this disconnect, we study the following question: what properties of real-world instances will enable us to design efficient algorithms and prove guarantees for finding the optimal clustering? We consider a natural notion called additive perturbation stability that we believe captures many practical instances of Euclidean k-means clustering. Stable instances have unique optimal k-means solutions that does not change even when each point is perturbed a little (in Euclidean distance). This captures the property that kmeans optimal solution should be tolerant to measurement errors and uncertainty in the points. We design efficient algorithms that provably recover the optimal clustering for instances that are additive perturbation stable. When the instance has some additional separation, we can design a simple, efficient algorithm with provable guarantees that is also robust to outliers. We also complement these results by studying the amount of stability in real datasets, and demonstrating that our algorithm performs well on these benchmark datasets.

The authors propose a notion of additive perturbation stability (APS) for Euclidean distances that maintain the optimal k-means clustering solution when each point in the data is moved by a sufficiently small Euclidean distance. I think the paper is rather interesting; however, the results of the paper are not very surprising.
Here are my comments regarding the paper:
(1) To my understanding, the results of Theorem 1.2 are only under the condition of APS. They only hold for the case of k=2 components and may lead to exponential dependence on $k$ components for large $k$. However, under the additional margin condition between any two pairs of cluster, we will able to guarantee the existence of polynomial algorithm on $k$. Can you provide a high level idea of how this additional assumption actually helps? Is it possible to have situation without that margin condition that there exists no algorithm that is polynomial in terms of $k$?
(2) I find the term $\Delta = (1/2-\epsilon)D$, where $D$ is the maximum distance between any pairs of means, is interesting. It also has a nice geometric meaning, which is the distance between any center to the apex of cone. I wonder whether this term $\Delta$ is intrinsic to the setting of APS? May it still be available under other perturbation notions?
(3) In the result of Theorem 1.3 and later Theorem 3.2, we both require the condition that $\rho$, which is a threshold for the margin between any two pairs of cluster, to be at least the oder of $\Delta/\epsilon^{2}$. As $\epsilon$ is sufficiently small, which is also the challenging theme of APS, the term $\Delta/\epsilon^{2}$ becomes very big, i.e., the distances between any two pairs of cluster become very large.  The fact that there is an algorithm running in time polynomial of $k$ components, sample size $n$, and dimension $d$ under that setting is not surprising. Is it possible to improve the lower bound of the margin between any two pairs of cluster?
(4) In all the results with running time of the paper, the authors only provide the bound in terms of $n$, $d$, and $k$. I wonder about the constants that are along with these terms. How do they change with margin $\rho$ and $\epsilon$-APS?
(5) As the authors indicate in the paper, data in practice may have very small value of $\epsilon$. It seems that the conditions of the results with polynomial algorithms in the paper, e.g. Theorem 3.2, will not be satisfied by real data.
(6) It seems to me that the results in the paper can be extended to other distances that are different from Euclidean distance. The geometric explanation of additive perturbation stability may be different, which also leads to the modifications of other notions and results in the paper. Such extensions will be very useful in case that we want to work with kernel K-means to deal with sophisticated structures of data.


On-the-fly Operation Batching in Dynamic Computation Graphs
Dynamic neural network toolkits such as PyTorch, DyNet, and Chainer offer more flexibility for implementing models that cope with data of varying dimensions and structure, relative to toolkits that operate on statically declared computations (e.g., TensorFlow, CNTK, and Theano). However, existing toolkits-both static and dynamic-require that the developer organize the computations into the batches necessary for exploiting high-performance algorithms and hardware. This batching task is generally difficult, but it becomes a major hurdle as architectures become complex. In this paper, we present an algorithm, and its implementation in the DyNet toolkit, for automatically batching operations. Developers simply write minibatch computations as aggregations of single instance computations, and the batching algorithm seamlessly executes them, on the fly, using computationally efficient batched operations. On a variety of tasks, we obtain throughput similar to that obtained with manual batches, as well as comparable speedups over singleinstance learning on architectures that are impractical to batch manually.

Summary:
The authors of this paper extend neural network toolkit DyNet with automatic operation batching. Batching enables efficient utilization of CPUs and GPUs by turning matrix-vector products into matrix-matrix products and reducing kernel launch overhead (for GPUs) but it is commonly done manually. Manual batching is manageable for simple feed-forward-networks but it becomes increasingly a headache as we explore more flexible models that take variable-length input, tree-structured input, or networks that perform dynamic control decisions.
Chainer, DyNet, and PyTorch are recently proposed neural network toolkits that allow user to dynamically define the computation graph using the syntax of the host language (if, while, etc in python). This is desirable as it avoids tookit specific constructions (e.g., cond in TensorFlow) and make the network definition intuitive but it tends to limit performance because the network construction and computation happens at the same time. Thus although it would be straightforward to program a control flow that supports variable length or tree-structured input in these toolkits, in practice it would be too inefficient to process single instance at a time.
The  key contribution of this paper is to delay and automatically batch the computation so that a user can still define an arbitrary complex control flow using the host language as in original DyNet without worrying about operation batching. The approach is similar to TensorFlow Fold (TFF) but differs in two places: first, the computation graph is defined in a dynamic manner using the control flow of the host language and the user does not need to learn any new language (note that TFF introduces many new operations that are not in TensorFlow); second, it employs an agenda-based approach for operation batching in contrast to the depth-based approach employed in TFF. The authors empirically show that agenda-based batching is slightly more efficient than depth-based batching.
Detailed comments:
It wasn't clear to me if there is a stronger argument for dynamic definition of the computation graph other than the simplicity of coding compared to static definition of the graph in TFF, for example.
Would the lazy evaluation still work when the control decision depends on the computed value (this is the case for the training with exploration scenario [2, 4, 9]. In this case, static graph definition may have an advantage.


Learning Linear Dynamical Systems via Spectral Filtering
We present an efficient and practical algorithm for the online prediction of discrete-time linear dynamical systems with a symmetric transition matrix. We circumvent the non-convex optimization problem using improper learning: carefully overparameterize the class of LDSs by a polylogarithmic factor, in exchange for convexity of the loss functions. From this arises a polynomial-time algorithm with a near-optimal regret guarantee, with an analogous sample complexity bound for agnostic learning. Our algorithm is based on a novel filtering technique, which may be of independent interest: we convolve the time series with the eigenvectors of a certain Hankel matrix.

Linear dynamical systems are a mainstay of control theory. Unlike the empirical morass of work that underlies much of machine learning work, e.g. deep learning, where there is little theory, and an attempt to produce general solutions of unknown reliability and quality, control theorists wisely have chosen an alternative course of action, where they focused on a simple but highly effective linear model of dynamics, which can be analyzed extremely deeply. This led to the breakthrough work many decades ago of Kalman filters, without which the moon landing would have been impossible.
This paper explores the problem of online learning (in the regret model) of dynamical systems, and improves upon previous work in this setting that was restricted to the single input single output (SISO) case [HMR 16]. Unlike that paper, the present work shows that regret bounded learning of an LDS is possible without making assumptions on the spectral structure (polynomially bounded eigengap), and signal source limitations.
The key new idea is a convex relation of the original non-convex problem, which as the paper shows, is "the central driver" of their approach. The basic algorithm is a variant of the original projected gradient method of Zinkevich from 2003, The method is viewed as a online wave filtered regression method (Algorithm 1), where pieces of the dynamical system are learned over time and stitched together into an overall model. The paper shows that optimization over linear maps is appropriate convex relaxation of the original objective function,.
The paper is well written and has an extensive theoretical analysis. It represents  a solid step forward in the learning of dynamical systems models, and should spur further work on more difficult cases. The convex relaxation trick might be useful in other settings as well.
Some questions:
1. Predictive state representations (PSRs) are widely used in reinforcement learning to model learning from partially observed states. Does your work have any bearing on the learnability of PSRs, a longstanding challenge in the field.
2. There is quite a bit of work on kernelizing Kalman filters in various ways to obtain nonlinear dynamical systems. Does your approach extend to any of these extended models?
The approach is largely based on the simple projected gradient descent approach, but one wonders whether proximal gradient tricks that are so successful elsewhere (e.g, ADAGRAD, mirror prox etc.) would also be helpful here. In other words, can one exploit the geometry of the space to accelerate convergence?


A New Alternating Direction Method for Linear Programming
It is well known that, for a linear program (LP) with constraint matrix A ∈ R m×n , the Alternating Direction Method of Multiplier converges globally and linearly at a rate O(( A 2 F + mn) log(1/ )). However, such a rate is related to the problem dimension and the algorithm exhibits a slow and fluctuating "tail convergence" in practice. In this paper, we propose a new variable splitting method of LP and prove that our method has a convergence rate of O( A 2 log(1/ )). The proof is based on simultaneously estimating the distance from a pair of primal dual iterates to the optimal primal and dual solution set by certain residuals. In practice, we result in a new first-order LP solver that can exploit both the sparsity and the specific structure of matrix A and a significant speedup for important problems such as basis pursuit, inverse covariance matrix estimation, L1 SVM and nonnegative matrix factorization problem compared with the current fastest LP solvers.

This paper develops a novel alternating direction based method for linear programming problems. The paper presents global convergence results, and a linear rate, for their algorithm. As far as I could see, the mathematics appears to be sound, although I did not check thoroughly. Numerical experiments were also presented that support the practical benefits of this new approach; this new algorithm is compared with two other algorithms and the results seem favorable.
Note that the authors call their algorithm FADMM - my suggestion is that the authors choose a different acronym because (i) there are several other ADMM variants already called FADMM, and (ii) this is an ADMM for LP so it might be more appropriate to call it something like e.g., LPADMM, which is a more descriptive acronym.
The convergence results and numerical experiments are good, but the thing that I liked most about this paper is that the algorithm seems very practical because the first ADMM subproblem can be solved *inexactly*. This is a very nice feature. The authors also put effort to explain how to solve this first subproblem efficiently from a practical perspective (Section 5), which I felt was an important contribution of the paper.
One of the reasons that I did not give this paper a higher score is that there are a couple of typos in the paper and the descriptions and language could be clearer in places. For example, $n_b$ is used in (1) but is not defined; $n_f$ is used at the end of Section 2.1 but is not defined; the Hessian in Section 2.1 is not correct; throughout the paper the authors use minus signs instead of hyphens (e.g., $n-$dimensional should be $n$-dimensional, $l_1-$regularized should be $l_1$-regularized etc); the bibliography has several typos especially with $ signs missing: [4] should have $\ell_1$-problems; [11] D and R should be capitalized 'Douglas-Rachford'; [16,17] should be capitalized ADMM. All these are minor, but the bar for NIPS is very high and these should be corrected, and the paper should be thoroughly proofread for English mistakes.
Overall I liked this paper.


Revisit Fuzzy Neural Network: Demystifying Batch Normalization and ReLU with Generalized Hamming Network
We revisit fuzzy neural network with a cornerstone notion of generalized hamming distance, which provides a novel and theoretically justified framework to re-interpret many useful neural network techniques in terms of fuzzy logic. In particular, we conjecture and empirically illustrate that, the celebrated batch normalization (BN) technique actually adapts the "normalized" bias such that it approximates the rightful bias induced by the generalized hamming distance. Once the due bias is enforced analytically, neither the optimization of bias terms nor the sophisticated batch normalization is needed. Also in the light of generalized hamming distance, the popular rectified linear units (ReLU) can be treated as setting a minimal hamming distance threshold between network inputs and weights. This thresholding scheme, on the one hand, can be improved by introducing double-thresholding on both positive and negative extremes of neuron outputs. On the other hand, ReLUs turn out to be non-essential and can be removed from networks trained for simple tasks like MNIST classification. The proposed generalized hamming network (GHN) as such not only lends itself to rigorous analysis and interpretation within the fuzzy logic theory but also demonstrates fast learning speed, well-controlled behaviour and state-of-the-art performances on a variety of learning tasks.

The authors use a notion of generalized hamming distance, to shed light on the success of Batch normalization and ReLU units.
After reading the paper, I am still very confused about its contribution. The authors claim that generalized hamming distance offers a better view of batch normalization and relus, and explain that in two paragraphs in pages 4,5. The explanation for batch normalization is essentially contained in the following phrase:
“It turns out BN is indeed attempting to compensate for deficiencies in neuron outputs with respect to GHD. This surprising observation indeed adheres to our conjecture that an optimized neuron should faithfully measure the GHD between inputs and weights.”
I do not understand how this is explaining the effects or performance of batch normalization.
The authors then propose a generalized hamming network, and suggest that "it demystified and confirmed effectiveness of practical
techniques such as batch normalization and ReLU".
Overall, this is a poorly written paper, with no major technical contribution, or novelty, and does not seem to provide any theoretical  insights on the effectiveness of BN or ReLUs. Going beyond the unclear novelty and technical contribution, the paper is riddled with typos, grammar and syntax mistakes (below is a list from just the abstract and intro).
This is a clear rejection.
Typos and grammar/syntax mistakes:
—— abstract ——
generalized hamming network (GNN)
-> generalized hamming network (GHN)
 GHN not only lends itself to rigiour analysis
->  GHN not only lends itself to rigorous analysis
“but also demonstrates superior performances”
-> but also demonstrates superior performance
——
—— intro ——
“computational neutral networks”
-> computational neural networks
“has given birth”
-> have given birth
“to rectifying misunderstanding of neural computing”
-> not sure what the authors are trying to say
Once the appropriate rectification is applied ,
-> Once the appropriate rectification is applied,
the ill effects of internal covariate shift is automatically eradicated
-> the ill effects of internal covariate shift are automatically eradicated
The resulted learning process
-> The resulting learning process
lends itself to rigiour analysis
-> lends itself to rigorous analysis
the flexaible knowledge
-> the flexible knowledge
are equivalent and convertible with other
->  are equivalent and convertible with others, or other architectures?
successful applications of FNN
-> successful applications of FNNs


Interpretable and Globally Optimal Prediction for Textual Grounding using Image Concepts
Textual grounding is an important but challenging task for human-computer interaction, robotics and knowledge mining. Existing algorithms generally formulate the task as selection from a set of bounding box proposals obtained from deep net based systems. In this work, we demonstrate that we can cast the problem of textual grounding into a unified framework that permits efficient search over all possible bounding boxes. Hence, the method is able to consider significantly more proposals and doesn't rely on a successful first stage hypothesizing bounding box proposals. Beyond, we demonstrate that the trained parameters of our model can be used as word-embeddings which capture spatial-image relationships and provide interpretability. Lastly, at the time of submission, our approach outperformed the current state-of-the-art methods on the Flickr 30k Entities and the ReferItGame dataset by 3.08% and 7.77% respectively.

Summary
The approach proposes a simple method to find the globally optimal (under the formulation) box in an image which represents the grounding of the textual concept. Unlike previous approaches which adopt a two stage pipeline where region proposals are first extracted and then combined into grounded image regions (see [A] for example), this approach proposes a formulation which finds the globally optimal box for a concept. The approach assumes the presence of spatial heat maps depicting various concepts, and uses priors and geometry related information to construct an energy function, with learnable parameters for combining different cues. A restriction of the approach is that known concepts can only be combined linearly with each other (for instance the score map of “dirt bike” is “dirt” + “bike” with the learn weighting ofcourse), but this also allows for optimal inference for the given model class. More concretely the paper proposes an efficient technique based on [22] to use branch and bound for efficient sub-window search. Training is straightforward and clean through cutting-plane training of structured SVM. The paper also shows how to do efficient loss augmented inference during SVM training which makes the same branch and bound approach applicable to cutting-plane training as well. Finally, results are shown against competitive (near- state of the art approaches) on two datasets where the proposed approach is shown to outperform the state of the art.
Strengths
- Approach alleviates the need for a blackbox stage which generates region proposals.
- The interpretation of the weights of the model and the concepts as word embeddings is a neat little tidbit.
- The paper does a good job of commenting on cases where the approach fails, specifically pointing out some interesting examples such as “dirt bike” where the additive nature of the feature maps is a limitation.
- The paper has some very interesting ideas such as the use of the classic integral images technique to do efficient inference using branch and bound, principled training of the model via. a clean application of Structural SVM training with the cutting plane algorithm etc.
Weakness
1. Paper misses citing a few relevant recent related works [A], [B], which could also benefit from the proposed technique and use region proposals.
2. Another highly relevant work is [C] which does efficient search for object proposals in a similar manner to this approach building on top of the work of Lampert et.al.[22]
3. It is unclear what SPAT means in Table. 2.
4. How was Fig. 6 b) created? Was it by random sub-sampling of concepts?
5. It would be interesting to consider a baseline which just uses the feature maps (used in the work, say shown  in Fig. 2) and the phrases and simply regresses to the target coordinates using an MLP. Is it clear that the proposed approach would outperform it? (*)
6. L130: It was unclear to me how the geometry constraints are exactly implemented in the algorithm, i.e. the exposition of how the term k2 is computed was uncler. It would be great to provide details. Clear explanation of this seems especially important since the performance of the system seems highly dependent on this term (as it is trivial to maximize the sum of scores of say detection heat maps by considering the entire image as the set).
Preliminary Evaluation
The paper has a neat idea which is implemented in a very clean manner, and is easy to read. Concerns important for the rebuttal are marked with (*) above.
[A] Hu, Ronghang, Marcus Rohrbach, Jacob Andreas, Trevor Darrell, and Kate Saenko. 2016. “Modeling Relationships in Referential Expressions with Compositional Modular Networks.” arXiv [cs.CV]. arXiv. http://arxiv.org/abs/1611.09978.
[B] Nagaraja, Varun K., Vlad I. Morariu, and Larry S. Davis. 2016. “Modeling Context Between Objects for Referring Expression Understanding.” arXiv [cs.CV]. arXiv. http://arxiv.org/abs/1608.00525.
[C] Sun, Qing, and Dhruv Batra. 2015. “SubmodBoxes: Near-Optimal Search for a Set of Diverse Object Proposals.” In Advances in Neural Information Processing Systems 28, edited by C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, 1378–86. Curran Associates, Inc.


Learning Non-Gaussian Multi-Index Model via Second-Order Stein's Method
We consider estimating the parametric components of semiparametric multi-index models in high dimensions. To bypass the requirements of Gaussianity or elliptical symmetry of covariates in existing methods, we propose to leverage a second-order Stein's method with score function-based corrections. We prove that our estimator achieves a near-optimal statistical rate of convergence even when the score function or the response variable is heavy-tailed. To establish the key concentration results, we develop a data-driven truncation argument that may be of independent interest. We supplement our theoretical findings with simulations.

Summary: This article studies the estimation problem of single index and multiple index models in high dimensional setting. Using the formulation of second order Stein's lemma with sparsity assumptions, they propose an estimator formulated as a solution to a sparsity constrained semi-definite programming. The statistical rate of convergence of the estimator is derived and a numerical example is given.
Overall, I find the article interesting and useful, but their emphasis on "heavy tail" case and also partly non-Gaussianity is rather intriguing. Looking at the details, the main contribution the authors claim seems to be related to an extension of an optimality result for the estimation of sparse PCA from sub-Gaussian case to essentially sub-Exponential case through a Bernstein inequality. It would be helpful to clarify this point better.
Detailed comments are summarized below.
(a) Stein's lemma: I think one of the first references to an estimator based on Stein's lemma is H\"ardle and Stoker (1989), using the so-called "method of average derivatives", which does not require Gaussianity.
(b) Derivative condition: Normally the condition on the second derivative would be stronger than that on the first derivative, yet, the article suggests the other way around. Is there any other place to pay the price, or does it really provide more general framework?
(c) Heavy tail case: The notation of "heavy-tail" (as a pareto-like tail to most readers, I suspect, e.g. Barthe et al., 2005) is confusing and misleading in this context. Especially, I would think the heavy tail problem in economics has more to do with extremes and I doubt that the new approach is applicable in that context. Hence, I would suggest to remove the reference to the heavy-tail part, and instead make a better link to the contributions in terms of a relaxation of the tail conditions.
(d) Moment condition: In fact, I wonder how the moment condition on the score function could be translated in terms of the standard moment conditions (and tail conditions) on the original random variables. It would be helpful to demonstrate this point, and perhaps give a condition to make those assumptions comparable.
(e) Relation to Sparse PCA: Given the proximity to the formulation of the estimator to sparse PCA, I was imagining that a similar technique of the proofs from sparse PCA would have been used here, (except utilizing another (Berstein-type) concentration inequality), however, the relation to sparse PCA was presented as a consequence of their results. Then, in addition to the similarity to sparse PCA, could you also demonstrate a fundamental difference in techniques/considerations, if any, (other than the moment condition, or to overcome the moment condition?) required to deal with estimation problems of single index and multiple index models?
(f) Numerical algorithm: The algorithm to compute the estimator is mentioned only in Section 5. I would suggest to include some computational details in Section 3 as well, perhaps after (3.6).
(g) Minimum eigenvalue condition: I suppose without sparsity assumption, the minimum eigenvalue (line 174, p.5) should be zero for high dimensional case when d > k. Then, instead of "Finally" to introduce this condition, which seems to suggest that they are unrelated, would the connection need to be emphasized here as well?
Minor comments and typos:
(i) line 54, p2: (... pioneered by Ker-Chau Li, ...): From the flow of the arguments, it would be natural to add the references [20,21,22] here.
(ii) line 75, p2: I understood "SDP" as "semi-definite programming" in the end, but it would be better to be clear from the beginning here.
(iii) equation (4.1) p.4: typo.
(iv) heck references: the style of references is not coherent.
References:
Barthe, F., Cattiaux, P. and Roberto, C. (2005) Concentration for independent random variables with heavy tails, Applied Mathematics Research Express (2). 39-60.
H\"ardle, W. and Stoker, T.M. (1989) Investigating smooth multiple regression by the method of average derivatives, JASA 84 (408), 986-995.


Practical Bayesian Optimization for Model Fitting with Bayesian Adaptive Direct Search
Computational models in fields such as computational neuroscience are often evaluated via stochastic simulation or numerical approximation. Fitting these models implies a difficult optimization problem over complex, possibly noisy parameter landscapes. Bayesian optimization (BO) has been successfully applied to solving expensive black-box problems in engineering and machine learning. Here we explore whether BO can be applied as a general tool for model fitting. First, we present a novel hybrid BO algorithm, Bayesian adaptive direct search (BADS), that achieves competitive performance with an affordable computational overhead for the running time of typical models. We then perform an extensive benchmark of BADS vs. many common and state-of-the-art nonconvex, derivativefree optimizers, on a set of model-fitting problems with real data and models from six studies in behavioral, cognitive, and computational neuroscience. With default settings, BADS consistently finds comparable or better solutions than other methods, including 'vanilla' BO, showing great promise for advanced BO techniques, and BADS in particular, as a general model-fitting tool.

This paper presents a new optimization methods that combines Bayesian optimization applied locally with concepts from MADS to provide nonlocal exploration. The main idea of the paper is to find an algorithm that is suitable for the range of functions that are slightly expensive, but not enough to require the sample efficiency of standard Bayesian optimization. The authors applied this method for maximum likelihood computations within the range of a ~1 second.
      A standard critique to Bayesian optimization methods is that they are very expensive due to the fact that they rely on a surrogate model, like a Gaussian process that has a O(n^3) cost. The method presented in this paper (BADS) also rely on a GP. This paper solves the issue by computing the GP only of a local region, limited to 50+10D points.
      The paper ignores all the work that has been done in Bayesian optimization with much more efficient surrogate models, like random forests [A], Parzen estimators [B] or treed GPs [7], where available software shows that the computational cost is comparable to the one from BADS. It is known that those methods have worse global performance than GP based BO for problems in R^n, but given that this method uses local approximation, I would assume that the performance per iteration is also lower that GP-BO.
      Furthermore, because the main objective of BO is sample efficiency, some of the problems presented here could be solved with 50-100 iterations. Thus, being even more efficient that BADS as they would not require extra steps. In fact, optimized BO software [C] has a computational cost similar to the one reported here for BADS for the first 100 iterations. Note that [C] already have the rank-one updates implemented as suggested in section 3.2.
      The way the results are presented leaves some open questions:
      - Is the error tolerance relative or absolute? Are the problems normalized? What does it mean an error larger than 10? Is it 10%?
      - Given the fact that at the end of the plot, there is still 1/3 of the functions are unsolved. Without seeing the actual behaviour of the optimizers for any function, it is impossible to say if they were close, stuck in a local optimum, or completely lost...
      - Why using different plots for different problems? Why not doing the noisy case versus evaluations?
      - How is defined the "...effective performance of BADS by accounting for the extra cost..." from Figure 2?
      - If BO is used as a reference and compared in the text in many parts, why it is not included in the experiments? If the authors think it should not naively included for the extra cost, they could also "account for the extra cost", like in Figure 2.
      There is also a large set of optimization algorithms, mainly in the evolutionary computation community, that relies on GPs and similar models for local or global modeling. For example: [D, E] and references therein.
      [A] Frank Hutter, Holger Hoos, and Kevin Leyton-Brown (2011). Sequential model-based optimization for general algorithm configuration, Learning and Intelligent Optimization
      [B] Bergstra, James S., Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. "Algorithms for hyper-parameter optimization." In Advances in Neural Information Processing Systems, pp. 2546-2554. 2011.
      [C] Ruben Martinez-Cantin (2014) BayesOpt: A Bayesian Optimization Library for Nonlinear Optimization, Experimental Design and Bandits. Journal of Machine Learning Research, 15:3735-3739.
      [D] Zhou, Zongzhao, Yew Soon Ong, Prasanth B. Nair, Andy J. Keane, and Kai Yew Lum. "Combining global and local surrogate models to accelerate evolutionary optimization." IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 37, no. 1 (2007): 66-76.
      [E] Jin, Yaochu. "Surrogate-assisted evolutionary computation: Recent advances and future challenges." Swarm and Evolutionary Computation 1, no. 2 (2011): 61-70.


GibbsNet: Iterative Adversarial Inference for Deep Graphical Models
Directed latent variable models that formulate the joint distribution as p(x, z)

The authors proposed an extension over the Adverserially Learnt Inference(ALI) GAN that cycles between the latent and visible space for a few steps. The model iteratively refines the latent distribution by alternating the generator and approximate inference model in a chain computation. A joint distribution of both latent and visible data is then learnt by backpropagating through the iterative refinement process. The authors empirically demonstrated their model on a few imprint tasks.
        Strength:
           - The paper is well-organized and is easy to follow.
           - The experimental results on semi-supervised learning are encouraging. (more on that see the comment below. )
        Weakness:
           - The main objection I have with the paper is that the authors did not put in any effort to quantitatively evaluate their newly proposed GAN training method. Comparing the inception score on CIFAR-10 with ALI and other benchmark GAN methods should be a must. The authors should also consider estimating the actual log likelihood of their GAN model by running the evaluation method proposed:"On the Quantitative Analysis of Decoder-Based Generative Models", Wu et al. The bottom line is that without appropriate quantitative analysis, it is hard to evaluate how well the proposed method does in general. What should help is to see a plot where an x-axis is the number of Gibbs steps and y-axis as one of the quantitative measures.
           - The improvement of the proposed method seems to be very marginal compared to ALI. The appropriate baseline comparison should be a deeper ALI model that has 2N number of layers. The "Gibbs chain" used throughout this paper is almost like a structured recurrent neural network with some of the intermediate hidden layers actually represents x and z. So, it is unfair to compare a 3-step GibbsNet with a 2-layer feedforward ALI model.
===================
After I have read the rebuttal from the author, I have increased my score to reflect the new experiments conducted by the authors. The inception score results and architecture comparisons have addressed my previous concerns on evaluation.
I am still concerned regarding the experimental protocols. The exact experimental setup for the semi-supervised learning results was not explained in detail. I suspect the GibbsNet uses a very different experimental protocol for SVHN and MNIST than the original ALI paper. It is hard to evaluate the relative improvement over ALI if the protocols are totally different. It is necessary to include all the experimental details in a future revision for reproducibility.


The Scaling Limit of High-Dimensional Online Independent Component Analysis
We analyze the dynamics of an online algorithm for independent component analysis in the high-dimensional scaling limit. As the ambient dimension tends to infinity, and with proper time scaling, we show that the time-varying joint empirical measure of the target feature vector and the estimates provided by the algorithm will converge weakly to a deterministic measured-valued process that can be characterized as the unique solution of a nonlinear PDE. Numerical solutions of this PDE, which involves two spatial variables and one time variable, can be efficiently obtained. These solutions provide detailed information about the performance of the ICA algorithm, as many practical performance metrics are functionals of the joint empirical measures. Numerical simulations show that our asymptotic analysis is accurate even for moderate dimensions. In addition to providing a tool for understanding the performance of the algorithm, our PDE analysis also provides useful insight. In particular, in the high-dimensional limit, the original coupled dynamics associated with the algorithm will be asymptotically "decoupled", with each coordinate independently solving a 1-D effective minimization problem via stochastic gradient descent. Exploiting this insight to design new algorithms for achieving optimal trade-offs between computational and statistical efficiency may prove an interesting line of future research.

The paper studies the high-dimensional scaling limit of a stochastic update algorithm for online Independent Component Analysis.
The main result of the paper is an exact characterization of the evolution of the joint empirical distribution of the estimates
output by the algorithm and the signal to be recovered, when the number of observations scales linearly with the dimension of the problem.
The authors argue that in the limit, this joint distribution is the unique solution to a certain partial differential equation,
from which the performance of the algorithm can be predicted, in accordance with the provided simulations.
1- Overall, the result is fairly novel, and provides interesting insight into the behavior of stochastic algorithms in this non-convex problem.
The paper is written in a fairly clear manner and is (mostly) easy to read.
My main concern is that the mathematics are written in a very informal way, so that it is not clear what
the authors have actually proved. There are no theorem or proposition statements.
E.g., under what conditions does the empirical measure have a weak limit?
Under what (additional) conditions does this limit have a density?
And when is this density the unique solution to the PDE?
The heuristic justification in the appendix is not very convincing either (see bullet 4).
2- A very interesting insight of the theory is that the p variables of the problem decouple in the limit,
where each variable obeys a diffusion equation independently of the others.
The only global aspect retained by these local dynamics is via the order parameters R and Q.
This phenomenon is standard in mean-field models of interacting particle systems, where the dynamics of the particles decouple when the order parameters converge to a limit.
The authors should probably draw this connection explicitly; the relevant examples are the Sherrington-Kirkpatrick model, rank-one estimation in spiked models, compressed sensing, high-dim. robust estimation…
3- The connection to the SDE line 175 should be immediate given that the PDE (5) is its associated Fokker-Planck equation.
The iteration (14) is easier seen as a discretization of the SDE rather than as some approximate dynamics solving the PDE. Therefore, the SDE should come immediately after the statement of the PDE, then the iteration.
4- The derivations in the appendix proceed essentially by means of a cavity (or a leave-one-out) method,
but end abruptly without clear conclusion. The conclusion seems to be the decoupled iteration (14), not the actual PDE.
This should be made clear, or otherwise, the derivation should be conducted further to the desired conclusion.
The derivation contain some errors and typos; I couldn't follow the computations
(e.g., not clear how line 207 was obtained, a factor c_k should appear next to Q_k^p on many lines...)

5-There are several imprecisions/typos in the notation:
line 36: support recover*y*
eq 2: sign inconsistency in front of \tau_k (- gradient, not +)
line 100: x_k should be x
line 133: u should be \xi (in many later occurrences)
eq 8: \tau not defined (is it the limit of the step sizes \tau_k?)
line 116: t = [k/p] should be k = [tp]


Invariance and Stability of Deep Convolutional Representations
In this paper, we study deep signal representations that are near-invariant to groups of transformations and stable to the action of diffeomorphisms without losing signal information. This is achieved by generalizing the multilayer kernel introduced in the context of convolutional kernel networks and by studying the geometry of the corresponding reproducing kernel Hilbert space. We show that the signal representation is stable, and that models from this functional space, such as a large class of convolutional neural networks, may enjoy the same stability.

The primary focus of the paper is CKN (convolutional kernel network) [13, 14]. In this manuscript the authors analyse the stability [w.r.t. C^1 diffeomorphisms (such as translation), in the sense of Eq. (4)] of the representation formed by CKNs. They show that for norm-preserving and non-expansive kernels [(A1-A2) in line 193] stability holds for appropriately chosen patch sizes [(A3)]. Extension from (R^d,+) to locally compact groups is sketched in Section 4.
The paper is nicely organized, clearly written, technically sound, combining ideas from two exciting areas (deep networks and kernels). The stability result can be of interest to the ML community.
-The submission would benefit from adding further motivation on the stability analysis. Currently there is only one short sentence (line 56-57: 'Finally, we note that the Lipschitz stability of deep predictive models was found to be important to get robustness to adversarial examples [7].') which motivates the main contribution of the paper.
-Overloading the \kappa notation in (A3) [line 193] might be confusing, it also denotes a function defining kernel K in Eq. (10).
-In the displayed equation between line 384 and 385, the second part ('and \forall v...') is superfluous; given the symmetry of kernel k, it is identical to the first constraint ('\forall u ...').
-Line 416: the definition of \phi is missing, it should be introduced in Eq. (10) [=<\phi(z),\phi(z')>_{H(K)}].
-Line 427-428: The inequality under '=' seems to also hold with equality, | ||z|| - ||z|| |^2 should be | ||z|| - ||z'|| |^2.
References:
[3,6,8,13,14,18,25-27,30,31]: page information is missing.
[9]: appeared -> Amit Daniely, Roy Frostig, Yoram Singer. Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity. Advances in Neural Information Processing Systems (NIPS), pages 2253-2261, 2016.
[17]: appeared -> Krikamol Muandet, Kenji Fukumizu, Bharath Sriperumbudur, Bernhard Sch{\"o}lkopf. Kernel Mean Embedding of Distributions: A Review and Beyond. Foundations and Trends in Machine Learning, 10(1-2): 1-141.
[19]: appeared -> Anant Raj, Abhishek Kumar, Youssef Mroueh, Tom Fletcher, Bernhard Sch{\"o}lkopf. International Conference on Artificial Intelligence and Statistics (AISTATS), PMLR 54:1225-1235, 2017.
[32]: accepted (https://2017.icml.cc/Conferences/2017/Schedule?type=Poster) -> Yuchen Zhang, Percy Liang, Martin Wainwright. Convexified Convolutional Neural Networks. International Conference on Machine Learning (ICML), 2017, accepted.


Alternating minimization for dictionary learning with random initialization
We present theoretical guarantees for an alternating minimization algorithm for the dictionary learning/sparse coding problem. The dictionary learning problem is to factorize vector samples y 1 , y 2 , . . . , y n into an appropriate basis (dictionary) A * and sparse vectors x 1 * , . . . , x n * . Our algorithm is a simple alternating minimization procedure that switches between 1 minimization and gradient descent in alternate steps. Dictionary learning and specifically alternating minimization algorithms for dictionary learning are well studied both theoretically and empirically. However, in contrast to previous theoretical analyses for this problem, we replace a condition on the operator norm (that is, the largest magnitude singular value) of the true underlying dictionary A * with a condition on the matrix infinity norm (that is, the largest magnitude term). Our guarantees are under a reasonable generative model that allows for dictionaries with growing operator norms, and can handle an arbitrary level of overcompleteness, while having sparsity that is information theoretically optimal. We also establish upper bounds on the sample complexity of our algorithm.
Erratum, August 7, 2019: An earlier version of this paper appeared in NIPS 2017 which had an erroneous claim about convergence guarantees with random initialization. The main resultTheorem 3 -has been corrected by adding an assumption about the initialization (Assumption B1).

This paper proposes and analyzes an alternating minimization-based algorithm to recover the dictionary matrix and sparse coefficient matrix in a dictionary learning setting.  A primary component of the contribution here comes in the form of an alternate analysis of the matrix uncertainty (MU) selector of Belloni, Rosenbaum, and Tsybakov, to account for worst-case rather than probabilistic corruptions.
Pros:
+ The flavor of the contribution here seems to improve (i.e., relax) the conditions under which methods like this will succeed, relative to existing works.  Specifically, the motivation and result of this work amounts to specifying sufficient conditions on the vectorized infinity norm of the unknown dictionary matrix, rather than its operator norm, under which provable recovery is possible.  This has the effect of making the method potentially less dependent on ambient dimensions, especially for "typical" constructions of the (incoherent) dictionaries such as certain random generations.
+ The alternate analysis of the MU selector is independently interesting.
Cons:
- It would be interesting to see some experimental validation of the proposed method, especially ones that investigate the claimed improvements in the conditions on the unknown dictionary relative to prior efforts.  In other words, do the other efforts that state results in terms of the operator norm fail in settings where this method succeeds?  Or are the methods all viable, but just a more refined analysis here?  This is hard to determine here, and should be explored a bit, I think.
- The paper is hard to digest, partly because of notation, and partly because of some pervasive grammatical and formatting issues:
    - Line 8 of algorithm 1, as written, seems to require knowledge of the true A,x quantities to compute.  In reality, it seems this should be related somehow to the samples {y} themselves.  Can this be written a bit more clearly?
    - Condition (c4) on page 6, line 206 is confusing as written.  The dimension of x is r, and it is s-sparse, so there are more than r options for *sets* of size s; this should be r-choose-s, I guess.  The subsequent conditions are apparently based on this kind of model, and seem to be correct.
    - Why include the under brace in the first equation of line 251 on page 7?  Also, repeating the LHS is a little non-standard.
    - The "infinite samples" analysis is a little strange to me, too.  Why not simply present and analyze the algorithm (in the main body of the paper) in terms of the finite sample case?  The infinite case seems to be an analytical intermediate step, not a main contribution in itself.
    - There are many sentence fragments that are hard to parse, e.g., "Whereas..." on line 36 page 2, "While..." on line 77 page 2, and "Given..." on line 131 page 4.