Solving Non-smooth Constrained Programs with Lower Complexity than O(1/"): A Primal-Dual Homotopy Smoothing Approach
We propose a new primal-dual homotopy smoothing algorithm for a linearly constrained convex program, where neither the primal nor the dual function has to be smooth or strongly convex. The best known iteration complexity solving such a non-smooth problem is O(" 1 ). In this paper, we show that by leveraging a local error bound condition on the dual function, the proposed algorithm can achieve a better primal convergence time of O " 2/(2+ ) log 2 (" 1 ) , where 2 (0, 1] is a local error bound parameter. As an example application of the general algorithm, we show that the distributed geometric median problem, which can be formulated as a constrained convex program, has its dual function non-smooth but satisfying the aforementioned local error bound condition with = 1/2, therefore enjoying a convergence time of O " 4/5 log 2 (" 1 ) . This result improves upon the O(" 1 ) convergence time bound achieved by existing distributed optimization algorithms. Simulation experiments also demonstrate the performance of our proposed algorithm.

The authors propose a primal dual homotopy smoothing algorithm for the minimization of a convex function over a convex compact set with additional equality constraints. Being based on the primal dual algorithm, the method requires an accuracy level which needs to be carefully tuned in practice and is critical for the convergence of the algorithm. It is difficult to parse Theorem 3.1. I would like to see an explicit convergence rate in terms of feasibility and suboptimality. In particular, the algorithm does not converge to the optimum, but only to an epsilon accurate solution (as in the universal primal-dual). This should be carefully discussed. In particular, the whole method should be put in perspective with “Frank-Wolfe Splitting via Augmented Lagrangian Method” for the problem in equation 16-17 and “A conditional Gradient Framework for Composite Convex Minimization with Applications to Semidefinite Programming”. In particular, the latter seem to propose the same smoothing technique. While they obtain a slower rate, their subproblem is easier to solve and they prove convergence to the optimal value and feasibility. Therefore, a discussion on the differences of the two approaches and an empirical comparison would be important. -------------------------------------------------------------------- After the rebuttal I decided to increase my score to 6 as the approach can deal with non smooth f as well.


Zeroth-order (Non)-Convex Stochastic Optimization via Conditional Gradient and Gradient Updates
In this paper, we propose and analyze zeroth-order stochastic approximation algorithms for nonconvex and convex optimization. Specifically, we propose generalizations of the conditional gradient algorithm achieving rates similar to the standard stochastic gradient algorithm using only zeroth-order information. Furthermore, under a structural sparsity assumption, we first illustrate an implicit regularization phenomenon where the standard stochastic gradient algorithm with zeroth-order information adapts to the sparsity of the problem at hand by just varying the stepsize. Next, we propose a truncated stochastic gradient algorithm with zeroth-order information, whose rate depends only poly-logarithmically on the dimensionality.

The paper considers the problem of extending the Frank-Wolfe method to scenarios that only an unbiased estimate of the function value is available, i.e.  we don't have access to any (stochastic) gradient o higher order information and only (an unbiased version of) the function values can be obtained from the oracle. The authors use Nestrov's smoothing method to find a (stochastic) proxy of the gradient and use such a proxy to design their algorithms.  I have carefully read the paper and I have two major issues with this paper. While I find the problem and the solutions interesting and somehow novel, I believe the following two issues have to be resolved before I can give a higher mark for accepting the paper.   1) Related work and giving credit to previous work: Algorithm 1 in the paper is exactly the algorithm SFW proposed in [12,24] with the difference that the stochastic gradients are replaced by unbiased estimators of the smoothed function (from equation (1.3)). While I find the smoothing step crucial and novel due to having only zero-order information, I believe that the authors should have given credit to [12] or [24] by specifying what steps Algorithm 1 has different with respect to those algorithms.    Furthermore, the problem of stochastic FW (with first order information) has been studied recently in various papers and the authors have failed to review recent progress and related work on this topic. For example, the following paper by Mokhtari et al proposes a stochastic Frank-Wolfe method for both convex and structured non-convex problems with O(1/eps^3) samples (potentially dimension-independent): https://arxiv.org/abs/1804.09554  I expect that in the revised version the authors highlight the differences of their algorithms with the recent results (perhaps the authors can add a section called related work). I understand that the authors consider zero-order information, but in principle by using the smoothing trick one can come up with (unbiased estimates of) first order information of a sufficiently close function (as done in equation 1.3 in the paper).  2) To get the O(d/eps^2) bound the authors add a proximal step (equation 2.15) and solve this step by using the FW method (Algorithm 2). Well, I can not really call the whole algorithm (Algorithm 3) a stochastic and projection-free conditional gradient method. The authors are just solving an expensive quadratic optimization problem by the FW method and claim that the overall algorithm is a FW-type method.  Indeed, one can solve any quadratic projection problem with FW. For example, consider projected gradient descent. We can solve the projection step with FW and claim that we have a projection-free method with rate O(1/eps^2). Can the authors comment on this?


The challenge of realistic music generation: modelling raw audio at scale
Realistic music generation is a challenging task. When building generative models of music that are learnt from data, typically high-level representations such as scores or MIDI are used that abstract away the idiosyncrasies of a particular performance. But these nuances are very important for our perception of musicality and realism, so in this work we embark on modelling music in the raw audio domain. It has been shown that autoregressive models excel at generating raw audio waveforms of speech, but when applied to music, we find them biased towards capturing local signal structure at the expense of modelling long-range correlations. This is problematic because music exhibits structure at many different timescales. In this work, we explore autoregressive discrete autoencoders (ADAs) as a means to enable autoregressive models to capture long-range correlations in waveforms. We find that they allow us to unconditionally generate piano music directly in the raw audio domain, which shows stylistic consistency across tens of seconds.

The authors claim that there is no suitable metric to evaluate the quality of the generated audio, which is plausible, so they listened to the audio and evaluated on their own. The only shortcoming here is that no systematic and blind listening test has been conducted yet. The authors themselves might be biased and thus, the capabilities of the proposed approach cannot be considered as fully proven from a scientific perspective. However, a link to the audio is provided so that the readers can convince themselves from the proposed method.  Minor comments: -"nats per timestep": should be defined -p. 3, l. 115-116: The reviewer is not sure, whether the term "hop size" is suitable to describe the ratio between the sample rates. -p. 7, l. 278: For AMEA models, PBT turns (a comma helps the reader) -References: [12], [22], [43] are using "et al." even though the author list is not much longer. All authors should be named to be consistent.   Quality: The authors address a prevailing research topic and has some novel contributions (see below) to the field of audio and music generation. The topic is going along with the recent advances in deep architectures for audio generation.  Clarity: The paper is well written and the proposed methods are explained carefully. A good description of the state-of-the-art in audio generation and their shortcomings are included. The references are complete.  Originality: To the knowledge of the reviewer, in this contribution, music is modelled directly on the waveform for the first time, modelling also long-term dependencies related to consistent musical expression and style. The conditioning signal is learnt with an autoencoder. Moreover, argmax autoencoder is proposed as an alternative (faster and more robust) for the quantisation step employed in vector quantisation variational autoencoders, avoiding the "codebook collapse" problem.  Significance: A systematic evaluation is missing (see above). However, a link to audio samples is provided in the paper and the proposed approach is well-founded.   Confidence: The reviewer is confident with audio processing and machine learning/deep learning and has many peer-reviewed publications in journals and conference proceedings in this field.


Efficient Gradient Computation for Structured Output Learning with Rational and Tropical Losses
Many structured prediction problems admit a natural loss function for evaluation such as the edit-distance or n-gram loss. However, existing learning algorithms are typically designed to optimize alternative objectives such as the cross-entropy. This is because a naïve implementation of the natural loss functions often results in intractable gradient computations. In this paper, we design efficient gradient computation algorithms for two broad families of structured prediction loss functions: rational and tropical losses. These families include as special cases the n-gram loss, the edit-distance loss, and many other loss functions commonly used in natural language processing and computational biology tasks that are based on sequence similarity measures. Our algorithms make use of weighted automata and graph operations over appropriate semirings to design efficient solutions. They facilitate efficient gradient computation and hence enable one to train learning models such as neural networks with complex structured losses.

This paper proposes finite-state approaches for computing a specific loss function (eq 1) and its gradient.  The approach targets improving the training-time efficiency of a finite-state transducer (FST) structured model (e.g., a chain structured CRF) under an rich family FST structured loss function (e.g., edit distance against a gold reference output string). The key contribution of the paper is the efficient computation of the family of structured losses.   Finite-state methods are an obvious choice for performing these computations - it will be familiar/obvious to many NLP researchers (like myself), but perhaps less so in a machine learning venue such as NIPS/ICML. This paper goes beyond the basic construction and includes some useful discussion and correctness details.   Other comments =============  - Why limit to loss functions of the form (1)? It might be useful to consider expected loss, e.g., Li & Eisner (2009) -- adding some discussion of this paper would be a useful pointer for readers.  - If I'm not mistake, the loss function (1) is equivalent to soft-max margin (Gimpel & Smith, 2010; additionally Kevin Gimpel's thesis) and reward-augmented maximum likelihood (RAML; Norouzi et al., 2016). These references contain strong empirical evidence to support the choice of this loss, which are complementary to the theoretical results of reference 7 in the submission.  - Have you considered a REINFORCE-style approximation as another experimental baseline (this is the approach taken in reference 26 in the submission, if I'm not mistaken)? This approach gives an unbiased (but high variance) estimate of a similar (although not the same) objective function (expected reward, which is essentially Li&Eisner'09). If I'm correct that loss function (1) is the same as RAML, then the REINFORCE-style estimator can be modified to compute this loss as well. Another reasonable approximation is to take the K-best outputs as used in Gimpel & Smith (2010), this type of approximation is very common in NLP.  References ========= (Li&Eisner'09) http://www.cs.jhu.edu/~jason/papers/li+eisner.emnlp09.pdf (Gimpel & Smith, 2010) http://www.aclweb.org/anthology/N10-1112 (Norouzi et al 2016) https://papers.nips.cc/paper/6547-reward-augmented-maximum-likelihood-for-neural-structured-prediction.pdf (Kevin Gimpel's thesis) http://ttic.uchicago.edu/~kgimpel/papers/gimpel_thesis.pdf   Post discussion and author response ============================  The authors have addressed my concerns in their response. These concerns were mainly about connections to other work and additional suggestion of experimental comparisons that could be made to strengthen the paper. The empirical evaluation in this paper is kind of silly: they compare an (truly) exponential-time algorithm to their polynomial-time algorithm... no surprise poly-time is much better. I suggested a randomized approximation scheme (based on Monte Carlo sampling / REINFORCE) as a baseline. They didn't bite...  Re: "novelty" of the paper: Most people familiar with finite-state methods (e.g., NLP folks such as myself) would come up with this same approach.  However, these authors did a reasonable good job of "packing up" their method, which will be useful to researcher.  I do wish there was more "tutorial material" to help orient readers that don't know finite-state methods (since this is probably the target audience of the paper). To that end, I **really** like R2's suggestion to include a detailed worked example in the appendix.  I also strongly encourage the authors to release their code in order maximize the impact of their work.  I was too harsh about novelty in my initial assessment so I have increased my overall score to 7.   In regards to some of R1 comments: I wanted to point out that the type of "state-space augmentation" found by finite-state methods is more fine grained than the junction-tree method used on graphical models. There are many cases where the finite-state approach yields exponentially smaller dynamic programs than the best one could do with graphical models. This is discussed in the Sum-Product networks paper (https://arxiv.org/pdf/1202.3732.pdf).  The authors might want to make that connection in their paper.


Forward Modeling for Partial Observation Strategy Games -A StarCraft Defogger
We formulate the problem of defogging as state estimation and future state prediction from previous, partial observations in the context of real-time strategy games. We propose to employ encoder-decoder neural networks for this task, and introduce proxy tasks and baselines for evaluation to assess their ability of capturing basic game rules and high-level dynamics. By combining convolutional neural networks and recurrent networks, we exploit spatial and sequential correlations and train well-performing models on a large dataset of human games of StarCraft R : Brood War R † . Finally, we demonstrate the relevance of our models to downstream tasks by applying them for enemy unit prediction in a state-of-the-art, rule-based StarCraft bot. We observe improvements in win rates against several strong community bots.

Update after rebuttal: the new details added by the authors in the rebuttal helped clarify some of the doubts I have, so I have changed my score accordingly.  -------------------------------- Summary:  The paper presents a deep learning approach to predict some hidden features in a StarCraft game, such as existence of opponent building types and unit types at any location and time in the game. The paper suffers from lack of details on both the game, approach and the experiment setup, so it is quite hard to read. More  Strengths: The paper presents new ideas in applying deep learning in the context of strategy games, which is interesting.  Weaknesses: The paper does not provide enough details on the game and various concepts in the game to justify the design of the proposed infrastructure. Due to the lack of sufficient details on the game to justify and inform the design of the deep network architecture, it is hard to convince readers why the approach works or does not work. The reproducibility is therefore affected. The take-away from the paper or generalization beyond StarCraft is also limited.  Other general comments: - Tables and figures should appear near where they are mentioned, so Table 4 and Figure 4 should not be in the appendix. - Too many papers referenced are from non-peer reviewed websites, such as Arxiv and CoRR, weakening the paper in terms of argumentation. The authors are highly encouraged to replace these papers by their peer-reviewed counterparts - The selected baselines are too simple. They are all simple, rule-based, and myopic, with no attempt to learn anything from the provided training data, which the authors’ architecture makes use of. It would be a fairer comparison should authors have tried to design a little more sophisticated baselines that incorporate some form of learning (via classical prediction models, for example).  Other detailed comments: - Various statements about the game and how players behave/act in the game miss explanation and/or supporting reference     * Page 1, line 31: “Top-level human players perform about 350 actions per minute“    * Page 3, line 86: “... humans generally make forward predictions on a high level at a long time-scale; ...”     * Page 6, line 240-241: “Since units often move very erratically...”: not really, units often move according to some path. This also illustrates why the selected baselines are too weak; they make no use of the training data to try to infer the path of the moving units. - Page 2, line 81: Full state is not defined properly. What are included in a full state? - Page 3, line 96: what are “walk tiles”? - Page 3, line 119: What is “the faction of both players”? - (minor) Figure 4 does not print well in black and white - Page 3, line 131-132: The author refers to the g_op_b task that is introduced later in the paper. It would be better if the authors can describe such concepts in a section devoted to description of the games and related tasks of interest. - Page 4, line 139: “h” is not defined yet. - Page 5, line 208-210: What game rules were used for this baseline? In general, how do game rules help? - Page 7, line 271-276: This part needs more elaboration on what modules are available in the bots, and how they can use the enhanced information to their advantage in a game. - Page 7, line 286-287: The reason to exclude Zerg is not convincing nor well supported externally. The author is encouraged to include a Zerg bot regardless, then in the discussion section, explain the result and performance accordingly.


Efficient Neural Network Robustness Certification with General Activation Functions
Finding minimum distortion of adversarial examples and thus certifying robustness in neural network classifiers for given data points is known to be a challenging problem. Nevertheless, recently it has been shown to be possible to give a nontrivial certified lower bound of minimum adversarial distortion, and some recent progress has been made towards this direction by exploiting the piece-wise linear nature of ReLU activations. However, a generic robustness certification for general activation functions still remains largely unexplored. To address this issue, in this paper we introduce CROWN, a general framework to certify robustness of neural networks with general activation functions for given input data points. The novelty in our algorithm consists of bounding a given activation function with linear and quadratic functions, hence allowing it to tackle general activation functions including but not limited to four popular choices: ReLU, tanh, sigmoid and arctan. In addition, we facilitate the search for a tighter certified lower bound by adaptively selecting appropriate surrogates for each neuron activation. Experimental results show that CROWN on ReLU networks can notably improve the certified lower bounds compared to the current state-of-the-art algorithm Fast-Lin, while having comparable computational efficiency. Furthermore, CROWN also demonstrates its effectiveness and flexibility on networks with general activation functions, including tanh, sigmoid and arctan.

Summary: This paper proposes a general framework CROWN to efficiently certify robustness of neural networks with general activation functions.  CROWN adaptively bounds a given activation function with linear and quadratic functions, so it can tackle general activation functions including but not limited to the four popular choices: ReLU, tanh, sigmoid, and arctan. Experimental results demonstrate the effectiveness, efficiency, and flexibility of the proposed framework.  Quality: We are glad to find a work which conducts the efficiently certifying of the non-trivial robustness for general activation functions in neural networks. It is also interesting that the proposed framework can flexibly select upper bounds and lower bounds which can reduce the approximation error. This paper is of high quality. The problem is well-motivated, and the experimental comparisons and analysis are sufficient and complete. The proofs of the theorems and bounds are reasonable.     Clarity: This paper is very well written and enjoyable to read. The theory analysis and proof are clear and easy to follow.  Originality: This is an original research work especially of the robustness certification with the general activation functions, which may provide a good direction for the future research on this problem.  Overall impression, this is a high-quality paper with solid analysis, proof, and experimental results.


Approximating Real-Time Recurrent Learning with Random Kronecker Factors
Despite all the impressive advances of recurrent neural networks, sequential data is still in need of better modelling. Truncated backpropagation through time (TBPTT), the learning algorithm most widely used in practice, suffers from the truncation bias, which drastically limits its ability to learn long-term dependencies.The Real-Time Recurrent Learning algorithm (RTRL) addresses this issue, but its high computational requirements make it infeasible in practice. The Unbiased Online Recurrent Optimization algorithm (UORO) approximates RTRL with a smaller runtime and memory cost, but with the disadvantage of obtaining noisy gradients that also limit its practical applicability. In this paper we propose the Kronecker Factored RTRL (KF-RTRL) algorithm that uses a Kronecker product decomposition to approximate the gradients for a large class of RNNs. We show that KF-RTRL is an unbiased and memory efficient online learning algorithm. Our theoretical analysis shows that, under reasonable assumptions, the noise introduced by our algorithm is not only stable over time but also asymptotically much smaller than the one of the UORO algorithm. We also confirm these theoretical results experimentally. Further, we show empirically that the KF-RTRL algorithm captures long-term dependencies and almost matches the performance of TBPTT on real world tasks by training Recurrent Highway Networks on a synthetic string memorization task and on the Penn TreeBank task, respectively. These results indicate that RTRL based approaches might be a promising future alternative to TBPTT.

SUMMARY:  This paper proposes a Kronecker factored variant of the Real-Time Recurrent Learning algorithm (RTRL) proposed by Ollivier et al, called KF-RTRL. The main obstacle of using RTRL in place of Truncated Back-Propagation Through Time (BPTT) to train RNN is its computational complexity of O(n^4) where n is the number of units. To address this issue, the Unbiased Online Recurrent Optimization (UORO) algorithm proposed to approximate RTRL using a rank one matrix to approximate the matrix of derivatives of the hidden state h_t wrt the parameters (henceforth denoted by G_t, of size n x n^2), reducing the complexity to O(n^2) at the cost of introducing variance to the unbiased estimator of the gradient. Instead, KF-RTRL approximates G_t by the Kronecker product of a 1xn vector and an nxn matrix, to obtain an unbiased estimator with less variance with a complexity of O(n^3), thus trying to find a middle ground between UOFO and RTRL. The correctness of the proposed algorithm is analyzed but only informal theoretical arguments are given for the improvement in terms of variance wrt UORO. Still, the experiments on the synthetic copy/paste task and on the Penn tree bank show that KF-RTRL performs better than UORO in terms of convergence rate of the training algorithm.   OPINION:  The contribution is interesting and the idea to replace the restrictive rank one approximation of UORO by a more expressive factorization while still achieving better computational complexity than RTRL is worthwhile. Regarding clarity the paper could be first improved by a careful proof reading and polishing. The exposition of the results could also be improved: it is for example clumsy to defer the definition of the class of RNN for which Theorem 1 holds to Lemma 1, when Lemma 1 actually has not much to do with Theorem 1 (it only serve the argument that the class of RNN considered is not too restrictive); the actual definition of the class of RNN in Lemma 1 is also difficult to grasp/parse and would necessitate more discussion (see my questions below). I also find the arguments used to claim that the variance introduced by KF-RTRL is smaller than the one of UORO very 'hand-wavy' and I would have appreciated a more formal analysis. On the positive side, the experiment section indeed suggests that KF-RTRL is an attractive alternative to both BPTT and UORO and the code is directly supplied as supplementary material.   REMARKS/QUESTIONS:  - line 106-108: I don't understand what is meant here, could you clarify?   - Algorithm 1: It may be worth precising what the z^k_j are, what SGDopt is, and which norm is used.  - Lemma 1 - definition of the RNN class: What do the matrices W^1,...,W^R correspond to? Is it correct that for vanilla RNN with one hidden layer we would have r=1 and W^1 would be the concatenation of the input and recurrent matrices? How is the bias term of the hidden layer handled in this definition? Does this class encompasses multi-layer RNNs? I think more attention should be put on this definition and more discussion/intuition given.  - line 178: what are the distributions over u_{t-1} and A_{t-1} mentioned here?  - I think RTRL should be included in the comparison for the experiments in Sections 5.1 and 5.2. Is there a reason why it is not the case?   MINOR COMMENTS/TYPOS:  -line 51: that -> than -line 58: on -> in -line 164: this sentence is a bit misleading, maybe add 'in expectation' somewhere. -line 176: the phrasing is strange, maybe use 'since u_{t-1}' is a row vector' instead. -line 203: $UORO$ -> UORO -line 230: with out -> without -line 236: do you mean 'validation error' instead of 'RNN error'? -line 240: RHN -> RNN -footnote 1 p.6: it is not clear to me how this simplify the implementation.  -caption of Fig 1: an -> a -Fig 1 x-axis legend: what is 'data time'? -line 403: Equqtion


Neural Code Comprehension: A Learnable Representation of Code Semantics
With the recent success of embeddings in natural language processing, research has been conducted into applying similar methods to code analysis. Most works attempt to process the code directly or use a syntactic tree representation, treating it like sentences written in a natural language. However, none of the existing methods are sufficient to comprehend program semantics robustly, due to structural features such as function calls, branching, and interchangeable order of statements. In this paper, we propose a novel processing technique to learn code semantics, and apply it to a variety of program analysis tasks. In particular, we stipulate that a robust distributional hypothesis of code applies to both human-and machine-generated programs. Following this hypothesis, we define an embedding space, inst2vec, based on an Intermediate Representation (IR) of the code that is independent of the source programming language. We provide a novel definition of contextual flow for this IR, leveraging both the underlying data-and control-flow of the program. We then analyze the embeddings qualitatively using analogies and clustering, and evaluate the learned representation on three different high-level tasks. We show that even without fine-tuning, a single RNN architecture and fixed inst2vec embeddings outperform specialized approaches for performance prediction (compute device mapping, optimal thread coarsening); and algorithm classification from raw code (104 classes), where we set a new state-of-the-art.

This paper proposes to use programming code embeddings for individual statements to be used in all kinds of program analysis tasks, such as classification of type of algorithms, placing of code on a heterogeneous cpu-gpu architecture and scheduling of programs on gpus.  The novelty of the paper consists of the way the code embeddings are computed/trained. Instead of looking at code statements in high-level languages and following either data or control flow, the paper proposes to use statements at an intermediate-representation level (which are independent of the high-level language used) and take both data and control flow into account. As such, the proposed technique builds "contextual flow graphs" where the nodes are connected either through data or control flow edges. The nodes are variable or label identifiers.  The statements embeddings are computed using the skip-gram model on paths of the contextual flow graphs.  The embeddings are trained on a sizeable dataset that includes the code of tensorflow, gpu benchmarks, linux kernel and synthetic code.  The trained embeddings are used for three different predictive tasks: classification of algorithm type, placing code on a cpu or a gpu, and optimizations for gpu scheduling. In general, the embeddings provide competitive results when compared to either hand-tuned versions or state of the art baselines.  I think this paper is a nice bridge from traditional program analysis and machine learning. For a researcher with experience in program analysis the contextual graphs are somewhat intuitive. The authors choose to represent variables as nodes instead of whole sentences. It would be nice to provide some intuition why that performs better or is a better choice than having nodes as statements (while keeping the edges to represent both data and control flow).  I would have liked to see some simple experiments with code similarity. One idea for such an experiment: take two implementations for the same algorithm and try to show that in terms of code embeddings they appear similar (it's common to find the implementation of the same neural network in two different frameworks, such as tensorflow and pytorch).  The authors could include a discussion on the limitation of the analysis. In particular, a discussion about how library code is treated. For the suggestion above with two different frameworks, would the embeddings work? Or would the information be lost due to library calls and the reliance of frameworks? For which type of tasks are code embeddings expected to work and for which ones they don't provide sufficient information?  I'm not sure "semantics" is the right term to use for characterizing similarity wrt to consuming/producing similar resources. In general, "semantics" makes me think of a deeper meaning (potentially expressed at a higher granularity than one IR statement).


Exploiting Numerical Sparsity for Efficient Learning : Faster Eigenvector Computation and Regression
In this paper, we obtain improved running times for regression and top eigenvector computation for numerically sparse matrices. Given a data matrix A ∈ R n×d where every row a ∈ R d has a 2 2 ≤ L and numerical sparsity at most s, i.e. a 2 1 / a 2 2 ≤ s, we provide faster algorithms for these problems in many parameter settings.
For top eigenvector computation, we obtain a running time ofÕ(nd + r(s + √ rs)/gap 2 ) where gap > 0 is the relative gap between the top two eigenvectors of A A and r is the stable rank of A. This running time improves upon the previous best unaccelerated running time of O(nd + rd/gap 2 ) as r ≤ d and s ≤ d.
For regression, we obtain a running time ofÕ(nd + (nL/µ) snL/µ) where µ > 0 is the smallest eigenvalue of A A. This running time improves upon the previous best unaccelerated running time ofÕ(nd + nLd/µ). This result expands the regimes where regression can be solved in nearly linear time from when L/µ =Õ(1) to when L/µ =Õ(d 2/3 /(sn) 1/3 ).
Furthermore, we obtain similar improvements even when row norms and numerical sparsities are non-uniform and we show how to achieve even faster running times by accelerating using approximate proximal point [9] / catalyst [15]. Our running times depend only on the size of the input and natural numerical measures of the matrix, i.e. eigenvalues and p norms, making progress on a key open problem regarding optimal running times for efficient large-scale learning.

** Summary ** The paper studied two important problems in machine learning and numerical linear algebra: top eigenvector computation and l_2 regression. The authors introduce ``numerical sparsity’’ as a new parameter to measure the efficiency of algorithms for solving these problems. For a vector x, its numerical sparsity is defined to be |x|_1^2 / |x|_2^2. This quantity is a ``softer’’ version of sparsity and is never larger than the l_0 sparsity of x. The main new insight of this paper is that, it is possible to achieve better running time if we measure the efficiency of algorithms using this new parameter. To accomplish this goal, instead of devising completely new convex optimization methods, the authors develop new stochastic gradient oracle for these problems by importance sampling the matrix A (Section 3), whose variance bound depends on the numerical sparsity of rows of A, and then directly apply existing stochastic first order methods to solve l_2 regression and top eigenvector computation (Section 4).   The main new technical result in this paper is an unbiased stochastic oracle to estimate A^T Ax for a matrix A and a vector x, which can serve as a stochastic gradient oracle when solving l_2 regression and top eigenvector computation. The variance of the oracle depends on the numerical sparsity of rows of A. I found this idea interesting and natural: consider a sparse vector a, if we slightly perturb a then its l_0 sparsity would be large. However, it is still possible to estimate such vector a accurately by (i) finding out heavy coordinates of a and (ii) importance sampling coordinates according to their magnitudes. The authors develop new stochastic oracle to estimate A^T Ax by formalizing these intuitions in Section 3. The main difficulty here is to relate the variance to the function error, and the authors achieve this goal by exploiting the numerical sparsity of rows of A. I found this sampling procedure interesting and may have applications in other machine learning / numerical linear algebra tasks.   Overall, this is a good paper and I would recommend acceptance.   ** Minor Comments ** Line 39: be achieved -> to be achieved Line 72: The definition of \lambda_min should be \lambda_d(A^T A) Line 79: (m/2) (a_i^T x - b_i) -> (m/2) (a_i^T x - b_i)^2 Line 212: In Algorithm 4, remove M in the denominator from the definition of M Line 378: how can we -> how we can Line 379: in smooth -> is smooth Line 380: remove the space after ``problems’’ Line 420: RHS of the equation should be (a^T x)^2 Line 429: The second inequality seems redundant Line 518: It seems you are assuming \kappa \ge d instead of \kappa \le d^2 in Lemma 11, since you want \sqrt{\kappa} \ge \sqrt{s_i}. This will also affect some related parts of the paper.


Confounding-Robust Policy Improvement
We study the problem of learning personalized decision policies from observational data while accounting for possible unobserved confounding in the data-generating process. Unlike previous approaches that assume unconfoundedness, i.e., no unobserved confounders affected both treatment assignment and outcomes, we calibrate policy learning for realistic violations of this unverifiable assumption with uncertainty sets motivated by sensitivity analysis in causal inference. Our framework for confounding-robust policy improvement optimizes the minimax regret of a candidate policy against a baseline or reference "status quo" policy, over an uncertainty set around nominal propensity weights. We prove that if the uncertainty set is well-specified, robust policy learning can do no worse than the baseline, and only improve if the data supports it. We characterize the adversarial subproblem and use efficient algorithmic solutions to optimize over parametrized spaces of decision policies such as logistic treatment assignment. We assess our methods on synthetic data and a large clinical trial, demonstrating that confounded selection can hinder policy learning and lead to unwarranted harm, while our robust approach guarantees safety and focuses on well-evidenced improvement.

This paper studies the conditions under which IPW-based policy may fail due to the existence of latent confounders, and then provides methods for obtaining a robust policy through optimization of the worst-case regret (which is motivated by the sensitivity analysis literature). Given a user-set \Gamma, the bound of the odds ratio (as in the sensitivity-analysis), and the assumption of e(x,u) (propensity) as a logistic model, the authors derive bounds over the true inverse propensity. This leads to the bound on an empirical regret.   Overall, I enjoyed the paper; the writing is clear and well organized. It successfully conveys the idea of the authors. However, as the paper currently stands, there are still important issues that are somewhat unclear and/or not present, which I list below.   Q1. Setting parameters  Given `properly specified` bounds — \Gamma and \Lambda, one can see improvements on the policy. Then, it is natural to ask how the user should think about and decide the values of \Gamma and-or \Lambda. In sensitivity analysis, researchers can try different values of \Gamma to ‘investigate’ how the results of a study would vary. However, the purpose of this paper is to ‘replace’ the given policy w/ the resulting robust-policy w.r.t. \Gamma, so it's not entirely clear to me how to perform such a thought experiment. Please, elaborate on this issue.   Q2. Data-driven Confounding Robust Policy Improvement  The sensitivity parameter is set by the user, leading to a bound on the odds ratio, which eventually will constrain the policy value, or the causal effect of the treatment. There is, however, an entirely data-driven approach for constraining the experimental distribution, which is the final target of the analysis.  As I understand the paper, it tries to bound the PS since the same is not identifiable when unobserved confounding is present. Still, the real goal of learning the PS, in the first place, is to estimate the Y_T distribution.    If one can bound Y_T directly from P(X, T, Y), it seems that asking for the user input may be unnecessary (since one could immediately get a "data-driven" gamma). The idea of bounding E[Y_T] directly from data (under confounding) seems to be the route taken by (ZB, 2017) (see also Ch. 8, Causality, 2000). Given that both approaches appear concerned with optimizing online learning, how are they related?    We further note that it's possible that the user parameters may lead to bounds that are less informative (larger) than the ones coming directly from the observational (logged) data. Please, can you elaborate on this point?   Q3. Empirical Evaluation  a. If the data is obtained without confounding, what would be the performance of the robust policy returned by the algorithm? Even though the paper demonstrated how robust policy could be worsened with unnecessarily large \Gamma, still there is no natural way, as far as I can see, to tell what \Gamma should be.  b. How about the robustness against the violation of the logistic model assumption of e(x,u)? In such a case, is there any natural interpretation of \Gamma?  c. Why is the IPW worse than RIC family when \Gamma=1? (Figure 2 — referenced as Fig 1a) Is it because the used IPW is a vanilla implementation?  It would be crucial to see results of weight-clipped IPW or weight-normalized IPW.  d. If the data is generated from a different distribution, is it possible for IPW to perform better than RIC* w/ \Gamma=1?  e. Similarly, the performance gain due to the use of \Gamma should be compared to \Gamma=1, not to a vanilla IPW. Please, justify and clarify the meaning of RIC performance at \Gamma=1. Isn’t it assuming that there is no confounder?  The above questions are similarly applied to the experiments with IST data. RIC*s appear to be improved as \Gamma increases (Fig 2a, 2b). However, considering \Gamma=1, we only see a degradation of the performance with smaller \Gammas. It is somewhat hard to assess where the performance gain is coming from.  Summary:  - Clarity: The paper is overall well-written with a right balance between text and math. - Originality: It is original in that it provides a new efficient method to yield a robust policy by optimizing worst-case regret, under the assumptions that the user can provide bounds. This judgment is contingent on how this approach relates (generalizes? subsumes?) the data-driven approach discussed above.  - Significance: Without knowing how to set \Gamma, the applicability of the method may be limited, and the improvement guarantee may be somewhat illusive. The baseline should be replaced to accurately assess how the proposed methods work w/ different \Gammas.  - Quality: Overall okay, with good mathematical details but without strong empirical evidence and not very accurate literature review and comparison.  Minor comments:  for for (Line 173)  an uncertainty (Line 9)  There are two figure 2s without figure 1.  Figure 2 (a), (b), (c) not marked.  Reference:  Zhang, J. and Bareinboim, E. Transfer Learning in Multi-Armed Bandits: A Causal Approach. In Proceedings of the 26th International Joint Conference on Artificial Intelligence (IJCAI), 1340-1346, 2017.    Pearl, J. Causality: Models, Reasoning and Inference. Cambridge University Press, 2000.  --- Post-rebuttal   We believe the authors addressed most of the issues and updated my grades accordingly. We would like to highlight that we expect the authors doing a better job in the final version of the paper regarding the two central points:   1. Gamma is the critical parameter that the whole analysis hinges on, so it should have more emphasis an explanation on how, in any cognitively meaningful way, users can conceive setting these parameters to some value.   2. I don’t think the answer regarding Q2 is accurate. The paper should acknowledge that there’s a tradeoff between knowledge an inferential power, and they are not proposing anything magical here. If the user wants to be entirely data-driven, they should go with the work of ZB'17 (or some obvious/natural generalization), while if they do have the substantive knowledge needed to set \Gamma (whatever this could be), they can consider the approach proposed in the paper.   These points should be front and center in the paper since it currently conveys the image that nothing was done before and that \Gamma is a somewhat "magical" parameter. Further, it seems the results are obtained without any effort and there isn't a more data-driven alternatively to their method.  Furthermore, the work on Manski and other folks are great for bounding in the offline setting, but the desirable property about the setting discussed is coupling the knowledge from one policy on another and proving that it can indeed help in practice.   Still, I think the work proposed here is nice and should, therefore, be accepted.


Contextual bandits with surrogate losses: Margin bounds and efficient algorithms
We use surrogate losses to obtain several new regret bounds and new algorithms for contextual bandit learning. Using the ramp loss, we derive new margin-based regret bounds in terms of standard sequential complexity measures of a benchmark class of real-valued regression functions. Using the hinge loss, we derive an efficient algorithm with a √ dT -type mistake bound against benchmark policies induced by d-dimensional regressors. Under realizability assumptions, our results also yield classical regret bounds.

[Summary]  This paper studies a contextual bandit problem in which regret is defined against a given set of randomized policies that map from a context space to an arm space. As sets of randomized policies, the authors consider classes of composite functions of a K-dimensional 0-component-sum real vector-valued function over the context space  and a component-wise ramp or hinge loss function. For ramp and hinge losses, they propose algorithms and prove their regret upper bounds.  [Originality]  Problem setting, proposed algorithms and obtained their regret upper bounds look new.  [Strengths]  * Nice theoretical results * technically sound  [Weakness]  * No empirical demonstration * Bad readability  [Recommendation]  I recommend this paper to be evaluated as "Marginally above the acceptance threshold". The theoretical results look nice, but it is very difficult to understand the role of surrogate losses. Benchmark sets of policies themselves are defined using surrogate loss functions, then are those losses what loss's surrogates are? In Lemma 1, make clear what loss is translated to what loss for what purpose.  [Detailed Comments]  p.2: Specify \gamma's range. p.4 Th 3 R^3: Don't use a footnote mark as a superscript of a mathematical formula. p.4 l.138: Don't refer a theorem whose statement is in Appendix.  [Comments to Authors' Feedback]  In my understanding, a surrogate loss function means a surrogate of some loss function such as a 0-1 loss function. However, the authors' surrogate losses do not replace some other losses. In that meaning, it might be better to be called not a surrogate loss but a somewhat other name.


Learning Pipelines with Limited Data and Domain Knowledge: A Study in Parsing Physics Problems
As machine learning becomes more widely used in practice, we need new methods to build complex intelligent systems that integrate learning with existing software, and with domain knowledge encoded as rules. As a case study, we present such a system that learns to parse Newtonian physics problems in textbooks. This system, Nuts&Bolts, learns a pipeline process that incorporates existing code, pre-learned machine learning models, and human engineered rules. It jointly trains the entire pipeline to prevent propagation of errors, using a combination of labelled and unlabelled data. Our approach achieves a good performance on the parsing task, outperforming the simple pipeline and its variants. Finally, we also show how Nuts&Bolts can be used to achieve improvements on a relation extraction task and on the end task of answering Newtonian physics problems.

Summary: The paper addresses an important aspect of learning pipelines with limited data while incorporating domain specific knowledge relevant to a task.  The main idea is the use of PSL (probabilistic soft logic) as a framework to map partial estimates from multiple feedforward algorithms, along with domain specific logical rules, to parse visual diagrams from physics texts.  Specifically, the pipelines use feature extractors for lines, arcs, corners , text elements, object elements (e.g.blocks in physics diagrams). These are combined along with human specified rules for groupings, high-level elements, text/figure labeling schemes along with the inference engine to produce the parse into a formal logical language.  Experiments illustrate how the learned system:  1)  is superior to state of the art diagram parsing scheme, 2) can utilize labelled as well as unlabelled data to achieve improved performance, 3) can handle various degrees of supervision in different parts of the pipeline and is robust , and 4) through integrative modeling of the stages in pipeline prevents error propagation.  Quality, Clarity, originality, significance of the paper:  The paper is well written and has extensive references to relevant literature, adequate experimentation.  The paper is a systems application paper and adequate information is presented in supplemental material to provide details of the limitation of the results at a systems level. The current work is a systems paper combining well known aspects/algorithms in the literature.   Section 5 (C1) claims that there is no prior work on end to end parsing of diagrams. While I believe this may be true for physics diagrams, the paper misses citation to  key papers that exploit parallel pipelines for feature extraction, bi-lattice logic formalism for reasoning and parsing, and translation to a formal representation.   See:  Vinay D. Shet et al: Predicate Logic Based Image Grammars for Complex Pattern Recognition. International Journal of Computer Vision 93(2): 141-161 (2011)  Toufiq Parag, Claus Bahlmann, Vinay D. Shet, Maneesh Singh: A grammar for hierarchical object descriptions in logic programs. CVPR Workshops 2012: 33-38  P. Sajda et al,  In a Blink of an Eye and a Switch of a Transistor: Cortically Coupled Computer Vision, Vol. 98, No. 3, March 2010 | Proceedings of the IEEE  The current paper also does not refer to a class of past work involving performance modeling of vision pipelines ( including some in the document understanding area) that involve building appropriate trust models for pipelines.   Proper understanding of errors in processing stages can allow appropriate fusion mechanisms to get robust outputs (see: http://haralick.org/journals/performance.shtml).  While it is clear that integrative learning at a holistic systems level reduces impact of error propagation, error modeling and appropriate bayesian fusion has been done in Bayesian model based vision literature (see: Binford, T.O., Levitt, T.S.,  Evidential reasoning for object recognition, PAMI(25), No. 7, July 2003, pp. 837-851. ).  Given that the application domain involving physics diagrams still involve human produced diagrams, I would imagine that a systematic model-based approach can produce robust interpretations.    Other comments: It will be quite difficult to reproduce the results in the paper given the complexity of the system. It is recommended that the datasets, groundtruth, and executables of the system be made available for research purposes.  Comments Based on Author's rebuttal: The authors have put in substantial effort to address my concerns in my first review.  Unfortunately this paper is a complex system and a lot of implementation detail for the alternatives is omitted in its present form.   It is impossible to judge given the one page rebuttal why the alternative frameworks (e.g. bi-lattice reasoning)  perform worse.  I would want to know more details: e.g.  Did they implement Shet et al paper exactly as specified?  what types of rules did they use?  How did they learn the rule weights?  etc.   The authors state that there is no prior work that uses software modules, parallel estimators, and fusion. I refer the authors to a few patents and publications in systems engineering for vision in the early 2000's address this:  Methodology for model based systems engineering / performance modeling: US Patents: 7,899,209 -Statistical modeling and performance characterization of a real-time dual camera surveillance system  (also appeared in: Greiffenhagen et al, CVPR 2000, CVPR 2001, Proc. of IEEE 2001)  7,428,337 - Automatic design of morphological algorithms for machine vision , by Gao et al, 2002  -  addresses learning of nonlinear operator chains for a given classification task  While these references are not central to the present paper, they may be useful for the authors.   Overall recommendation:  I don't have a problem with acceptance of the paper based on what the authors have demonstrated.  While the authors definitely made attempts to address my review,  comparison of alternative systems approaches are often challenging and an open question is to ask how one can make rigorous claims of superiority of one approach over another.   As a systems scientist, I would not be upset with rejection of the paper.  As an applications engineer, I would be ok with the acceptance of the paper.  Thus, my rating for the paper does not change.


Learning Sparse Neural Networks via Sensitivity-Driven Regularization
The ever-increasing number of parameters in deep neural networks poses challenges for memory-limited applications. Regularize-and-prune methods aim at meeting these challenges by sparsifying the network weights. In this context we quantify the output sensitivity to the parameters (i.e. their relevance to the network output) and introduce a regularization term that gradually lowers the absolute value of parameters with low sensitivity. Thus, a very large fraction of the parameters approach zero and are eventually set to zero by simple thresholding. Our method surpasses most of the recent techniques both in terms of sparsity and error rates. In some cases, the method reaches twice the sparsity obtained by other techniques at equal error rates.

The paper addresses the actual problem of structural sparsity of deep neural networks (NNs). The authors propose a general training procedure aiming for sparsity that measures relevance of a model parameter (weight) as analytical output sensitivity with respect to the particular weight. - The approach comprises a novel regularization technique that gradually lowers the absolute values of weights with low sensitivities, while weights with high sensitivities are not modified. After each training epoch, weights lower then a given threshold are zeroed out. - The authors also present a theoretical explanation and analysis of the method and its brief experimental evaluation on MNIST and ImageNet data sets.  Clarity: - The presentation is comprehensible and well-organized, the new method and experiments are described adequately. - The authors also summarize some the major recent related works. However, they omit existing works concerning regularization and pruning of neural networks using sensitivity analysis (e.g., [1,2]).  Typos: line 78 "rule.Then" (missing gap) line 224 "over [9] (... 68x compression ratio) over the nearest" ... 68x corresponds to the 5th row of Table 1, labeled by SparseVD[14] (not Han et al. [9])  Quality: - The paper has a fairly good technical quality. The explanation, derivation and theoretical analysis of the new method are detailed and comprehensible. However, I missed a greater experimental evaluation, especially concerning more complex NN-architectures.  It is unusual, that the authors first propose the adaptation rule (Eq. 9) and after that they derive the corresponding error function (eq. 15). The general shape of the penalization function (eq. 15) is therefore quite complex and, moreover, different activation functions thus lead to different error terms. Usually, the process is opposite: to design an error function first and then the corresponding adaptation rules.  Novelty and significance: The paper presents a new stochastic regularize-and-prune technique that is an interesting extension and a concurrent of weight decay. Contrary to weight decay, it doesn't reduce all of model weights but only weights with great "insensitivity", which may prevent the model from undesirable reduction of important weights.  (+) The new method overcomes the reference pruning techniques on MNIST data by reaching twice the sparsity for similar error rates, while the proposed regularization technique itself proved to improve generalization ability of the NN-model and prevent overfitting (it seems that it slightly overcomes weight-decay in this respect). (-) However, the experimental results for more complex NN-architecture (VGG-16) show only relatively small benefit over the baseline technique. Therefore, more experimental results for complex tasks would be beneficial.  Questions:  - Based on the presented experimental results, it seems that the method prunes more extensively layers closer to the input layer. Have you observed and investigated this possible property? - A further important question is, what is the computational complexity of the algorithm (e.g., when compared to weight-decay).  [1] Engelbrecht, A. and Cloete, I. A Sensitivity Analysis Algorithm for Pruning Feedforward Neural Networks. In: IEEE International Conference in Neural Networks, Washington, DC, USA. 1996. Vol. 2 of IEEE ICNN’96, pp. 1274–1277. [2] Mrazova, I., Kukacka, M. Can Deep Neural Networks Discover Meaningful Pattern Features?, Procedia Computer Science, Vol. 12, 2012, pp. 194-199.  _________________________________________________________________ The rebuttal addressed most of my points (recent works, complexity) except the point that the adaptation rules seem to be the same for all activation functions.  I think, the method is worth of publication, although the authors should consider more previous works and alter the paper by deeper experimental evaluation.


Adversarial Risk and Robustness: General Definitions and Implications for the Uniform Distribution
We study adversarial perturbations when the instances are uniformly distributed over {0, 1}
n . We study both "inherent" bounds that apply to any problem and any classifier for such a problem as well as bounds that apply to specific problems and specific hypothesis classes. As the current literature contains multiple definitions of adversarial risk and robustness, we start by giving a taxonomy for these definitions based on their direct goals; we identify one of them as the one guaranteeing misclassification by pushing the instances to the error region. We then study some classic algorithms for learning monotone conjunctions and compare their adversarial robustness under different definitions by attacking the hypotheses using instances drawn from the uniform distribution. We observe that sometimes these definitions lead to significantly different bounds. Thus, this study advocates for the use of the error-region definition, even though other definitions, in other contexts with context-dependent assumptions, may coincide with the error-region definition. Using the error-region definition of adversarial perturbations, we then study inherent bounds on risk and robustness of any classifier for any classification problem whose instances are uniformly distributed over {0, 1}
n . Using the isoperimetric inequality for the Boolean hypercube, we show that for initial error 0.01, there always exists an adversarial perturbation that changes O( √ n) bits of the instances to increase the risk to 0.5, making classifier's decisions meaningless. Furthermore, by also using the central limit theorem we show that when n → ∞, at most c · √ n bits of perturbations, for a universal constant c < 1.17, suffice for increasing the risk to 0.5, and the same c √ n bits of perturbations on average suffice to increase the risk to 1, hence bounding the robustness by c · √ n.

The paper studies the adversarial examples phenomenon from the theoretical perspective with a focus on discrete distributions. The authors make two main contributions:  1. They introduce a new robustness measure called "error region robustness" and compare it to other definitions of robustness (the proposed measure is not specific to discrete distributions). The main difference to prior work is that the proposed definition does not count an example as robustly misclassified if the small perturbation actually changes the correct class.  2. They study learning problems under the uniform distribution on the hypercube and show that *any* classifier with a given standard risk will have a certain amount of adversarial / robust risk. The argument is based on isoperimetry results for the hypercube.  I think both points are relevant and the theoretical arguments in the main paper seem correct. Compared to prior work, the novelty is somewhat limited however:  - While their error region robustness is indeed interesting, it only differs from prior definitions of robustness when small changes to an example can change the correct class. In adversarial examples, the focus is typically on small perturbations that do not change the correct class. In this regime, the proposed definition and the definition in prior work coincide.  - At a high level, the isoperimetry argument relating standard accuracy to robust accuracy is similar to the work of Gilmer et al. (cited by the authors) for a continuous distribution. The authors mention twice that they give the first results on adversarial robustness for a discrete distribution. It seems however that the work of Schmidt et al. (also cited) already establishes robustness results for a distribution on the hypercube. It would be helpful if the authors could comment on this in the rebuttal. But it is worth noting that as far as I know, the hypercube isoperimetry argument for the uniform distribution has not appeared before in the context of adversarial examples.  Another area of improvement is that the authors could discuss in more detail what their results mean for the image classification setting that is common in the adversarial examples literature. For instance, it is not clear how restrictive the assumption of a uniform distribution on the hypercube is (in terms of images, this would correspond to fully random images). As far as I can tell, the theoretical results crucially rely on working with a uniform distribution. While simple models are certainly a good starting point for theoretical studies, it would be good to discuss the connections to empirical phenomena.  Finally, it seems that the error region notion of robustness might have counter-intuitive behavior in regimes where the Bayes error is strictly greater than 0. In that case, the error-region risk of the Bayes-optimal classifier will be 0 while the "standard" risk is non-zero (i.e., the robust risk is less than the "standard" risk). The prior notions of robust risk do not have this shortcoming. It would be helpful if the authors could comment on this in their rebuttal and potentially include a discussion about this in their paper.  Overall, my current assessment is a weak accept since the two main points are important and there is currently a lack of theoretical results regarding adversarial robustness.     Minor comments:  - There are some formatting issues. The abstract is too long and the paper was submitted using the "preprint" option of the NIPS LaTeX package.  - On Page 3, it would be helpful to remind the reader what a monotone conjunction is.  - The second paragraph from the bottom of Page 3 seems out of context as it mentions several terms that are not defined yet (OT, PC, h, c, etc.).  - The abstract contains two sentences that seem ungrammatical:      * "Our experimental calculations point for range of [...]"      * "[...] previously proposed definitions have risk and robustness e.g., by pushing [...]"  - Near the bottom of Page 2, should the bound 2.15 sqrt(n) depend on mu from the previous sentence? Or is it for a specific mu?  - Page 4: "We did not state the loss function explicitly [...]" is missing a "." at the end.  - In Definition 3.6, would it be easier to use |A| (cardinality of A) instead of V(A) since we are working with a uniform distribution over the hypercube? The paper uses |A| in other places such as Lemmas 3.7 and 3.8.  - What is p_i in the proof of Lemma 3.9? In particular, it is not clear what the index i refers to.  - In Equation (2) on Page 8, should it be (2^w + 2^u - 1)?  - The sentence before Theorem 3.10 refers to a discussion in Section D of the appendix, but the section seems to contain only proofs. Is this intentional?  - Appendix B is empty.  - Appendix D.7 mentions "10,00" random examples. The use of "," here is non-standard.


Adversarial Scene Editing: Automatic Object Removal from Weak Supervision
While great progress has been made recently in automatic image manipulation, it has been limited to object centric images like faces or structured scene datasets. In this work, we take a step towards general scene-level image editing by developing an automatic interaction-free object removal model. Our model learns to find and remove objects from general scene images using image-level labels and unpaired data in a generative adversarial network (GAN) framework. We achieve this with two key contributions: a two-stage editor architecture consisting of a mask generator and image in-painter that co-operate to remove objects, and a novel GAN based prior for the mask generator that allows us to flexibly incorporate knowledge about object shapes. We experimentally show on two datasets that our method effectively removes a wide variety of objects using weak supervision only.

The details provided on the experiment with SDI[2] are somewhat lacking, and the question on how that works combined with standard inpainting was not really addressed. However, they raise a valid point in regards to it requiring box-level supervision compared to the image-level supervision they use. Overall I'm still leaning to accept this submission.  ====================================== The paper proposes a system that can learn to remove objects of specific categories from images automatically (e.g. persons). The system is composed of two stages: 1)  a mask generator which zero-es out regions in the image such that an image classifier does not recognize the specified category (e.g. it is not classified as person) 2) an inpainting network which infills the masked image so that it looks realistic again  Notably, the mask network uses weak supervision: it only can see real masks from a smaller dataset through a GAN loss. More specifically, it is trained to produce masks that hide the target object (fooling an image level classifier) and to also produce masks that are realistic from the view of a mask discriminator.   Overall the paper is well written and provides extensive experiments, building a strong baseline for the task from Mask-RCNN + inpainting network (which uses stronger supervision since mask-rcnn is trained on a large corpus of segmentation masks). Qualitatively the network works fairly well, and quantitatively it is competitive with the mask-rcnn baseline. The benefit of the weak supervision is demonstrated by showing that it can work on a dataset that has no segmentation masks available (a logo dataset).  The paper makes the point that it is actually helpful to have coarse masks that hide the (to be removed) object's boundaries, compared to fine grained masks. This is demonstrated both when comparing the proposed approach (which has coarse masks because of weak supervision) to the mask-rcnn baseline, and also by improving the mask-rcnn baseline by dilating its segmentation masks.  However, based on this I question the need for the the GAN based mask generator. One could interpret the results as that for object removal, you don't really need a very good instance segmentation network (since dilating mask-rcnn helps). Therefore, we should also expect to see good results when using a weakly supervised instance segmentation network. In principle, we can view the first stage of the system as exactly that, but shouldn't the baseline for weak supervision then be as follows:  * Standard weakly supervised instance segmentation network, such as (Simple Does It: Weakly Supervised Instance and Semantic Segmentation, Khoreva et al, CVPR 2017 ). * Possibly dilate its masks * Use standard inpainting approach on top  So basically, I question the need for the first stage: is it not just a weakly supervised instance segmentation approach? If so, is competitive with established such approaches, and should they not be the baseline?  Another concern is what will the whole system be used for? I understand that removing people from images in an unsupervised manner is a difficult problem, but it would be nice to see a concrete example where it is useful.  Nontheless, I found the paper overall interesting in terms of the problem studied, the approach and the quality of the experiments, so I'm currently leaning towards accepting it.


Bipartite Stochastic Block Models with Tiny Clusters
We study the problem of finding clusters in random bipartite graphs. We present a simple two-step algorithm which provably finds even tiny clusters of size O(n ε ), where n is the number of vertices in the graph and ε > 0. Previous algorithms were only able to identify clusters of size Ω( √ n). We evaluate the algorithm on synthetic and on real-world data; the experiments show that the algorithm can find extremely small clusters even in presence of high destructive noise.

This paper introduces and analyzes a simple algorithm for bipartite block model clustering in which clusters can be very small.  The model in which the work is situated is a natural variation of a blockmodel for bipartite random graphs which allows the clusters on the "right" to overlap. The model involves a bijection between the clusters on the left and the clusters on the right. Two probabilities must be specified: the probability that an edge exists between two nodes whose blocks are in bijection, and the probability that an edge is generated between two nodes whose blocks are not in bijection (though it is likely that this constraint can be relaxed).  The algorithm is simple: first, cluster the nodes on the left by applying a mixture model approach to the rows of the biadjacency matrix. Next, having determined the clusters on the left, assign to each the nodes on the right which have sufficiently many edges to it. To analyze the first part of the algorithm, the authors make use of the results of Mitra concerning mixture models.  The authors prove that -- quite interestingly -- the algorithm is able to recover right clusters which are O(n^eps) small, which represents a significant improvement over previous analyses. This is perhaps surprising due to previous analyses which proved a sqrt(n) lower bound on cluster size, however (as the authors note) those analyses assumed that the bipartite clusters were square. In contrast, the current paper allows the left cluster and right cluster to be of vastly different sizes. This relaxation is quite natural and provides for a smooth tradeoff -- very small right clusters can be recovered, provided that the left cluster is correspondingly large.  Ultimately, I found the paper clear and interesting. The fact that such small clusters can be recovered is intriguing, though it would be interesting to investigate whether the smooth tradeoff mentioned is indeed optimal.


Structural Causal Bandits: Where to Intervene?
We study the problem of identifying the best action in a sequential decisionmaking setting when the reward distributions of the arms exhibit a non-trivial dependence structure, which is governed by the underlying causal model of the domain where the agent is deployed. In this setting, playing an arm corresponds to intervening on a set of variables and setting them to specific values. In this paper, we show that whenever the underlying causal model is not taken into account during the decision-making process, the standard strategies of simultaneously intervening on all variables or on all the subsets of the variables may, in general, lead to suboptimal policies, regardless of the number of interventions performed by the agent in the environment. We formally acknowledge this phenomenon and investigate structural properties implied by the underlying causal model, which lead to a complete characterization of the relationships between the arms' distributions. We leverage this characterization to build a new algorithm that takes as input a causal structure and finds a minimal, sound, and complete set of qualified arms that an agent should play to maximize its expected reward. We empirically demonstrate that the new strategy learns an optimal policy and leads to orders of magnitude faster convergence rates when compared with its causal-insensitive counterparts.

The authors propose an algorithm that could improve the efficiency of multi-bandits algorithms by selecting a minimal, sound and complete set of possibly optimal arms. In their frameworks arms correspond to interventions on a causal graph, which could possibly have latent confounders. If the structure of the causal graph is known (but not the parameters), the proposed algorithms can exploit this structure to decide which interventions are provably nonoptimal and filter them out.  I think the paper presents some interesting and novel ideas. The clarity could be improved, especially for a reader who is not familiar with the do-calculus literature (e.g. section 2). The experiments are limited, but in my opinion the theory makes the paper interesting in itself.  Minor details: Abstract: I’m a bit confused about how could one empirically demonstrate that an algorithm leads to optimal (...) convergence rates.  Some inconsistency in the notation, e.g.: line 88: PA_i is never defined, although it is clearly pa(X_i),  Lines 117-123: maybe it’s pedantic, but shouldn’t there be some universal quantification on x,y,z, w? For example P(y|do(x), do(z),w) = P(y|do(x),w) for all x \in D(X), z \in D(Z) etc? Line 129: isn’t X = X intersected with an(Y)_{G after do(X)} just X \subset an(Y)_{G after do(X)} ? line 198: pa(T)_G\T … doesn’t pa(.) already exclude itself (in this case T)?  Typos: footnote 4: “that are not latent confounders are not explicitly represented”, line 252 “four strategies”, line 214, 216 “intervening ON”  Appendix B: Instead of Algorithm 4, wouldn’t it be enough to have a much simpler “while (T != oldT) { oldT=T; T=desc(cc(T)); }”?


Memory Augmented Policy Optimization for Program Synthesis and Semantic Parsing
We present Memory Augmented Policy Optimization (MAPO), a simple and novel way to leverage a memory buffer of promising trajectories to reduce the variance of policy gradient estimates. MAPO is applicable to deterministic environments with discrete actions, such as structured prediction and combinatorial optimization. Our key idea is to express the expected return objective as a weighted sum of two terms: an expectation over the high-reward trajectories inside a memory buffer, and a separate expectation over trajectories outside of the buffer. To design an efficient algorithm based on this idea, we propose: (1) memory weight clipping to accelerate and stabilize training; (2) systematic exploration to discover high-reward trajectories; (3) distributed sampling from inside and outside of the memory buffer to speed up training. MAPO improves the sample efficiency and robustness of policy gradient, especially on tasks with sparse rewards. We evaluate MAPO on weakly supervised program synthesis from natural language (semantic parsing). On the WIKITABLEQUESTIONS benchmark, we improve the state-of-the-art by 2.6%, achieving an accuracy of 46.3%. On the WIKISQL benchmark, MAPO achieves an accuracy of 74.9% with only weak supervision, outperforming several strong baselines with full supervision. Our source code is available at goo.gl/TXBp4e.

This paper describes a Reinforcement Learning algorithm adapted to settings with sparse reward and weak supervision, and applies it to program synthesis, achieving state-of-the-art and even outperforming baselines with full supervision.  The two first sections explain very clearly the motivation of this work, presenting the current limitations of reinforcement learning for tasks like contextual program synthesis. It is nicely written and pleasant to read. Section 3 presents the Reinforcement Learning framework that is the basis of the proposal, where the goal is to find a food approximation of the expected return objective. Section 4 presents the MAPO algorithm and his three key points: "(1) distributed sampling from inside and outside memory with an actor-learner architecture; (2) a marginal likelihood constraint over the memory to accelerate training; (3) systematic exploration to discover new high reward trajectories" (I did not find a better phrasing to summarize than the one in the abstract and the conclusion).  The experiments are carried out on two public datasets, includes details for reproducibility, and show compelling results.  The references are relevant throughout the paper. I cannot attest the completeness, not being an expert in RL. I don't see a reason why this paper should not be accepted!  Some comments:    * Probably a mistake on end of line 152: "smaller stochastic space of size xxx ..." -> "... of measure xxx ...".    * Section 4.1 could be better explained. Why is the enumeration prohibitive? Because of the cost of evaluating new gradients and log probabilities for those trajectories?    * End of section 4.2: "Once the policy is off the ground, almost never the buffer probabilities are truncated given that they are naturally larger than alpha". It would be interesting to see plots of this. no experimental results are given about any properties of these trajectories, but it seems like they would be important to look at to understand the technique. For example, what fraction of the gradient magnitude comes from the buffer and what fraction from the MC samples? Illustrating this would be the cherry on the cake.


Understanding the Role of Adaptivity in Machine Teaching: The Case of Version Space Learners
In real-world applications of education, an effective teacher adaptively chooses the next example to teach based on the learner's current state. However, most existing work in algorithmic machine teaching focuses on the batch setting, where adaptivity plays no role. In this paper, we study the case of teaching consistent, version space learners in an interactive setting. At any time step, the teacher provides an example, the learner performs an update, and the teacher observes the learner's new state. We highlight that adaptivity does not speed up the teaching process when considering existing models of version space learners, such as the "worst-case" model (the learner picks the next hypothesis randomly from the version space) and the "preference-based" model (the learner picks hypothesis according to some global preference). Inspired by human teaching, we propose a new model where the learner picks hypotheses according to some local preference defined by the current hypothesis. We show that our model exhibits several desirable properties, e.g., adaptivity plays a key role, and the learner's transitions over hypotheses are smooth/interpretable. We develop adaptive teaching algorithms, and demonstrate our results via simulation and user studies.

This paper considers when knowing a learner's current hypothesis can lead to more effective teaching than if the hypothesis is not known. The benefits of adaptivity are explored in two cases where there is an impact, and algorithms are developed for adaptive teaching in those domains that attain efficient performance. The growth rate of the number of examples for adaptive and non-adaptive teachers is shown, and adaptive algorithms are compared to alternatives both in simulation and with human learners.  This paper addresses an important issue in algorithmic teaching, as adaptivity is often costly and understanding when adaptivity is important could be useful for building adaptive teaching systems. I appreciate that the paper made clear conceptually, as well as mathematically, when adaptivity matters. The paper focuses on local preference updating, often making reference to cases where the hypothesis does change but changes in a way that's dependent on the current hypothesis, but my reading is that any dependence on the current hypothesis has the potential to have adaptivity matter, including simply having the hypothesis stay the same if new data doesn't violate it. I bring up this case because it's consistent with some simple human learning models, such as the "win stay lose shift" model (e.g., Levine, M. (1975). A cognitive theory of learning: Research on hypothesis testing; Bonawitz, E., Denison, S., Gopnik, A., & Griffiths, T. L. (2014). Win-Stay, Lose-Sample: A simple sequential algorithm for approximating Bayesian inference. Cognitive psychology, 74, 35-65; Rafferty, A. N., Brunskill, E., Griffiths, T. L., & Shafto, P. (2016). Faster teaching via POMDP planning. Cognitive science, 40(6), 1290-1332). It would be interesting to explore how much adaptivity is helpful with such a simple model that is less constrained in what happens when the current hypothesis is violated - the preference function \sigma is essentially local only when the current hypothesis is still valid. Making links to this type of work with human learning models could strengthen the impact of the present paper.  The main questions I had center on the human evaluation, where I believe the results are overstated. The "Real-world Human Preferences" uses data from participants to argue that people perform incremental updates when learning in 2-Rec. While I agree this is probably the case, the section doesn't acknowledge the (likely large) impact of the interface on these results. The participant has a hypothesis shown on screen at all times, and I believe participants were asked to "update" this based on the new information.  This interface constrains people to match the version space learner (at least in terms of what we can observe) by having exactly one hypothesis, and that hypothesis must be consistent with all data provided so far; while other representations are possible and have been frequently used for representing human category learning (e.g., maintaining an implicit distribution over possible hypotheses), these can't be expressed. The interface also nudges people towards local updates - e.g., I expect results would be at least somewhat different if people had to draw new rectangles every time, on a different image than the one with the hypothesis displayed (and probably more different still if they couldn't see the previously drawn hypothesis). Thus, the human evaluation seems to show that people are helped by an adaptive algorithm (as compared to random examples or set cover), given an interface that pushes them towards the learner model that is assumed by the adaptive algorithm. I still think this is important and useful to show, but the limitations of this evaluation should be acknowledged.  Additional questions about the human evaluation: - Why wasn't the non-adaptive algorithm (non-R) considered? That seems like it would be the natural comparison for understanding how much adaptivity matters in this case, and I would very much like to see results for the non-R case, more closely mirroring the simulated case. -  I did not understand the different between the target hypothesis being M_{1} versus M_{1/2} in Figure 3a. My understanding from the other cases where the subscript 1/2 is used is that it is for cases where either 1 or 2 rectangles could be consistent with the data. Is the distinction here that M_{1} is a single square (so non-representable using a 2 rectangle hypothesis), but M_{(1/2)} is a larger rectangle, which could be expressed by 1 rectangle or by 2 adjacent rectangles? If so, why this case is separated out for the human learners and not for the simulated evaluation? I may very well be missing something and clarification would be appreciated.  The consideration of noise in the preference function in the supplement was helpful in thinking about applications of this work. In the noisy case, why were only SC and Ada considered, not Non? As above, Ada versus Non seems like the most interesting comparison to see the impact of adaptivity with both adaptive and non adaptive having the (noisily) correct preference function. I look forward to the authors making the point about SC outperforming Non and incorporating the additional baseline, as discussed in their response.  To summarize based on the four review criteria: - Quality: The theoretical work here is sound, and the simulations support these theoretical results. The behavioral experiment is a key part of supporting the relevance of this work for human teaching, and it provides moderate support for the relevance of the theoretical work, but as discussed above, the limitations of the current design and how it drives people's behavior towards the proposed model are not adequately discussed. - Clarity: The paper is well-organized and understandable, with an appropriate level of detail in both the main paper and the supplementary materials. There are several individual points that were not clear (my questions are provided elsewere in this review), but as a whole, clarity is good. - Originality: To the best of my knowledge, framing the adaptivity question in this rigorous theoretical way is novel, and the work appropriately cites related work and argues for why this work adds to the existing literature. - Significance: The paper goes beyond previous work, exploring the importance of adaptivity from three methodologies (theoretical, simulation, and behavioral experiments), and is likely to be of interest both to those interested specifically in the theoretical results for machine teaching and those who are more oriented towards educational applications. Other researchers in machine teaching are likely to build on this work, and the rigorous approach of determining when adaptivity makes teaching/learning more efficient is likely to be relevant to applied work in using machine learning approaches for education.  After reading the author response and the other reviews, my general evaluation of the paper remains the same. The response clarifies several questions, and the proposed revisions will, I believe, increase the clarity of the paper. While computing the optimal teaching policy is hard (and that problem is not solved by this work), I think it is an important contribution to formalize the problem in this way, and will perhaps facilitate future work that focuses on techniques for the optimization problem. I'm unsure from the response if the new experimental study is intended to be included in this work or in future work; if the latter, I again want to strongly urge the authors to note the limitation of how the present interface may encourage participants towards the assumed learner model.  Small issues/typos: - Line 10: "learner picks hypothesis according to..." -> hypotheses - Line 241: "poorly.An" - Line 329: "an web-interface" - Line 348: "There were randomly assigned" - Caption Figure 3: "require less examples" -> "require fewer examples" - Line 397 in the supplement: "100 participants on Mechanical Turk were each shown the same set of then initial configurations." -> ten, not then


Are GANs Created Equal? A Large-Scale Study
Generative adversarial networks (GAN) are a powerful subclass of generative models. Despite a very rich research activity leading to numerous interesting GAN algorithms, it is still very hard to assess which algorithm(s) perform better than others. We conduct a neutral, multi-faceted large-scale empirical study on state-of-the art models and evaluation measures. We find that most models can reach similar scores with enough hyperparameter optimization and random restarts. This suggests that improvements can arise from a higher computational budget and tuning more than fundamental algorithmic changes. To overcome some limitations of the current metrics, we also propose several data sets on which precision and recall can be computed. Our experimental results suggest that future GAN research should be based on more systematic and objective evaluation procedures. Finally, we did not find evidence that any of the tested algorithms consistently outperforms the non-saturating GAN introduced in [9].

In this paper the authors tackle the non-trivial problem of GANs comparison. Comparing the different approaches toward training GANs is a major issue towards a better understanding of deep generative models.   This paper introduces a methodological procedure for comparing GANS via two different metrics FIA, and Precision, recall and F1. The authors compare 7 Different GANs (different Generative/Discriminator loss, and regularization terms)  under the same network architecture. A procedure for testing different hyperparameters, that allows for different computational budgets, is introduced. The randomness due to network initialization is also analyzed. The experiments are run on four popular datasets, and also a synthetic dataset consisting of binary images of triangles is used. This synthetic dataset allows to directly compute the distance from a generated sample to the considered manifold (via non-convex optimization).  There are two main aspects that show the importance/contributions of the  paper.   Introduces a methodology for comparing GANs (choose a bunch of different loss functions, different regularizations, fix architecture, adopt a metric, choose different datasets, repeat experiments since training involve randomness, introduce the computational budget as a way of quantitatively comparing how difficult is to reach good hyperparameters); Reaches interesting conclusions. As a first example, the paper concludes from empirical evidence that it does not make sense to compare the best FID results. Comparison should involve the distribution of the adopted metric under different runs.  As a second example, the authors show empirically that given the right hyperparameters the best GAN (according to FID) depends on the dataset. There is no clear winner.   I believe this paper will help the community to start a discussion on how to fairly compare deep generative models.  *Major Points*  The authors claim that algorithmic differences in state-of-the-art GANs become less relevant, as the computational budget increases. This is true according to the way the authors carried out the experiments. But, what happens if we start the hyperparameters tuning by doing a random search from the recommended values for each algorithm. Maybe the hyperparameters recommended by the authors are already good enough (or they are close to the best ones) for that particular dataset. For instance, according to the authors of WGAN-GP the same set of hyperparameters worked OK in many different experiments. Please comment.  A clear limitation of the current work (in terms of the conclusions that are drawn) is that only one architecture is considered (infoGAN). All the conclusions are with respect to this architecture. So, one wonders whether this analysis is  general enough. The implicance of this should be commented in the text.  The conclusions are too strong. In fact, the paper shows that the average performance (across datasets) is similar but the best one depends on the dataset. This does not mean that all the GANs will reach the same best performance on every dataset.   Inability to detect overfitting by FID. This is commented in Section 5, but then it is not linked to the rest of the paper. In particular, the finding regarding precision and recall on the synthetic dataset are not commented in the conclusions. I recommend to improve the connection of this experiment to the large-scale experimental evaluation.  *Minor Comments / Typos*  l26  - metric → metrics l40 - Fréchet Inception Distance → Fréchet Inception Distance (FID) l77 - mean and covariance is estimated → mean and covariance are estimated l95 - "... and corresponding loss, and the presence..." → "... and corresponding loss, and (ii) the presence..." l103 - MM GAN  and WGAN should be in \textsc l119 - Add (\textsc{began}) somewhere in the sentence. l93-l120 - What about the different hyperparameters that only a few configurations have: $c$, $k_t$, $\gamma$. l155- In general when refer to a Figure/Table in Appendix please say that it is in the Appendix/Supplementary Material. l225- How bad is to get stuck in a local minima when doing this minimization? It seems that this implies that you are giving an upper bound on the performance (i.e., false negative rate might be lower). How many times did you repeat from different initializations. Please comment. l230- Which dimension did you consider for the latent space (d_z)? l239 - " as a popular alternative" → "as a popular alternative to GANs" l294- "with distance below" → "with distance to the manifold below"   **Updated After Rebuttal**  I see three main limitations of the paper: 1. Only one architecture is tested, 2. Hyperparameters search is not completely fair (in rebuttal the authors mention sequential Bayesian optimization as FW on this matter), 3. Very hard to replicate due to the large amount of computational resources needed.  Even with 1. and 2., I think the paper worths publication since one can build on this work to compare different architectures and more rational ways of searching for hyperparameters. Regarding 3, the authors open sourced code and trained models which is a step forward on reproducibility. Finally the current conclusions of the paper are too strong (this point needs to be revised).


Wavelet regression and additive models for irregularly spaced data
We present a novel approach for nonparametric regression using wavelet basis functions. Our proposal, waveMesh, can be applied to non-equispaced data with sample size not necessarily a power of 2. We develop an efficient proximal gradient descent algorithm for computing the estimator and establish adaptive minimax convergence rates. The main appeal of our approach is that it naturally extends to additive and sparse additive models for a potentially large number of covariates. We prove minimax optimal convergence rates under a weak compatibility condition for sparse additive models. The compatibility condition holds when we have a small number of covariates. Additionally, we establish convergence rates for when the condition is not met. We complement our theoretical results with empirical studies comparing waveMesh to existing methods.

This paper proposes regression methods using wavelets, which does not require irregularly spaced data or the number of observation to be a power of 2. The key idea is to interpolate the raw data to fitted values on the regular grid and run regression algorithm with l1-penalty, i.e., proximal gradient descent. It is natural to generalize additive models for a potentially large number of covariates. The authors analyze (minimax) convergence rates of the proposed methods over Besov spaces. In experiments, they benchmark their methods under some synthetic functions and report that they perform better than competitors.  The necessary assumptions of traditional wavelets regressions (equi-spaced and a power of 2 data) restricts to apply them into practical applications. Therefore, it is nice work as authors overcome this issue. The analysis for convergence rates justifies to solve l1-penelty optimization. However, the interpolation idea is very natural to deal with irregular data points. More detail comparisons with other interpolating works would be required.  In addition, the experiments are only benchmarked with a power of 2 observations. This is not enough to demonstrate that the proposed methods are practically applicable. Authors should report the cases with general number of observations. In addition, although the experiments works well in some artificial functions (e.g., polynomial, sine and etc), a lack of benchmarks under real-world functions makes the paper weaken. It is worth to report additional experiments under real-world regression problems.   It would be better to understand if authors give more intuitions of interpolation schemes, e.g., construction of R in section 2.2, and how this can be computed by providing quantitative complexity in section 2.3. In addition, some readers may wonder how the linear interpolation matrix R is defined in Theorem 1.  The minor typos: In line 76: \phi_{jk}(x) = 2^{j/2} “”\phi””(2^j x - k) In line 143: O(K log k) -> O(K log K) In line 177: ca -> can