Context-Dependent Upper-Confidence Bounds for Directed Exploration
Directed exploration strategies for reinforcement learning are critical for learning an optimal policy in a minimal number of interactions with the environment. Many algorithms use optimism to direct exploration, either through visitation estimates or upper-confidence bounds, as opposed to data-inefficient strategies like ✏-greedy that use random, undirected exploration. Most data-efficient exploration methods require significant computation, typically relying on a learned model to guide exploration. Least-squares methods have the potential to provide some of the data-efficiency benefits of model-based approaches-because they summarize past interactions-with the computation closer to that of model-free approaches. In this work, we provide a novel, computationally efficient, incremental exploration strategy, leveraging this property of least-squares temporal difference learning (LSTD). We derive upper-confidence bounds on the action-values learned by LSTD, with context-dependent (or state-dependent) noise variance. Such contextdependent noise focuses exploration on a subset of variable states, and allows for reduced exploration in other states. We empirically demonstrate that our algorithm can converge more quickly than other incremental exploration strategies using confidence estimates on action-values.

Summary: In this paper, the authors have described how context dependent, i.e. state and action pair dependent, Upper Confidence Bounds can be calculated for the Least Squares Temporal Difference (LSTD) Learning algorithm. They argue that this is a principled way of exploring the interactions with the environment because is specially important for least square methods because other strategies are either inefficient (e-greedy) or cannot be used in a meaningful manner (optimistic initialization). The computation of the upper confidence part is interesting and deviates from the standard proofs for Multi-arm bandit cases because the samples in case of LSTD are not independent of each other. The authors address this issue by imposing a bound using the (weaker) Chebychev's inequality instead of the Bernstein's inequality. In their evaluation, they use a bound on the total compute rather than on the number of episodes. Though non-standard, the evaluation procedure did make sense for the claims they make that their algorithm is able to attain better performance with fewer steps wasted exploring. The authors have provided pseudo-code for all their baselines and, hopefully, would make the source code for their experiments available, making the paper reproducible.  Key Contributions:  - First principled method for using optimistic Upper Confidence Bound algorithms with LSTD.  - Interesting use of Chebychev's inequality to handle the non-IID setting and allowing updates in an online fashion for RL.  The paper is clearly written and is easy to follow. The problem is interesting and their algorithm fills in a gap in the space of algorithms we have in RL. The derivation is novel through relatively straight forward, though some clarifications which have been relegated to the appendix ought to appear in the main body (see below). The baselines that the authors have chosen to compare with are competitive and their description of the experimental setup is exhaustive. The future directions of work also looks promising, and may lead to significant advances in the future.  That said, there are a few minor issues with the work. The regret analysis of their approach is preliminary at best, though, as stated, the future directions look promising. Some heuristics in their algorithms (e.g. handling third issue discussed in appendix C) explicitly depend on the tile coding of the states/actions. It is unclear how easy it will be to design similar heuristics in case a non-linear embedding of the input state is used (e.g. using NNs). Authors mention this as a potential direction for future work, but a comment about the reliance of the implementation on this particular encoding of the state in the main-text is warranted.  However, the writing/presentation of the proof should be improved. For example, in line 165, the term "This estimation" is ambiguous. The equation after line 167 does not seem to follow from the equation presented above, but rather directly from the equation above line 165. Also, the last inequality above line 175 is true only with high probability, which is not clearly mentioned. The relationship of c√(a^2 + b^2) to the previous equation can also use more explanation. It is also important to state in the last paragraph of the section (instead of in the appendix) that since the estimates for noises are *not* available, the TD error will be used as the proxy. As this detail is somewhat of a departure from the multi-arm bandit setting, it would be useful for readers familiar with that domain.  Overall, I believe that the paper makes an interesting contribution and should be accepted for publication.  Minor issues:  - line 94: "direclty"  - line 129: "When" -> "we"  - line 203: "are constitute"  - line 205: "high-lighted" -> "highlighted"  - line 229: "it's" -> "its"  - "River Swim" task does not describe the termination condition/goal state in the description.  - line 280: "the the"  - lines 285 -- 287: the paragraph is out of place.  - line 289: "a sound upper bounds": unconventional use of the term 'sound'.  - Error bars are missing from several figures (notably, 3(c) and 5).  - Figure 3 caption: LSTD-CG-... are not defined. Also in Figure 5 caption, CG seems to be used without prior definition.  - Is it possible to also show somehow how quickly the confidence bounds for various states decrease and how it is related to the value of the state? It would be interesting to see actually how much exploration effort the algorithms wastes on low valued states.  ----  Update: Having read the authors' response, I would recommend that the explanations/changes they have proposed for (1),(2) in the minor changes section ought to be included in the main paper. Overall, my opinion about the paper remains unchanged.


Improving Online Algorithms via ML Predictions
In this work we study the problem of using machine-learned predictions to improve the performance of online algorithms. We consider two classical problems, ski rental and non-clairvoyant job scheduling, and obtain new online algorithms that use predictions to make their decisions. These algorithms are oblivious to the performance of the predictor, improve with better predictions, but do not degrade much if the predictions are poor.

Update after rebuttal & discussion: I am satisfied with the authors' response, and while I agree with another reviewer that a more general setup/investigation would be nice, I believe the present work focussing on special cases/problems is of sufficient novelty and marks a good first step toward more general frameworks on the future research agenda. Thus, I uphold my vote and strongly recommend accepting this submission.  -------- The present paper introduces deterministic and randomized algorithms for online ski rental and non-clairvoyant single-machine scheduling problems that incorporate (machine-learned) predictions in order to improve competitiveness in case of good predictions while maintaining robustness, i.e., avoiding performance degradation, in case of bad predictions. Both problems are intuitively amenable to make use of predictions of rental or job processing times, respectively, and the proposed algorithms show how such intuition can be exploited on-the-fly for robust and competitive online solution algorithms. Thus, the work is well-motivated and contributes practically interesting methods to well-studied and relevant problems. The paper is also generally well-written and mostly easy to follow. Therefore, I recommend accepting this submission.  Although the considered online-optimization problems may be easily described, they are in fact somewhat challenging. A clear strength of the present work is fusing known approximation approaches for either problem with the assumed availability of data predictions into new online algorithms that are  *provably* robust in the sense that they do not deteriorate much (w.r.t. the best-known competitive ratios – i.e., ratio between computed solution and the theoretical „offline, clairvoyant“  optimum – achievable by algorithms not exploiting predictions) in case the predictions are rather bad, but are *provably* able to improve competitiveness if the incorporated predictions are good. Some numerical experiments further support the theoretical results and demonstrate practicality of the proposed schemes. I particularly like the theoretical disconnection of the introduced data-driven algorithms and the actual origin of the predictions that are exploited.  A weakness of the paper is that some parts are a bit vague or unclear, and that it contains various typos or formatting issues. I shall provide details in the following list; typos I found will be listed afterwards.  1. In the section about related work, I was wondering whether there is no related literature from the robust scheduling literature. I am not overly familiar with the area, so I cannot give concrete references, but I am aware that robust scheduling is a very large field, so I would be surprised if there were no robust online scheduling problems/algorithms that are connected to the work at hand in one way or another. Could you please shed some light on this aspect (and provide some reference(s) if appropriate)?  2. From the description in the „main results“ section of the Introduction, the precise connection between competitive ratio and robustness or consistency did not become entirely clear to me. Please provide some additional clarification on what is to be compared here – in particular, I also stumbled over why (as stated in several corresponding theorems) one ends up with competitive ratios „at most min{ robustness-ratio, consistency-ratio }“. At first glance, I had thought that if an algorithm does well for good predictions but the robustness bound is bad, that the latter should dominate. As this is apparently not the case, I ask for a brief explanation/clarification, preferably already in the introduction, as to why the minimum of the two values yields a bound on the competitive ratio.  3. In Algorithm 3, please clarify what „chosen“ means exactly. (Is it „drawn uniformly at random from the k (or l, resp.) values q_i (r_i, resp.)“ ? Or „chosen as index yielding maximal q_i/r_i-value“? Or something else?) Also, on a higher level, what makes q_i and r_i probability distributions?  4. On p.5, line 151 (multi-line inequality chain): I could not immediately verify the 3rd equation (in the middle line of the chain) – if it is based on a known formula, I do not recall it (maybe provide a reference?), otherwise please elaborate.  5. Please provide a reference for the statement that „... no algorithm can yield any non-trivial guarantees if preemptions are not allowed“ (lines 187-188, p. 6, 1st paragraph of Section 3).  6. At the very end of Section 3, you omit a proof for lack of space. If possible, please do include the proof in the revision (at least as supplementary material, but maybe it fits into the main paper after all).   7. Beginning of Section 2.2: should it not be „... must incur a competitive ratio of at most $b$.“ (not „at least“) ? The worst case would be x=1, then the ratio would be b/1=b, but in all other cases (x>1), the ratio is either b/x or eventually b/b=1 (as soon as x>b).   8. In Section 2.4, perhaps add „as in Theorems 2.2 and 2.3, respectively.“ to the sentence „Both Algorithms 2 and 3 extend naturally to this setting to yield the same robustness and consistency guarantees.“ ?   Finally, here's a (sorry, quite pedantic) list of typos or formatting improvement suggestions for the authors' consideration: -- line 1: „machine-learned“ (hyphen is missing here, but used subsequently) -- l. 13: I think it should be „aimed at tackling“, not „aimed to tackle“  -- l. 18: missing comma between „Here“ and „the effort...“ -- The dollar sign (used in the context of the ski rental problem) somehow looks awkward; actually, could you not simply refrain from using a specific currency?  -- Regarding preemptions, I think „resume“ is more common than „restart“ (ll. 54, 186) -- l. 76: I think it should be „used naively“, not „naively used“ -- Proof of Lemma 2.1, last bullet point: to be precise, it should be „ … x < b+x-y ... “ -- l. 119: comma is missing after „i.e.“ -- Proof of Theorem 2.2: There is a „d“ that should probably be a „b“ (ll. 134 and 136) -- Throughout the paper, you use four different ways to typeset fractions: in-line, \frac, \tfrac, and that other one with a slanted „frac-line“. I would appreciate a bit more consistency in that regard. In particular, please avoid using \frac when setting fractions in-line, as in Theorem 2.2, where it messes up the line spacing.  -- l. 144 (1st sentence of Sect. 2.3): comma missing after „In this section“ -- p. 5: the second inequality chain violates the textwidth-boundary – can probably easily be fixed be linebreaking before the last equality- rather than the first inequality-sign.  Also, „similar to“ should be „similarly to“ (ll. 157 and 159). -- l. 216, comma missing after „i.e.“ -- Lemma 3.2: Statement should start with a „The“ (and avoid \frac in-line, cf. earlier comment) Given the end of the proof, the statement could also be made with the less-coarse bound, or writing „less than“ instead of „at most“. -- l. 234: insert „the“ between „...define d(i,j) as“ and „amount of job“ (perhaps use „portion“ instead of „amount“, too) -- l. 238: missing comma between „predicted“ and „the longer job“ -- l. 243: suggest replacing „essentially“ by „asymptotically“ -- l. 249: comma missing after „Finally“ -- Theorem 3.3: Statement should start with a „The“. -- l. 257: comma missing after „e.g.“ -- l. 276: hyphenate „well-modeled“ -- l. 279: comma missing before „where“ -- Ref. [11]: use math-mode for e/(e-1), and TCP should be upper-case -- Ref. [14]: „P.N.“, not „PN“, for Puttaswamy's initials -- Ref.s [15, 16, 17]: Journal/Source ? Page numbers?  -- Generally, the reference list is somewhat inconsistent regarding capitalization (in paper and journal titles) and abbreviations (compare, e.g., [11] Proceedings title with [21] Proceedings title).


Watch Your Step: Learning Node Embeddings via Graph Attention
Graph embedding methods represent nodes in a continuous vector space, preserving different types of relational information from the graph. There are many hyperparameters to these methods (e.g. the length of a random walk) which have to be manually tuned for every graph. In this paper, we replace previously fixed hyperparameters with trainable ones that we automatically learn via backpropagation. In particular, we propose a novel attention model on the power series of the transition matrix, which guides the random walk to optimize an upstream objective. Unlike previous approaches to attention models, the method that we propose utilizes attention parameters exclusively on the data itself (e.g. on the random walk), and are not used by the model for inference. We experiment on link prediction tasks, as we aim to produce embeddings that best-preserve the graph structure, generalizing to unseen information. We improve state-of-the-art results on a comprehensive suite of real-world graph datasets including social, collaboration, and biological networks, where we observe that our graph attention model can reduce the error by up to 20%-40%. We show that our automatically-learned attention parameters can vary significantly per graph, and correspond to the optimal choice of hyperparameter if we manually tune existing methods.

The paper proposes a new algorithm for learning node embedding, by bringing together the attention model and the graph likelihood objective function suggested in a recent work [2]. By learning the context distribution determining the coefficients of powers of the transition matrix, it leads to a more flexible representation learning model.  Quality:  Although the technical content of the paper seems to be correct, I would like to point out some issues in the paper: - In Line 121, the sentence "In practice, random walk methods based on Deepwalk do not use $C$ as a hard limit" is not generally correct because many methods (such as Node2vec) perform fixed length walks.  - Although the proposed method is described  using the random walk concept, it computes the expected value of the number of node pair occurrences in a length $c \sim \mathcal{U}(1,C)$ by applying matrix multiplication. On the other hand, Deepwalk or Node2vec implicitly approximate it, by performing only a limited number of random walks. Therefore, I think it could be also better to see its performance against the methods such as HOPE.  - In classification, only two datasets have been used. I would strongly recommend  to examine the performance of the method over different training and test sizes.  - In the Table b of Figure 1, the figures for ca-AstroPh and ca-HepTh are not depicted. Those results are important to better understand the behavior of the proposed method: in Table 1, Graph Attention uses windows size of $C=10$ but node2vec uses $C=2$ or $C=5$.  - Although experimental results are given with standard deviation for the Graph Attention method, in Table 1 only mean scores are shown for the other methods.  - Is the proposed method scalable for large networks? This is a very important question, since the proposed algorithm performs matrix factorization.  Clarity: I think the paper is well-organized and quite readable. In the beginning, the authors shortly mention two previous works on "Attention Models" and "Graph Likelihood", and then they introduce the proposed method by incorporating ideas from the above two methodologies. However, I would like to make some suggestions for the notation used in the paper:  - Between lines 29-30, the symbol $C$ is used to indicate the length of the walk and the italic $\mathcal{C}$ is also used to denote the context distribution but it can be confused with the other notation $C$.  - In Line 63, the expression "$|E|$ non-zero entries" is not correct. For example the adjacency matrix of a simple graph have $2|E|$ non-zero entries due to symmetry.  - In line 72, if $E[v_i]$ are the outgoing edges from $v_i$ then $sample(E[v_i])$ returns an edge and not a node -- so it could be better to define $E[v_i]$ as the neighbourhood of vertex $v_i$.  - In line 121, the sentence "In practice, random walk methods based on Deepwalk do not use $C$ as a hard limit" is not generally correct because many methods such as Node2vec perform fixed length walks.  Originality: The authors extend a previous work on the “graph likelihood objective” [2] with the Attention Model, and the proposed method outperforms the state-of-the-art baselines for the link prediction task.  Significance: This work incorporates the attention model with the "graph likelihood" objective function proposed by a recent work [2], in such a way that it enables to learn context distribution which is used in the paper to compute the expected value of the number of co-occurrences of nodes. Although the context distribution corresponds to the coefficients of the powers of the transition matrix, I believe that the same idea can be applied for different models. The experiments for node classification appear to be insufficient to evaluate the model, but the suggested method shows significant improvements for the link prediction task.


Bayesian Nonparametric Spectral Estimation
Spectral estimation (SE) aims to identify how the energy of a signal (e.g., a time series) is distributed across different frequencies. This can become particularly challenging when only partial and noisy observations of the signal are available, where current methods fail to handle uncertainty appropriately. In this context, we propose a joint probabilistic model for signals, observations and spectra, where SE is addressed as an exact inference problem. Assuming a Gaussian process prior over the signal, we apply Bayes' rule to find the analytic posterior distribution of the spectrum given a set of observations. Besides its expressiveness and natural account of spectral uncertainty, the proposed model also provides a functional-form representation of the power spectral density, which can be optimised efficiently.
Comparison with previous approaches, in particular against Lomb-Scargle, is addressed theoretically and also experimentally in three different scenarios. Code and demo available at github.com/GAMES-UChile.

Spectral estimation is the task of finding the power associated with a signal across the frequency range of the signal. In classical signal processing applications, this is a solved problem when the time series is fully observed and equally spaced in time. However, in many cases, there are either missing observations, or the data is unevenly spaced in time. Here, probabilistic methods can be extremely valuable, as we can use nonparametric methods to provide better interpolation of the frequency domain, whilst handling the uncertainty in a principled way. This paper proposes a method to tackle this problem, with exact inference.  A crucial component of this paper is windowing of the signal: without it, the Fourier transform of the GP would not exist. The authors here chose a version of the exponential window, which I presume is chosen for mathematical convenience rather than for any particular property (such as spectral leakage properties). From here, paper follows through the mechanics of deriving the local-spectrum GP using this windowing technique, the full derivation of which appears in Appendix 1.1.   As I understand it, inference proceeds by sampling the time-domain GP, and then sampling in the spectral domain conditioned on the observations and given the time-domain functions.   The final part is to compute the spectral energy, which is the (random) squared norm of the real and imaginary parts, which since they are independent GPs, results in a Chi-squared process.  The rest of the paper is concerned with model consistency, and showing that the proposed model forms the limit of the popular Lomb-Scargle method (used for non-uniformly sampled data) as the number of sinusoids tends to infinity. Empirical analysis demonstrates that the proposed method is competitive with the Lomb-Scargle method, whilst being able to proved uncertainty estimates.  Overall this is a well written paper, and provides significant contributions to the field of Bayesian spectral analysis.  Specific comments: I find the use of the same K to represent the kernel function (of two variables) and the stationary kernel function (taking the difference between variables as input) confusing - perhaps $K_S$ could be used? When is K a function of 1 and two variables? (e.g. in 64 it is again used as $K(T)$). 71 “the signal and its spectrum are jointly Gaussian”. Is the spectrum Complex not Gaussian? What does it mean to say that they are jointly Gaussian? Eq 21, t on LHS is a vector 212 Fig 5.1 -> Fig 4? 233 what does “training” mean in the context of the Lomb-Scargle method? 252 available -> able  Note: Thanks to the authors for the clarifications in the feedback.


Optimal Subsampling with Influence Functions
Subsampling is a common and often effective method to deal with the computational challenges of large datasets. However, for most statistical models, there is no well-motivated approach for drawing a non-uniform subsample. We show that the concept of an asymptotically linear estimator and the associated influence function leads to asymptotically optimal sampling probabilities for a wide class of popular models. This is the only tight optimality result for subsampling we are aware of as other methods only provide probabilistic error bounds or optimal rates. We also show that these optimal weights can differ depending on whether the task is parameter estimation or prediction. Furthermore, for linear regression models, which have well-studied procedures for non-uniform subsampling, we empirically show our optimal influence function based method outperforms previous approaches even when using approximations to the optimal probabilities.

The paper presents a sampling methodology for statistical estimation tasks, in which the weight of a sample (to be included in final estimation) is determined by the sample's Gateaux derivative. The authors point out that Gateaux derivatives are defined for several estimators of note, and that asymptotically characterize estimate errors for so-called asymptotically linear estimators, which is also a very broad class. The authors then propose a probabilistic sampling method in which samples are included with a probability proportional to the norm of the gateaux derivative, and show that this is optimal, in that it minimizes the trace of the variance of the estimator under certain conditions. They then proceed to apply this method to least squares estimation, GLMs, and quantile regression, on all of which estimation leads to improved performance (both in estimation and prediction) compared to alternative methods for subsampling.  This is an interesting, solid contribution, that addresses a fundamental problem. On the negative side, the paper is unpolished/somewhat written in a rush, and is not appropriately positioned with respect to related work. This seems a significant drawback when dealing with a topic as fundamental as sampling.  To begin with, the related work (there is no dedicated section) is very sparse, citing mostly TCS papers; what is the state of the art in sampling in statistics literature? Have gateaux derivatives been used before for this purpose? How does the present work relate to other  uses of gateaux derivatives in statistics and estimation in particular?   On a similar vein, the paper mentions several results, presumably already known without any citations or context whatsoever. Asymptotically linear estimators are an example. Clearly, gateaux derivatives for the specific estimation tasks in Section 4 must also have been derived before, and although the discussion of the resulting sampling methods is interesting, again credit for this is due. Similarly, competing sampling methods are sometimes vaguely disparaged as "only provid(ing) probabilistic or rate guarantees". What are the competing methods, and what is the exact formulation of their probabilistic and rate guarantees?  Given the broad array of estimators termed to be asymptotically optimal, one would expect more examples listed and evaluated (beyond the usual LSE and GLS). The same holds for competing sampling methods.  Finally, the theoretical statement (Theorems 1 and 2) and their preamble are too dense, making it difficult to assess the breadth and applicability of the conditions required  for the theorems to hold.  ---Added post-rebuttal--- I appreciate the authors' detailed response, and the explanation of the lack of work in statistics on subsampling. If some of these comments could make it to the paper, that would be great. --------------------------------


Fairness Through Computationally-Bounded Awareness
We study the problem of fair classification within the versatile framework of Dwork et al. [6], which assumes the existence of a metric that measures similarity between pairs of individuals. Unlike earlier work, we do not assume that the entire metric is known to the learning algorithm; instead, the learner can query this arbitrary metric a bounded number of times. We propose a new notion of fairness called metric multifairness and show how to achieve this notion in our setting. Metric multifairness is parameterized by a similarity metric d on pairs of individuals to classify and a rich collection C of (possibly overlapping) "comparison sets" over pairs of individuals. At a high level, metric multifairness guarantees that similar subpopulations are treated similarly, as long as these subpopulations are identified within the class C.

The paper studies the problem of fair classification in the framework of ``fairness through awareness" of Dwork et al which requires similar individuals to be labeled similarly by the classifier. Crucially, the work of Dwork et al requires the existence of a similarity metric between the pairs of individuals; an assumption that is ``one of the most challenging aspects" of the framework. This work relaxes this assumption by requiring similar identifiable subpopulations to be treated similarly by the classifier. The authors assume while the metric is not known, it can be queried for specific pairs of individuals by the learner. The authors show under what conditions this relaxed notion of fairness can be efficiently satisfied when the learner's goal is to maximize classification's utility subject to satisfying this new relaxed fairness constraint.  Questions: (1) Does the post-processing approach in section 3.1 come with any formal theoretical guarantees? Woodworth et al 2017 have shown that post-processing to satisfy equality of odds can be highly sub-optimal in the worst case.  Strengths: (1) While the paper is similar in spirit to the work of Hebert-Johnson et al and Kearns et al, I feel like relaxing the fairness through awareness framework is a rather interesting contribution.  (2) The paper is rather well-written and the high level ideas are easy to follow.   Weaknesses: (1) If I understand correctly, the authors assume a realizable setting for the generation of labeled examples i.e. the unlabeled examples are generated by a distribution D and then the unlabeled examples are labeled by a deterministic function. Can the authors clarify if this is the correct understanding? If so, given that the realizability assumption is rather restrictive/outdated, can the authors elaborate how/whether their results can be extended to an unrealizable setting? (2) The authors draw connections to agnostic learning problems to solve the search step of algorithm 1. Agnostic learning is generally hard computationally but regardless have been commonly used/solved in practices. While I strongly believe that theoretical contributions are indeed important, in a topic such as an algorithmic fairness, it would be rather crucial to see how the techniques proposed in this paper fare when dealing with real datasets even if heuristic techniques are used to solve the computationally hard subroutines. (3) The assumption that the hypothesis class is the class of linear functions is rather restrictive. As far as I can understand the constraint in the convex program might not be convex if a different hypothesis class was used.  Typos and minor comments: (1) line (17): attributes (2) line (55): requiring (3) line (83): linear function "with" bounded weights (4) No "." needed in the middle of line 247 (5) typo in line 292/293 -- last sentence of the paragraph (6) the paper uses rather unconventional notation for the ML audience: for example in a typical ML paper delta is used for confidence parameter, epsilon is used for accuracy, n/d is used for dimension and m is used for sample complexity. While this is a matter of choice, the paper would be much easier to read for a NIPS/ICML audience if the standard notation were followed.  ------------------------------------------------------------- I read the other reviews and the author response.


Safe Active Learning for Time-Series Modeling with Gaussian Processes
Learning time-series models is useful for many applications, such as simulation and forecasting. In this study, we consider the problem of actively learning time-series models while taking given safety constraints into account. For time-series modeling we employ a Gaussian process with a nonlinear exogenous input structure. The proposed approach generates data appropriate for time series model learning, i.e. input and output trajectories, by dynamically exploring the input space. The approach parametrizes the input trajectory as consecutive trajectory sections, which are determined stepwise given safety requirements and past observations. We analyze the proposed algorithm and evaluate it empirically on a technical application. The results show the effectiveness of our approach in a realistic technical use case.

This paper proposes safe active learning algorithm for learning time-series models with piecewise trajectory sections. The basic model for the trajectory section is Gaussian Process, and the safety criterion is modeled as probability of certain safe value, which is modeled as a continuous function of inputs, being in the right region is greater than some threshold. The active learner explores the input space by generating trajectories that maximizes certain criterion in system identification. The paper also provides theoretical analysis in the safety perspective, as well as in the predictive variance reduction perspective. In the evaluation, besides simulated examples on predefined continuous functions and low dimensional models, this paper provides a specific use case for learning surrogate model of the high-pressure fluid system.   Significance The active learning here is not based on pool of data or stream of data. This work is more of active generation of the piecewise input trajectory in order to learn system property models while receiving output from the physical system and keeping the generation in the safety region. Safe exploration in the same flavor I believe has been studied before in regression when there is some hard threshold. This paper generally makes good contribution to safe active learning in system identification.  My concern though is that this work is not general enough for time series data. It mainly focuses on learning system models for physical systems. We need to have a good model for piecewise trajectory generation, we need to have continuous safety value function, and we need to have efficient optimization method for maximizing the optimality criteria when exploring. Therefore, my general suggestion here is to summarize all the conditions clearly in the paper for efficiently applying the method in practice.  Quality I have several questions which I wish could be answered to help me better judge the quality of the paper:  1. There is this constrained optimization problem that needed to be solved for every iteration of exploration. And there is several different criteria mentioned. But I did not find anything about how to solve the problem efficiently. Also, in evaluation, example 2 does not solve the optimization problem while in the use case is not clearly mentioned what criteria is used there.  2. Instead of using some hard threshold as safety criterion, the probability based safety criterion in this paper seems just a little softer. What is the consideration here?   3. There are actually many small hyperparameters needed to be set beforehand for this method. For example, the length of the initialization trajectories and the timestep in each trajectory section. There is no discussion about these in the paper. Both 10 and 25 are used in different experiments. I want to know what is the general principle of choosing them.  Originality This work reminds me of a previous thread of work on safe exploration based also on GP, but not on time series (like Safe Exploration for Optimization with Gaussian Processes), which is not cited in this paper. I think more discussion in related work may be needed.   Clarity The paper is generally well-written with clear presentation and examples.


Which Neural Net Architectures Give Rise to Exploding and Vanishing Gradients?
We give a rigorous analysis of the statistical behavior of gradients in a randomly initialized fully connected network N with ReLU activations. Our results show that the empirical variance of the squares of the entries in the input-output Jacobian of N is exponential in a simple architecture-dependent constant , given by the sum of the reciprocals of the hidden layer widths. When is large, the gradients computed by N at initialization vary wildly. Our approach complements the mean field theory analysis of random networks. From this point of view, we rigorously compute finite width corrections to the statistics of gradients at the edge of chaos.

The paper studies the gradient vanishing/exploding problem (EVGP) theoretically in deep fully connected ReLU networks. As a substitute for ensuring if gradient vanishing/exploding has been avoided, the paper proposes two criteria: annealed EVGP and quenched EVGP. It is finally shown that both these criteria are met if the sum of reciprocal of layer widths of the network is a small number (thus the width of all layers should ideally be large). To confirm this empirically, the paper uses an experiment from a concurrent work.  Comments:  To motivate formally studying EVGP in deep networks, the authors refer to papers which suggest looking at the distribution of singular values of the input-output Jacobian. If the singular values are far from 1 and take both small and large values, it implies the network is suffering from exploding gradient or vanishing gradient problem or both. However, to study the Jacobian analytically, the authors propose to study the annealed and quenched criteria, which in some sense measure how much fluctuation the Jacobian has across its elements. The analysis suggests a high fluctuation is bad while small fluctuation avoids EVGP.   If I understand the paper correctly, my main concern with the proposed criteria is that having a small fluctuation in the Jacobian across its elements would mean all elements are similar. In this case, all but one sigular values of the Jacobian would collapse to ~0, which is bad. So the proposed criteria seems problematic to me.  Regarding the cited experiment for an empirical justification of the proposed theory, the paper from which the figure is taken proposes an alternative theoretical reason for this behavior. This makes it hard to understand whether a more appropriate reason for this behavior is the justification in the original paper or this paper.  The paper is clearly written and original in the sense that it proposes to study EVGP but may be problematic.  Comments after reading the rebuttal:  I am still not sure I understand how the input-output Jacobian Z is well conditioned (singular values are close to one) if each entry of the Jacobian Z_pq^2 has a finite expected value (Theorem 1 Eq 10) and variance as shown in Eq 11 (of course these results are extended to talk about the empirical variance of the squared entries of Z in theorem 2 and 3). For instance, consider Z to be of dimension 400x800. If I generate the entries of Z by sampling from a gaussian with mean 1/400 and variance 1/400 (I have assumed n0=400 and width of the network layers to be infinite in theorem 1 to roughly use these numbers), I get a matrix Z that has singular values spanning the range 0.02 to 1.4 (thus condition number 1.4/0.02=70>>1).  Another point I would like to raise is that even if the input-output Jacobian is well conditioned, it is essentially a product of terms like d h_t/d h_{t-1}, where h_t is the hidden state at depth t. Now I can have a matrix Z that is even identity, but is the result of the product of two matrices as Z = AB, where A has very large singular values and B has the inverse singular values of A, thus resulting in the singular values of 1 for Z. Here A and B signify partial derivatives like d h_t/d h_{t-1} for 2 consecutive layers. As can be seen, the gradient for layer corresponding to A are exploding while that of B will are vanishing.  I feel these aspects need to be made clear in the paper.


Explanations based on the Missing: Towards Contrastive Explanations with Pertinent Negatives
In this paper we propose a novel method that provides contrastive explanations justifying the classification of an input by a black box classifier such as a deep neural network. Given an input we find what should be minimally and sufficiently present (viz. important object pixels in an image) to justify its classification and analogously what should be minimally and necessarily absent (viz. certain background pixels). We argue that such explanations are natural for humans and are used commonly in domains such as health care and criminology. What is minimally but critically absent is an important part of an explanation, which to the best of our knowledge, has not been explicitly identified by current explanation methods that explain predictions of neural networks. We validate our approach on three real datasets obtained from diverse domains; namely, a handwritten digits dataset MNIST, a large procurement fraud dataset and a brain activity strength dataset. In all three cases, we witness the power of our approach in generating precise explanations that are also easy for human experts to understand and evaluate.

The paper proposes a Contrastive Explanation Method (CEM) with the goal of justifying, or explaining, the decisions made by deep neural networks. A notable characteristic of the proposed method is that for every class of interest it aims at identifying two sets of features that can be used to jutify the decisions related to a given class. The first type, pertinent positives (PN), are features whose occurrence are the minimum necessary to justify the classification prediction.  The second type, pertinent negatives (PN) are features whose absence is necessary to justify the prediction of a given class. Given an input image, explanations are given by extending the output by highlighting these two types of features.    Experiments on the MNIST [38], ABIDE-I [11] and a preocurement fraud dataset [9] show the performance of the proposed method.  ===================  Overall the content of the manuscript is sound, clear and easy to follow.  I find valuable the fact that the evaluation includes experiments datasets on multiples data modalities, i.e. visual data (Sec. 4.1 and 4.3) and text-based data (Sec. 4.2). In addition, this seems to be one of the first works focusing  explictly on the identification of pertinent negatives.  In addition, there seems to be code available in the supplementary material. If the plan is to release code to run the proposed method, this could be good for reproducibility of the presented results.  Having said this, my concerns with the manuscript are the following:   Even when I find novel the fact that the proposed method is explictly designed to identify and indicate PN features, this is not the first work to highlight this type of negative feature. Examples of existing works include Samek et al., 2017 and more recently, Oramas et al. arXiv:1712.06302, which as part of their generated output provided some explanations supporting the absence of specific features. I consider positioning with respect to these works a must in order to effectively highlight the novelty brought by the proposed method.  In Section 3, it is stated that the proposed method aims at identify the features that could serve as PPs and PNs for model explanation.  However, in several of the figures/Tables displaying the explanations of the proposed method just one output of each type of feature (PP or PN) is given. To these outputs already highlight more than one feature of each type? if yes, how can we distinguish between the different features highlighted within a type.  if not, an multiple outputs are posible, what is the criterion to select which features to highlight as part of the output?  In Section 3 it is mentioned the need for an autoencoder (AE). DUring the evaluation, this autoencoder is only employed in 2/3 of the conducted experiments. At this point it is not clear to me when this autoencoder is necessary or not. Is there a principled means to verify whether this autoencoder is necessary?  Experiments are conducted on relatively simple and small datasets. Moreover, the Procurement Fraud dataset used in Section 4.2 does not seem to be public. If this is indeed the case, this would limit future comparisons with respect to this work.  In addition, while the manuscript stresses in the abstract the black box characteristic of deep neural networks (DNNs), the tested architectures are relatively simple and shallow. Therefore the potential of the proposed method is unclear in more complex scenarios. For the case of DNNs for visual data, I would suggest evaluating on one of the standard architectures, e.g. alexnet, VGG, ResNet, etc., for which pre-trained models can be found online. Results on one of this deeper more complex settings will strengthen the potential of the proposed method.  Finally, the presented results are mostly qualitative.  In this regard I would suggest performing a either: a) an occlussion analysis(Zeiler et al., ECCV 2014, Samek et al., 2017) , b) a pointing experiment (Zhang et al., ECCV 2016) or, c) a measurement of explanation accuracy by feature coverage (Oramas et al. arXiv:1712.06302).  Any of these protocols should allow to quantitatively evaluate the relevance of the features highlighted by the proposed as part of the explanation. In addition, I find the set of provided qualitative examples quite reduced. In this regard, I encourage the authors to update the supplementary material in order to show extended qualitative results of the explanations produced by their method.  I would appreciate if my concerns are addressed in the rebuttal.   ========================== Full references ==========================    Jose Oramas, Kaili Wang, Tinne Tuytelaars "Visual Explanation by Interpretation: Improving Visual Feedback Capabilities of Deep Neural Networks" (2017), arXiv:1712.06302.  Wojciech Samek, Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, and Klaus-Robert Müller: "Evaluating the Visualization of What a Deep Neural Network has Learned" IEEE Transactions on Neural Networks and Learning Systems, 28(11):2660-2673, 2017  Matthew D. Zeiler and Rob Fergus.  "Visualizing and understanding convolutional networks" ECCV, 2014.  Jianming Zhang, Zhe Lin, Jonathan Brandt, Xiaohui Shen, Stan Sclaroff "Top-down Neural Attention by Excitation Backprop" ECCV 2016.    ======================================== Post Rebuttal Update ========================================  I thank the authors for their rebuttal. The provided rebuttal satisfied to some extent most of the points I raised on my review.  I share the R1's view that the proposed idea is interesting. Though, I do not consider the fact of highlighting features that will produce a negative effect that novel. Indeed, most current work in interpretable machine learning (e.g. Selvaraju et al., arXiv:1610.02391,  Chattopadhay et al., WACV'18, Reddy et al., arXiv:1708.06670 ) only highlights features that are present in the instance. However, highlighting other, potentially conflicting, features related to other classes will mostly require generating such visualizations for other non-max classes with high score for the instance being explained.  I share the concerns of R1 on that the evaluation aspect could be improved. Currently, results are reported mostly on a  qualitative fashion. While some deterministic behavior is expected when giving the PPs and PNs as part of the input to the neural network, the identification of these PPs and PNs is not perfect. I still would like to see empirical (quantitative) evidence of to what these elements satisfy that expected behavior.  Moreover, for the case of the MNIST experiment, the rebuttal (responding to R1) states that there is no widely accepted quantitative analysis for measuring explanations on visual datasets. Indeed, there is no unique metric for this yet. However, as I stated on my review, there are several evaluation methods that have been proposed and can help position the proposed method appropriately.   At this point, the rebuttal does not change my initial opinion of the manuscript. I still consider the manuscript a good paper and a good match for NIPS.


Recurrent Relational Networks
This paper is concerned with learning to solve tasks that require a chain of interdependent steps of relational inference, like answering complex questions about the relationships between objects, or solving puzzles where the smaller elements of a solution mutually constrain each other. We introduce the recurrent relational network, a general purpose module that operates on a graph representation of objects. As a generalization of Santoro et al. [2017]'s relational network, it can augment any neural network model with the capacity to do many-step relational reasoning. We achieve state of the art results on the bAbI textual question-answering dataset with the recurrent relational network, consistently solving 20/20 tasks. As bAbI is not particularly challenging from a relational reasoning point of view, we introduce Pretty-CLEVR, a new diagnostic dataset for relational reasoning. In the Pretty-CLEVR set-up, we can vary the question to control for the number of relational reasoning steps that are required to obtain the answer. Using Pretty-CLEVR, we probe the limitations of multi-layer perceptrons, relational and recurrent relational networks. Finally, we show how recurrent relational networks can learn to solve Sudoku puzzles from supervised training data, a challenging task requiring upwards of 64 steps of relational reasoning. We achieve state-of-the-art results amongst comparable methods by solving 96.6% of the hardest Sudoku puzzles.

This paper deals with designing a new neural network structure for problems involving one or more steps of relational reasoning. The authors propose a generalization of the Relational Network (RN) architecture proposed by Santoro 2017. Whereas RN does one step of processing on a fully-connected graph composed of nodes representing objects, the Recurrent Relational Network does multiple time steps of processing, maintaining a hidden state per node and parametrizing messages between nodes at each time step as a MLP.   The new architecture is evaluated on several benchmarks. The first is bAbI, which is a popular text-based question-answering dataset consisting of 20 different type of tasks, each of which involves receiving several supporting facts and answering a question related to those facts. The proposed method solves all 20 tasks and seems to display less variability between different training runs in answering these questions compared to other published methods. The authors observe that bAbI is solved by their model without requiring multiple time-steps of processing in their model and so create a more difficult variant of CLEVR dataset (called Pretty-CLEVR), which is specifically built so that questions have varying degrees of reasoning. The authors evaluate their method, while varying the number of time steps of processing, and show that as they run the Recurrent Relational Network for more time steps, it is able to answer more difficult questions (in terms of more degrees of reasoning being required). Lastly, the method is evaluated on solving 9 x 9 Sudoku puzzles and achieves higher accuracy than a simple convolutional network and other graphical model based methods.   Pros The proposed model has a simple definition but seems powerful based on its performance on various benchmarks. The Pretty-CLEVR experiment explicitly shows the benefit of allowing more time steps of processing with regard to questions that require more degrees of reasoning  Cons Why not evaluate method on CLEVR dataset to see where method’s performance is relative to other published methods?  Comments  The model suggested in this work bears some similarity to the one used in Sainbayar 2016, which involves learning communication between multiple agents performing an RL-based task and the communication vector for one agent there is similarly summed across all other agents. Might be worth mentioning in the related works section. For the methods evaluated against in the Sudoku benchmark, are the results there from listed papers or based on implementation of those methods by the authors in order to compare? It would be good to make a note if any of the results are based on author’s implementation.  Santoro, Adam et al. A simple neural network module for relational reasoning. 2017. Sukhbaatar, Sainbayar et al. Learning Multiagent Communication with Backpropagation. NIPS 2016.


Deep Anomaly Detection Using Geometric Transformations
We consider the problem of anomaly detection in images, and present a new detection technique. Given a sample of images, all known to belong to a "normal" class (e.g., dogs), we show how to train a deep neural model that can detect out-of-distribution images (i.e., non-dog objects). The main idea behind our scheme is to train a multi-class model to discriminate between dozens of geometric transformations applied on all the given images. The auxiliary expertise learned by the model generates feature detectors that effectively identify, at test time, anomalous images based on the softmax activation statistics of the model when applied on transformed images. We present extensive experiments using the proposed detector, which indicate that our technique consistently improves all known algorithms by a wide margin.

The authors proposed an original approach to image anomaly detection. They proposed to define a set of geometric transformations and then to construct a classifier to predict to which type of transformation a given image is more similar. In case the classifier score is low, the image is considered as anomalous.   As a test set the authors considered standard image datasets like CIFAR, Fashion-MNIST and CatsVsDogs. However, there exist special datasets to evaluate anomaly detection methods for images. E.g.   http://odds.cs.stonybrook.edu/#table5  The UCSD anomaly detection is annotated: each video frame is annotated as anomalous/non-anomalous and sometimes it contains pixel-wise annotations. Such datasets are closer to real-life applications and should be used for testing.  As baselines the authors used  - one-class SVM based either on pixels, or on low-dimension representation from the neural network - energy-based models - deep autoencoding GMM - GANs The authors claimed that in case of one-class SVM they used the best hyperparameters as there is no approach how to tune them. In fact, this is not true. There are approaches how to tune hyperparameters of one-class SVM: - Wang, S., Liu, Q., et al.: Hyperparameter selection of one-class support vector machine by self-adaptive data shifting. Pattern Recognition 74 (2018) 198–211 - Thomas, A., Feuillard, V., et al.: Calibration of one-class svm for mv set estimation.
CoRR abs/1508.07535 (2015)  - E. Burnaev, P. Erofeev, D. Smolyakov. Model Selection for Anomaly Detection // Proc. SPIE 9875, Eighth International Conference on Machine Vision, 987525 (December 8, 2015); 5 P. doi:10.1117/12.2228794; http://dx.doi.org/10.1117/12.2228794  Moreover, additional gain in accuracy when using one-class SVM can be obtained through end-to-end learning of embeddings specifically for one-class classification. There exists an approach how to do that, namely,  https://arxiv.org/pdf/1804.04888.pdf Also see https://arxiv.org/pdf/1801.05365.pdf  Of course, the one-class SVM is not the most powerful approach for anomaly detection. Experiments on numerous benchmark datasets with ground truth that compared popular anomaly detection algorithms find that One Class SVM ranks at the bottom, see  We note that the top performer in [1] is the Isolation Forest algorithm [2], an ensemble of randomized trees. Thus it is important to include in the comparison Isolation Forest algorithm based on the same features, as one-class SVM!  [1] Emmott, A., Das, S., et al.: Systematic construction of anomaly detection bench-marks from real data. In: Proceedings of KDD ODD. (2013) [2] Liu, F., Ting, K., Zhou, Z.H.: Isolation forest. In: Proceedings of ICDM. (2008)  - In general, features, generated by neural networks, are believed to provide efficient characterization of image data. In particular, very often it is enough to use linear classifier in the feature space, produce by deep layers of the neural network. Thus, why did the authors use kernel one-class SVM (CAE-OC-SVM)? It seems that it can be enough to use linear one-class SVM with features, produced by usual deep network such as VGG.  - another issue concerning section 5.1. PCA should be used as a baseline! It is very often used in image processing engineering applications, including anomaly detection.  - page 3, line 104. The authors proposed to use AUC. However, this may be not appropriate measure to estimate performance of anomaly detector. In fact, in real datasets the number of anomalies is small compareв to the size of the normal class. Thus area under the precision-recall curve is better suited for such imbalanced cases.  - displayed formula after the line 100: after the first line of the displayed formula a comma sign should be used  - The idea to use some class of geometric transformations T looks similar to the idea of image set augmentation. It is obvious that “richness” of the used class of geometric transformations should significantly influence anomaly detection accuracy. However, the authors did not investigate how anomaly detection accuracy depend on the diversity of the used class of geometric transformations. To which extent obtained anomaly detection results are sensitive to properties of this class?  - Practical image classification problems often contain a lot of classes (so-called extreme classification problems). Is the proposed method capable to deal with such case? To which extent obtained anomaly detection performance is sensitive to the number of classes, used to model anomalous observations?  - in section 5.3 the authors consider issues about hyperparameters. Are there any recommendations how to select a particular architecture of the neural network? Should we try to find such architecture which provides as accurate classification of types of image transformations as possible?   - in the paper by Liang et al. “Enhancing the reliability …” they proposed the method that using temperature scaling and adding small perturbations to the input can separate the softmax score distributions between in- and out-of-distribution images, allowing for effective detection of out-of-distribution images. In fact, in the current paper the authors proposed to use a set of geometric transformation as such perturbations.   Conclusion: - the topic of the paper is important - the paper is well-written. However, still there are a number of open questions, see comments above. - the results are novel. However, the novelty is not very big compared, e.g. to the results of the paper by Liang et al.


Generalization Bounds for Uniformly Stable Algorithms
Uniform stability of a learning algorithm is a classical notion of algorithmic stability introduced to derive high-probability bounds on the generalization error (Bousquet and Elisseeff, 2002). Specifically, for a loss function with range bounded in [0, 1], the generalization error of a γ-uniformly stable learning algorithm on n samples is known to be within O((γ + 1/n) n log(1/δ)) of the empirical error with probability at least 1 − δ. Unfortunately, this bound does not lead to meaningful generalization bounds in many common settings where γ ≥ 1/ √ n. At the same time the bound is known to be tight only when γ = O(1/n). We substantially improve generalization bounds for uniformly stable algorithms without making any additional assumptions. First, we show that the bound in this setting is O( (γ + 1/n) log(1/δ)) with probability at least 1 − δ. In addition, we prove a tight bound of O(γ 2 + 1/n) on the second moment of the estimation error. The best previous bound on the second moment is O(γ + 1/n). Our proofs are based on new analysis techniques and our results imply substantially stronger generalization guarantees for several well-studied algorithms.

This paper develops new generalization bounds for algorithms that are uniformly stable. Specifically, the main results are an improved high probability bound on generalization error and an improved bound on the second moment on the generalization error. Both bounds to appear to be real improvements over past results and advance what is known about the generalization error of uniformly stable algorithms. The key tools used to prove the new results are from recent developments in differential privacy and adaptive data analysis. While many of the core techniques were established in previous works, the authors do go further to combine these techniques with a more learning-specific analysis to get their results, and I believe their supporting lemmas and proofs of their theorems represent non-trivial contributions.  An unfortunate weakness of this paper is that it provides only a blurry coverage of applications near the end of the Introduction, but all new results related to applications are hidden away in the appendix. New results do not belong in the appendix, and to expect reviewers to take them into account is to implicitly try to overcome the page limit. Therefore, with regards to applications, my review only considers whatever I could learn from the very high-level Applications section on page 4. I recommend the authors remove Appendix A from the paper, and that they try to develop at least one of the applications properly in the main text.   DETAILED REVIEW   In comparing the results of Theorem 1.2 to what was previously known from equations (2) and (3), I found this paper's results to be significant improvements. These results are foundational and I expect can be further built upon to get new results in specific (still theoretical) applications. On the whole, the paper is clear, but I recommend rewriting the Applications subsection as it was hard to get a clear view into the authors' results here as so much is packed in within a small space. A good tradeoff would be to move some of the proofs appearing in the main text into the appendix, allowing more space in the main text for formal statements of (some of the) Applications-related results. This will no doubt improve the impact of the paper as more people will see what the Theorem 1.2 can buy us in ``practice''. Also, the citations for Theorem 3.3 were not very clear. Some more guidance as to what parts of these papers have the stated result would help (I found them finally but after a lot of extra effort on my part).   Regarding the technical quality of the authors' results: The new technical results of the authors are Theorem 1.2 (consisting of inequalities (5) and (6)) and its supporting lemmas: Lemma 3.4, 3.5 ,and 4.1, and 4.2. I looked through the proofs of all these results. I believe the proofs of the lemmas and also inequaity (5) of Theorem 1.2 are all correct. The proof of inequality (6) of Theorem 1.2 also is generally fine, modulo some small typos / missing details that I ask the authors to include. Specifically:    1. On the 4th line of page 7, when you say ``By Lemma 3.5 we have that'' I think you should also mention that you are using the part of Theorem 3.3 that says that the exponential mechanism in this case is differentially private, as this is needed to apply Lemma 3.5.    2. When you say ``To bound this expression we choose $\epsilon =$ ...'', the second equality is only an inequality (it should be $\leq$), and only if you set $\epsilon$ to the RHS (the term with the $\ln(e \ln(2)/\delta)$) does the rest of the proof work. I suggest making this fix so that the proof is correct. In the second to last math display (in Proof of eq. (6) in Theorem 1.2), I wasn't able to see immediately how the inequality follows (even after using $\gamma \leq \sqrt{\gamma}$). Saying a little extra here could help.    3. I was able to myself construct the argument for why you need only consider the regime of $(e^\epsilon - 1) \leq 2 \epsilon$, but as stated, your explanation of ``Our bound is at least $2 \epsilon$ and hence...'' is not very clear and should be rephrased/expanded.   Last, while many of the core original tools were developed in previous works, I believe that some amount of ingenuity was needed on the parts of the authors to apply these tools in the right way to get their new results for uniformly stable algorithms. As these techniques are still relatively new to the learning theory community, I believe mixing them into this space provides significant value.     Minor corrections/points:  In Definition 2.2, I believe $S^{(i \leftarrow z_i)}$ should be replaced by $S^{(i \leftarrow z)}$  There are suspicious issues with missing line numbers thorughout the paper, but especially on page 7.     EXPERTISE OF REVIEWER  I am familiar with the fundamental developments related to algorithmic stability (and more generally, learning theory), although I am less aware of new contributions within the last year or so. Likewise, I am familiar with many of the techniques used in differential privacy and adaptive data analysis.   UPDATE AFTER READING THE REBUTTAL  You should reconsider leaving the current Applications section as is. I think it really is not useful and will hurt the impact of your paper. It is sad that you would leave a discussion of core selling points of your paper to the appendix. While a learning theorist may get excited, a large number of other readers will not. Also, things that "follow easily" to the authors may not follow so easily to a reader that is not an author of the work; please consider the perspective the reader.


Efficient Formal Safety Analysis of Neural Networks
Neural networks are increasingly deployed in real-world safety-critical domains such as autonomous driving, aircraft collision avoidance, and malware detection. However, these networks have been shown to often mispredict on inputs with minor adversarial or even accidental perturbations. Consequences of such errors can be disastrous and even potentially fatal as shown by the recent Tesla autopilot crashes. Thus, there is an urgent need for formal analysis systems that can rigorously check neural networks for violations of different safety properties such as robustness against adversarial perturbations within a certain L-norm of a given image. An effective safety analysis system for a neural network must be able to either ensure that a safety property is satisfied by the network or find a counterexample, i.e., an input for which the network will violate the property. Unfortunately, most existing techniques for performing such analysis struggle to scale beyond very small networks and the ones that can scale to larger networks suffer from high false positives and cannot produce concrete counterexamples in case of a property violation. In this paper, we present a new efficient approach for rigorously checking different safety properties of neural networks that significantly outperforms existing approaches by multiple orders of magnitude. Our approach can check different safety properties and find concrete counterexamples for networks that are 10× larger than the ones supported by existing analysis techniques. We believe that our approach to estimating tight output bounds of a network for a given input range can also help improve the explainability of neural networks and guide the training process of more robust neural networks.

The paper improves on previous methods for formal verification of neural network properties over small intervals, such as formally verifying resistance to Linf adversarial examples in a ball around one point at a time.  The paper is a significant (~1 OOM) improvement relative to previous work on this problem.  I am somewhat unconvinced of the importance of this type of verification given that (1) these techniques provide absolute confidence about a tiny sliver of the possible attack space, (2) as-is they are many orders of magnitude too expensive to use at training time, and therefore not much of an improvement given that most networks do not satisfy even the safety properties these techniques can check, and (3) have little to no hope of scaling to interestingly sized networks such as ImageNet models.  However, since this paper is an order of magnitude speed improvement relative to previous work, I am marginally in favor of setting those concerns aside and accepting it on speed merits.  The paper proceeds by maintaining two types of bounds on every value in a relu network: a concrete interval bound and a linear interval bound whose lower and upper components are linear functions in terms of other values in the network.  The main novelty relative to previous work is in transforming these linear interval bounds through relu activation functions by bracketing the relu function between two linear functions over the interval given by the concrete bounds.  The details of the paper appear novel relative to previous work and are relatively simple (a plus) given the OOM speedup.  High level comments:  1. I would like at least a brief discussion of the limits of Linf balls as a threat model.  Even restricting to human-imperceptible adversarial examples, Linf balls make up only a tiny sliver of possible image edits, and it seems unlikely that these techniques would extend to all of them.  I am worried about overfitting this type of verification research to a part of the space that attackers can easily ignore. 2. Is there any hope of scaling these techniques to extremely large neural networks, such as those with more than 10k neurons? 3. I find the notation in the paper confusing, and this bleeds over into lack of clarity in the algorithm.  In particular, the paper never explicitly states what type of mathematical object the Eq's are.  Are they linear functions over the input?  Linear functions over other nodes?  I believe I can infer from the rest of the paper that it's the latter, but this should be stated.  This confusion is a consequence of taking the notation from [28] and applying it without change, which doesn't work: [28] uses a purely bottom up computation but this paper solves for constraint consequences recursively using an LP solver.  Relatedly, in many places in the paper the language seems to imply that [l,u] are computed from [Eq\_low, Eq\_up], which can't be true if the Eq's are just linear equations that do not know about the concrete bounds of other nodes. 4. The paper does not clearly specify how [l,u] are computed and refined.  Before DCR all the [l,u]'s and Eq's are computed in a bottom up pass using interval arithmetic.  Once we have a bunch of Eq < 0 or Eq > 0 constraints, are the [l,u] intervals simply recomputed bottom up and intersected with [-inf,0] or [0,inf] intervals at the constrained nodes?  I f so, this should be stated clearly.  The reason I am unclear is that it seems possible to derive stronger intervals b ased on the result of the linear programming solves (though I am not sure quite how to do this).  If so those stronger intervals could be fed back into SLR to get better linear bounds.  Low level comments: 0. What does SPEC mean?  It's capitalized like an acryonym, but no definition is given.  Is it just a strangely capitalized version of "specification"?  Also, SPEC is a bad name for speed-related method like this given https://www.spec.org. 1. "Piece-wise" should be "piecewise". 2. There are a few spelling and grammar errors such as "Sepcifically", "we define safety property as that the confidence value of outputting targeted class", "The refinement runs iteratively by calling underlying efficient linear solver", etc.  A proof reading pass is in order. 3. Line 213 is confusing, since it seems to imply that multiple overestimated nodes are split at once    which I believe is false.  I think this is likely just a plural vs. singular typo. 4. The inductive proof of property 1.2 in the supplementary material is needlessly complex, since it starts the induction at n = 1 (the first hidden layer) rather than n = 0 (the input).  As a result, it gives essentially the same argument twice. 5. The proof of lemma 1.2 in the supplementary material can be replaced with a single line.  It is obvious that once all the nodes are constrained to be linear that the result is perfectly approximable with linear functions. 6. The optimality of (1) on line 165 has a simpler, more intuitive proof than that given in supplementary     material.  For the upper bound, since relu is convex up the best approximating line touches the relu curve    at both interval ends; this immediately gives the answer.  Next, the optimal upper and lower bounds must have    the same slope formula, since translating an upper bound down by its maximum error gives a lower bound with    at most that same error and vice versa.   7. Although choosing linear interval bounds based on minimax error is simple, it's not obvious that it is the    most efficient heuristic (for lower bounds, since the optimal upper bound is simultaneously optimal over the    whole range by convexity).  For example, if we somehow knew that for most of the subdivision tree a given    relu node would be positive, we could choose a lower bound with higher slope and thus lower error in the positive    part.  There's no need to mention this observation if it isn't useful.


Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis
We describe a neural network-based system for text-to-speech (TTS) synthesis that is able to generate speech audio in the voice of different speakers, including those unseen during training. Our system consists of three independently trained components: (1) a speaker encoder network, trained on a speaker verification task using an independent dataset of noisy speech without transcripts from thousands of speakers, to generate a fixed-dimensional embedding vector from only seconds of reference speech from a target speaker; (2) a sequence-to-sequence synthesis network based on Tacotron 2 that generates a mel spectrogram from text, conditioned on the speaker embedding; (3) an auto-regressive WaveNet-based vocoder network that converts the mel spectrogram into time domain waveform samples. We demonstrate that the proposed model is able to transfer the knowledge of speaker variability learned by the discriminatively-trained speaker encoder to the multispeaker TTS task, and is able to synthesize natural speech from speakers unseen during training. We quantify the importance of training the speaker encoder on a large and diverse speaker set in order to obtain the best generalization performance. Finally, we show that randomly sampled speaker embeddings can be used to synthesize speech in the voice of novel speakers dissimilar from those used in training, indicating that the model has learned a high quality speaker representation.

This paper discusses a voice cloning TTS system that is based on speaker embeddings as opposed to speaker adaptation. The speaker embeddings are found using a speaker encoder. The key idea of this paper is that the speaker encoder is trained separately on a speaker verification task instead of training it together with the TTS system. This reduces the amount of speech-transcript pairs required to train the complete system. The performance of the proposed system is extensively analysed and compared to the performance of a similar system using fixed embeddings for seen speakers, based on the speaker ID.  It is shown that the proposed system achieves similar performance for seen speakers, but is also able to generalise to unseen speakers, which is not possible for the fixed embedding system.  The quality of the paper is pretty good, but is lacking key baseline systems. The experiments section is extensive. Both quantitative and qualitative results are presented and several evaluation schemes are used. What I feel is lacking are additional baseline systems. The paper should definitely include a baseline where the speaker encoder is not trained on a speaker verification task, but is trained jointly, like in [1] or [8]. Otherwise we cannot assess the value of training the speaker encoder separately. If such a baseline is added we can also compare the results for unseen speakers. Adding a speaker adaptation system as a baseline could also strengthen the paper, but is not as crucial since a comparison has already been done in [1].  The author has made some good points in their response about the experimentation, so I would say that the argumentation is sufficient.   The clarity of the paper is good. The core concepts of the paper are explained well and references are extensively cited as comparison with related work. The components of the system are not explained in full detail but references are provided to papers that do explain the details. If the components and hyper parameters of the proposed model are the same as in the reference material the system should be reproducible.   I would say the originality and significance of the paper are mediocre. There is not a lot of novelty in the paper. Pre-training the speaker encoder has been done in [5], but a different type of speaker encoder is used. All the other components in the system are not novel. I think putting the components together as proposed in the paper makes sense, so in my opinion it is an interesting system for this task. I would say that it is likely that this system will be used for voice cloning TTS systems in the future.


Generalizing to Unseen Domains via Adversarial Data Augmentation
We are concerned with learning models that generalize well to different unseen domains. We consider a worst-case formulation over data distributions that are near the source domain in the feature space. Only using training data from a single source distribution, we propose an iterative procedure that augments the dataset with examples from a fictitious target domain that is "hard" under the current model. We show that our iterative scheme is an adaptive data augmentation method where we append adversarial examples at each iteration. For softmax losses, we show that our method is a data-dependent regularization scheme that behaves differently from classical regularizers that regularize towards zero (e.g., ridge or lasso). On digit recognition and semantic segmentation tasks, our method learns models improve performance across a range of a priori unknown target domains.

The paper attacks a novel problem: one-shot domain generalization where given samples from a single domain one requires robustness to unknown domain covariate shift. The problem is extremely hard and the related works claim that no other work exists attacking the same problem.  The paper incrementally builds up on a recently proposed method to defend against adversarial attacks [33]. In fact, the work uses the formulation of [33] and repurposes the procedure for one-shot domain generalization. The original additions are 3-fold: 1) the closeness constraint is changed from pixel-space to feature-space 2) an ensemble of models are trained with different neighborhood thresholds 3) a new theoretical motivation is provided.  It tests the method on the transfer between digit datasets as well as road datasets. The results show that the proposed training work better when tested on unseen domains compared to standard training.  The main issue is that, since no prior works exist to compare against, it needs more comprehensive experimental design to show the real benefits of the method. The experimental setup is currently very limited. For instance, I would like to see how are the benefits of using the new method related to the capacity of the model and how regularized it is (by design and by choice of hyperparameters). Would we see the same benefits if the model is highly regularized already?  What is the benefit of the constraint in the feature space instead of pixel space? A transition between using different layers would probably give some insight.  Some analysis on the samples which are only correctly classified using the proposed method will be helpful. At the same time, an analysis of the samples that are added to the dataset during the training procedure can be insightful. These two sets can be further studied together.  All in all, the originality of the paper is lacking, the experimental setup is not convincing, and there are not much insights given by the paper into the novel problem. So, I think, the paper is not ready for publication at NIPS.  ------------------------------------------ I read the authors' rebuttal and other reviewers comments. The rebuttal partially mitigates my general concerns with the paper: 1) regularization and 2) general insights. On the other hand, I can see the other reviewers' main positive point that from the theoretical point of view it is a simple and clear work on top of the existing ones. So, I increase my rating slightly but still lean towards the rejection of the paper. I think the paper will become significantly stronger if along the lines of the rebuttal further and more comprehensive studies are pursued.


Coupled Variational Bayes via Optimization Embedding
Variational inference plays a vital role in learning graphical models, especially on large-scale datasets. Much of its success depends on a proper choice of auxiliary distribution class for posterior approximation. However, how to pursue an auxiliary distribution class that achieves both good approximation ability and computation efficiency remains a core challenge. In this paper, we proposed coupled variational Bayes which exploits the primal-dual view of the ELBO with the variational distribution class generated by an optimization procedure, which is termed optimization embedding. This flexible function class couples the variational distribution with the original parameters in the graphical models, allowing end-to-end learning of the graphical models by back-propagation through the variational distribution. Theoretically, we establish an interesting connection to gradient flow and demonstrate the extreme flexibility of this implicit distribution family in the limit sense. Empirically, we demonstrate the effectiveness of the proposed method on multiple graphical models with either continuous or discrete latent variables comparing to state-of-the-art methods.

The paper proposes a new distribution class that can be used as an auxiliary distribution class for variational inference. This new class is called 'Optimization embedding' and is derived based on two steps: first a primal-dual rewriting of ELBO, and second, and optimization procedure based on the minor descent algorithm. a Particular instance of this distribution class is described that can be trained using gradient-decent techniques. The model is evaluated among three axis: the generative ability of the approach, the flexibility of the resulting approximation posterior distribution and sample efficiency. These experiments are made on classical datasets (including toy datasets, MNIST and CelebA) and show that the proposed class has good properties.   First of all, the model presented in the paper is interesting and is a good contribution to the field.  The paper is very technical and very difficult to understand for a non-expert. It needs a strong background on different methods (e.g normalizing flows, Fenchel-duality and interchangeability principle, mirror descent algorithm, etc...) An effort could be put on briefly describing these different concepts to allow the reader to better understand the big picture of the paper. But the paper is quite well written, and the steps to obtain the final distribution class are clear enough. To help the reader, I would advise the authors to include some parts of the supplementary material directly in the paper, particularly Algorithm 2 and Section B that provide a concrete example of the use of the presented principles. The first paragraph of Section 3 would also benefit from presenting the intuitive ideas behind the obtained distribution class, and particularly the role of the embeddings in the resulting algorithm. Theorem 1 seems to have some typos problems (R^p instead of R^r), and the role of v^* is not clear in this theorem. Section 3.2 is particularly difficult to understand and would benefit from a small description of the MDA principles. The experimental results are interesting and demonstrate the power of the contribution. Of course, a quantitative study on the quality of the generated samples could help to validate the generative ability of the model (using mechanical turk for example). Moreover, the convergence speed is evaluated in term of epochs, while it would be more interesting to evaluate it in term of time since the complexity of each epoch is not clearly stated in the paper, so it is not clear if the proposed approach is really faster than baselines.    Summary: interesting contribution, but a paper lacking a clear presentation of the main ideas, making it difficult to read for non-experts, and thus restricted to a small audience.


Neural Guided Constraint Logic Programming for Program Synthesis
Synthesizing programs using example input/outputs is a classic problem in artificial intelligence. We present a method for solving Programming By Example (PBE) problems by using a neural model to guide the search of a constraint logic programming system called miniKanren. Crucially, the neural model uses miniKanren's internal representation as input; miniKanren represents a PBE problem as recursive constraints imposed by the provided examples. We explore Recurrent Neural Network and Graph Neural Network models. We contribute a modified miniKanren, drivable by an external agent, available at https://github.com/xuexue/neuralkanren. We show that our neural-guided approach using constraints can synthesize programs faster in many cases, and importantly, can generalize to larger problems.

The paper presents an approach to neural guided search for program synthesis of LISP programs.  A modified miniKanren constraint solver is used to synthesize a program from example IO pairs by "inverting" the EVAL function.  The search for candidate programs in the solver is guided by a neural network (either a GNN or an RNN with other layers on top).  The approach is compared to baselines and three symbolic methods.   Pros: - The paper is very well written and definitely relevant to NIPS - The evaluation against symbolic methods is reasonable - While the idea of scoring evaluation expansions is not novel, applying it to the EVAL function is   Cons: - Novelty is somewhat limited - Links to some recent related work are missing - No comparison to other neural methods   In general, I think that paper is clear and sound enough.  I am not so confortable about originality, but still, I am in favor of acceptance.   Major issues:  - The fact that learners can be used to score candidate rule expansions has already been demonstrated (several times) before, see for instance:      Learning to Search in Branch-and-Bound Algorithms, 2014, He, Daume' III, and Eisner (more distantly related; but note that, unsurprisingly, program synthesis and structure learning in general *can* be framed as a MILP problem)      End-to-End Differentiable Proving, 2017, Rocktäschel and Riedel (in a very related setting)    Essentially this paper "ports" the neural guidance idea from SLD resolution to constraint satisfaction, which in miniKanren is based on the same principle.  The handling of logical operators as aggregators is also similar.    As far as I can see, the novely here is in the choice of "what to guide", namely the expansion of the EVAL function, and in the choice of neural architecture.  This is less novel than the authors seem to claim.    This is good enough, but the authors should definitely better position their contribution w.r.t. the aforementioned works, especially the second one.   - The fact that the proposed system is "deeply integrated" with miniKanren and LISP prevents comparison to existing neural approaches.  This is very unfortunate---but unavoidable, I guess.    More generally, it is difficult to see what the advantages of the proposed approach are with respect to existing neural synthesis alternatives.  Unfortunately there is no ready-made, agreed-upon benchmark or problem setting for neural program synthesis, and I understand that it is difficult to compare against other neural techniques.  However, this is only a partial excuse.    Note that, for instance, since in principle prolog can be used to synthesize programs satisfying a set of examples, the method in the "end-to-end differentiable proving" paper can be used for program synthesis too.  I won't ask the authors to compare against it, as it is a lot of work, but this should be doable.  - The pros and cons of RNN versus GNN are not really studied in depth.  I guess that this is unavoidable, given that these models are quite opaque.   Minor issues:  - missing commas in Figure 2


Point process latent variable models of larval zebrafish behavior
A fundamental goal of systems neuroscience is to understand how neural activity gives rise to natural behavior. In order to achieve this goal, we must first build comprehensive models that offer quantitative descriptions of behavior. We develop a new class of probabilistic models to tackle this challenge in the study of larval zebrafish, an important model organism for neuroscience. Larval zebrafish locomote via sequences of punctate swim bouts-brief flicks of the tail-which are naturally modeled as a marked point process. However, these sequences of swim bouts belie a set of discrete and continuous internal states, latent variables that are not captured by standard point process models. We incorporate these variables as latent marks of a point process and explore various models for their dynamics. To infer the latent variables and fit the parameters of this model, we develop an amortized variational inference algorithm that targets the collapsed posterior distribution, analytically marginalizing out the discrete latent variables. With a dataset of over 120,000 swim bouts, we show that our models reveal interpretable discrete classes of swim bouts and continuous internal states like hunger that modulate their dynamics. These models are a major step toward understanding the natural behavioral program of the larval zebrafish and, ultimately, its neural underpinnings.

The authors propose a marked process latent variable model that leverages Gaussian processes for continuous latent sates, a generalized linear model for discrete latent states and an inference network for efficient inference. Results on real data suggest that the proposed approach is interpretable and outperforms standard baselines on held out data.  I really enjoyed reading the paper, it is very well written. That being said, the notation is sloppy and lack of details makes it difficult to appreciate the contributions of the paper. The authors seemed to have assumed that the reader is very familiar with point processes, Gaussian processes, variational inference and deep learning. To name a few, no distinction between the covariance function K() and the covariance matrix K is made; the forms of the parameter functions for Gamma and Gaussian distributions in the generative process below line 156 are never introduced or defined (closed-form conditional posteriors are available or are they specified as neural networks?); the form of the generalized linear model \pi_\theta() is never made explicit; no details of the RNN are provided; it is not clear whether the inducing points are estimated, set on a grid or why 50 or 25 inducing points are enough; in Table 2, no details about Poisson, GRP and AR apre provided. How were the many parameters of all models selected?  In a paper like this with so many details missing, a supplementary file with formulation, inference, model architecture details and experimental setup details is a must, however, none was provided. Besides, lack of space seems to be not a problem provided that Figure 2 is not terribly informative, and Figures 3-5 could be made a lot smaller without sacrificing readability.  Minor: - eq. (1), N_m not defined, typo? - line 137, GP mean and covariance functions not defined. - line 156, undefined quantities, \mu_\theta(), \Sigma_\theta(), a_\theta(), b_\theta(), m_n|1:n-1, K_n|1:n-1 - line 252, locomotion - line 259, remove finds or reveals  In their rebuttal, the authors have provided enough detail about the approach and experiments. I am happy to recommend acceptance (score changed from 3 to 7).


Binary Rating Estimation with Graph Side Information
Rich experimental evidences show that one can better estimate users' unknown ratings with the aid of graph side information such as social graphs. However, the gain is not theoretically quantified. In this work, we study the binary rating estimation problem to understand the fundamental value of graph side information. Considering a simple correlation model between a rating matrix and a graph, we characterize the sharp threshold on the number of observed entries required to recover the rating matrix (called the optimal sample complexity) as a function of the quality of graph side information (to be detailed). To the best of our knowledge, we are the first to reveal how much the graph side information reduces sample complexity. Further, we propose a computationally efficient algorithm that achieves the limit. Our experimental results demonstrate that the algorithm performs well even with real-world graphs.

Summary of paper: The paper considers the problem of completing a binary rating matrix, and theoretically studies whether side information in terms of graphs among users or items aid in the matrix completion task. On a technical level, the study assumes that the users belong to two communities, and users in the same group rate all items identically (upto some noise). The community structure is further revealed by the side information in terms of a graph generated from stochastic block model. The paper derives the optimal sample complexity for rating estimation in terms of the expected number of ratings that needs to be seen to exactly recover the rating matrix.  Review summary: The paper is a solid piece of theoretical work and clearly written. The problem tackled is quite important, and the paper clearly shows how much graph side information helps under the assumed setting. It is known that side information helps but a clear quantification of its benefits is not there in existing literature.  On the negative side, it seems that the model and the method are too restrictive for the general rating estimation problem. Compared to general methods like Rao et al [42], this work seems too fine-tuned for the theoretical problem (with clustered users) and has limited direct impact on practice.  Originality: The theory and algorithm relies heavily on the recent literature on block models. The use of this theory in rating estimation is somewhat novel, although some recent works have made similar connections in related problems (for example, [R1]).  Significance and quality: The main positive for the paper is its significance. Quantifying the usefulness of side information in any learning problem is a difficult task. The paper achieves this goal by presenting and analysing a formal problem, which shows that the optimal sample complexity is indeed lower if side information is used.  The theory looks sound, but I have not read the proofs in the supplementary. Apart from the correctness, I have a fundamental question: - it seems that graph side information only aids when the quantity I_s is of the order log(n)/n. Doesn’t this imply that side information does not provide significant advantages? That is, we cannot drastically reduce the sample complexity by using side graphs (unless parameters are fine-tuned as in Example 2).  The practicality of the work is rather limited. The model assumes that users are clustered into two groups, the graph shows exactly these two groups, and the spectral algorithm mainly tries to find the groups. Hence, in comparison to general methods like Rao et al [42], this work seems too fine-tuned to the theoretical problem (with clustered users).  - can methods like [42] that simply use the graph as a regulariser achieve near optimality in this setting?  I like the fact that the proposed spectral method adaptively estimates some parameters, though I am not sure how c1, c2 can be estimated. However, as noted above, the methods loses generality as it mainly uses the graph for its cluster structure and not as some sort of regulariser (side graphs without community structure don't contribute).  The numerical section is bit disappointing, which is fine for a theory paper. The networks are real, but the ratings are synthetically generated based on the ground truth communities. - can we use the proposed method for any real binary estimation problem? - the numerical comparisons are perhaps bit unfair since the study is on a setup where finding the cluster structure from graph is supposed to reveal correct ratings, and hence, proposed method outperforms    Figure 2 is supposed to show that there is a clear phase transition (white to black) at the derived theoretical sample complexity (orange line). Given how convincing empirical phase transition plots can be for some problems, I am not sure if Figure 2 validates the sudden transition proved in Theorem 1. Please clarify.  Clarity: The paper is generally well-written. Since this is a theory paper in a general ML conference, it would be great if the authors can add some discussions on the practical consequences of the results. For example: - I_s of the order log(n)/n is only regime when side information helps. Why doesn’t it help to have more dense disjoint graphs? This is practically counter-intuitive. - when does it not matter to have side information? Why? - what is the implication of exact recovery in practice?  Minor comments: - line 67: “out-of-reach” is too strong, perhaps “not studied” - line 205: “cluster size” should be “number of clusters" - line 220: should w be \omega? - line 223: footnote mark 3 looks confusing after the (log n).  [R1] Lee and Shah. Reducing crowdsourcing to graphon estimation, statistically. AISTATS 2018  UPDATE AFTER AUTHOR REBUTTAL: I thank the authors for their response. It will be great if they can add the mentioned discussions to the paper. On a general note, I would also request the authors to include some concluding remarks about the limitation of the study to clustered user scenario, and if (or how) one can extend these results to more general graph side information.