Algebraic tests of general Gaussian latent tree models
We consider general Gaussian latent tree models in which the observed variables are not restricted to be leaves of the tree. Extending related recent work, we give a full semi-algebraic description of the set of covariance matrices of any such model. In other words, we find polynomial constraints that characterize when a matrix is the covariance matrix of a distribution in a given latent tree model. However, leveraging these constraints to test a given such model is often complicated by the number of constraints being large and by singularities of individual polynomials, which may invalidate standard approximations to relevant probability distributions. Illustrating with the star tree, we propose a new testing methodology that circumvents singularity issues by trading off some statistical estimation efficiency and handles cases with many constraints through recent advances on Gaussian approximation for maxima of sums of high-dimensional random vectors. Our test avoids the need to maximize the possibly multimodal likelihood function of such models and is applicable to models with larger number of variables. These points are illustrated in numerical experiments.

# After authors response  I thank the authors for their detailed answers and explanations, that corrected some of the misconceptions I had.  I acknowledge that the application of the methodology of section 3 to the general framework of section 2 would require some further developments, that might be outside of the scope of this conference paper. Hence, the problem of the discrepancy between the two sections remains for this submission, although both sections are interesting in themselves.  # Summary  This well written and nicely presented work is mainly interested in the algebraic characterization and statistical test of a latent Gaussian tree structure. This problem has received some attention over the last few years, and can be related to the work of Shiers et al. 2016 (or Zwiernik et al. 2016 although with a different focus).  Section 2 focuses on the mathematical derivation of polynomial constraints that characterize variance matrices induced by Gaussian latent tree models. It follows closely the derivations of Shiers et al. 2016, extending them to the case where observations can be anywhere on the tree, including on inner nodes, and not just its leaf nodes. The results are presented in an intuitive way, and the proofs are given in the appendices.  In Section 3, a test statistic is derived, in the simplified case of a star-tree. A link is drawn between this star-tree case and the tetrads problem, as studied in the factor analysis literature. The derivation of the new statistics and its approximate distribution relies on recent work by Chernozhukov et al. 2013 and Zhang and Wu 2017. The statistic is justified by heuristic argument, but the proofs are not given.  In Section 4, some numerical experiments are conducted to study the empirical levels of the proposed test. It is compared with a standard maximum-likelihood test as traditionally used in factor analysis (function `factanal` in base R).  # Main comments  My main concern is about the coherence and unity of the ideas presented. In Section 2, the authors describe with much mathematical details a new framework allowing observations to be at inner nodes of a tree. However, in section 3 and following, only the case of a star tree is considered, with observations only at leaf nodes. Maybe I missed something and I would be happy to be corrected, but from my understanding all the developments in Section 3 could have been made with results already exposed in Shier et al. 2016, their main inspiration for Section 2. Hence, as it stands, section 2 seems somewhat decorrelated from Section 3, and, although the results presented in Section 2 are theoretically interesting in themselves, it is not clear what they bring in the progression of the article.  As a minor note related to the same point, only one reference is given to justify the need of the theoretical developments of section 2 (l.34). Adding more examples could help the reader understanding the pertinence of extending existing results to allow for inner-node measurements.  That being said, I tend to think that both parts of the paper (section 2 and sections 3-4) are valuable advances to the field, and have the potential to be the starting point for further developments. I found the results of section 2 and very well presented. The background, as well as the novelty of the results, are both clearly exposed. Section 3 makes use of novel statistical techniques to build a new test (heuristically justified) that might have the potential to be used, beyond the star tree, to the more general structures presented in section 2.  # Other issues  ## Section 2  While I spent a reasonable amount of time to try to understand the results in themselves, I did not review the proofs in details. I found that the new notions and differences with previous characterisations where well explained, and the intuitions were easy to follow.  Other comments: * Why are the observations limited to nodes with degree less or equal than 2 ?   Is this a constrain inherited from applications, or would the developments of   the paper be wrong otherwise ? From Fig. 2-e, it does not prevent sub-trees   to have observed nodes of degree 3. * I did not see why a pseudo-distance is needed, while a distance seems to be   used in Shiers et al 2016 ? * Cor. 2.2 is linked with Th. 3 in Shiers et al, the same way Lemma 2.1 is   linked with Cor 1. Although the links and differences between Lemma 2.1 and   Cor 1 are clearly mentioned, the relationships between Cor 2.2 and Th 3   remain allusive. An intuitive description of these differences might help the   reader understand the new constraints implied by the new problem. * Fig 2.1 and 2.2 are very helpful. Could it be useful to add as legends    {p, q, r, s} such that {p, q}|{r, s} on Fig 2.2 a-c ?  ## Sections 3 - 4    While section 2 seems theoretically sound, the results in section 3 are only justified heuristically, and heavily rely on cited references. Since I'm not very familiar with these techniques, I only have a few remarks about the form:  * l.141-142: should it be \epsilon_1, …, \epsilon_m and X_1, …, X_m (m instead   of p) ? * The authors compare their test statistics with the one in Shier et al 2016   at the end of Section 3, but then do not include this method in the numerical   experiments of Section 4. Are there arguments, other than computational time   (which could be a legitimate concern) to exclude it ? * Fig. 4.1: Maybe a legend on the rows/columns of the figure could help.   Also, since the interval (0, 0.1) is particularly significant, maybe a visual   aid, such as a vertical line, could help. * l.255-256: the statement "The sizes of our test based on T also align   better with 45° line as the sample sizes grow." is not clear to me in the   context of Setup 1 (which is discussed in this paragraph). In particular, it   does not seem to be the case in the interval of interest (0, 0.1). * Could a quantitative score, such as the area between the empirical curves   and the 45° line, be useful to make more objective statements about the   performances of the methods ? * It might be interesting to get an idea of the computational times involved   by the application of these tests.  ## Reproducibility  The test statistics as well as the numerical experiments are well described, so that it should be possible for a researcher willing to reproduce these analyses to write the necessary code. However, no package, or R script, is provided to make this exercise easier.


Navigating with Graph Representations for Fast and Scalable Decoding of Neural Language Models
Neural language models (NLMs) have recently gained a renewed interest by achieving state-of-the-art performance across many natural language processing (NLP) tasks. However, NLMs are very computationally demanding largely due to the computational cost of the decoding process, which consists of a softmax layer over a large vocabulary. We observe that in the decoding of many NLP tasks, only the probabilities of the top-K hypotheses need to be calculated preciously and K is often much smaller than the vocabulary size. This paper proposes a novel softmax layer approximation algorithm, called Fast Graph Decoder (FGD), which quickly identifies, for a given context, a set of K words that are most likely to occur according to a NLM. We demonstrate that FGD reduces the decoding time by an order of magnitude while attaining close to the full softmax baseline accuracy on neural machine translation and language modeling tasks. We also prove the theoretical guarantee on the softmax approximation quality.

Update after author response:  Thanks for the detailed response! It's a strong submission and I vote for an accept.  ===========  This paper aims to speed up the computation of the softmax over a large vocabulary, which is quite common in some NLP tasks like e.g., language modeling. Specifically, the proposed method formulates the problem into a nearest neighbor search in a small world graph, and applies a log time algorithm to find the approximate top K predictions. The resulting time complexity reduces to logarithmic in the vocabulary size in expectation, in contrast to the linear one in a standard softmax. The proposed method is empirically compared to both full softmax and a state-of-the-art baseline on language modeling and neural machine translation. Experimental results show that the method achieves significant speed-up, without sacrificing too much accuracy, and that it scales to larger-size models.  The paper is overall well-motivated and clearly-written, and I find it interesting to read. My major concern is perhaps how the method generalizes to training,  detailed below.  Concerns:  - Small world graph construction: It would be nice to see some discussions on the complexity of the graph construction step, and whether it could become a potential problem if the algorithm is applied to training.  - Probabilities: Are there any good reasons to not report perplexities in the language modeling task? It appears to me that in a FGD, it is less straightforward to assign probabilities to the corpus. One thing on top of my head is to attribute most of the mass to the retrieved top K, with some additional tricks to avoid numerical issues. Have the authors tried this?  - The title is a bit confusing, since ``language models'' usually mean the language modeling task, but here the algorithm applies to many other tasks involving a large softmax.  - There are several statements that look like ``finding top K in sublinear time complexity,'' especially around lines 30--40. I find such statements a bit misleading, since essentially one would need approximations to achieve this.


Neural Architecture Search with Bayesian Optimisation and Optimal Transport
Bayesian Optimisation (BO) refers to a class of methods for global optimisation of a function f which is only accessible via point evaluations. It is typically used in settings where f is expensive to evaluate. A common use case for BO in machine learning is model selection, where it is not possible to analytically model the generalisation performance of a statistical model, and we resort to noisy and expensive training and validation procedures to choose the best model. Conventional BO methods have focused on Euclidean and categorical domains, which, in the context of model selection, only permits tuning scalar hyper-parameters of machine learning algorithms. However, with the surge of interest in deep learning, there is an increasing demand to tune neural network architectures. In this work, we develop NASBOT, a Gaussian process based BO framework for neural architecture search. To accomplish this, we develop a distance metric in the space of neural network architectures which can be computed efficiently via an optimal transport program. This distance might be of independent interest to the deep learning community as it may find applications outside of BO. We demonstrate that NASBOT outperforms other alternatives for architecture search in several cross validation based model selection tasks on multi-layer perceptrons and convolutional neural networks.

The paper describes a new distance metric for neural networks based on optimal transport. Based on this distance, the authors introduce a new Bayesian optimization strategy to optimize the architecture of neural networks.   Overall I think the methods is interesting and that the proposed distance might be also of interest for other applications than Bayesian optimization. However, it seems that computing the distance also requires some manual tuning such as defining the cost matrix which in turn requires expert knowledge. My only point of criticism is that this might hinder its success in automated machine learning applications, where human interaction should be reduced to a minimum.  Having said that, the method is well explained and the authors provide some meaningful empirical insights, such that the paper represents an important contribution for Bayesian optimization.  The following points need some further clarification:  1) How many actual function evaluation did every method perform in Figure 2. i.e how many architectures were trained? In the figure it seems that every method returns something after a discrete time step?  2) What do you mean with at least 5 independent runs for each method in the caption of Figure 2? Why did you perform less runs for some methods?  3) How are pending jobs treated in the parallel set up described in section 5?  4) In table 3, it says CIFAR-10 has 1k features instead of 32*32*3=3072. I guess that this is a typo?    --- Post Rebuttal ---  I thank the authors for taking their time to write the rebuttal and clarifying my points.


Large Margin Deep Networks for Classification
We present a formulation of deep learning that aims at producing a large margin classifier. The notion of margin, minimum distance to a decision boundary, has served as the foundation of several theoretically profound and empirically successful results for both classification and regression tasks. However, most large margin algorithms are applicable only to shallow models with a preset feature representation; and conventional margin methods for neural networks only enforce margin at the output layer. Such methods are therefore not well suited for deep networks. In this work, we propose a novel loss function to impose a margin on any chosen set of layers of a deep network (including input and hidden layers). Our formulation allows choosing any l p norm (p ≥ 1) on the metric measuring the margin. We demonstrate that the decision boundary obtained by our loss has nice properties compared to standard classification loss functions. Specifically, we show improved empirical results on the MNIST, CIFAR-10 and ImageNet datasets on multiple tasks: generalization from small training sets, corrupted labels, and robustness against adversarial perturbations. The resulting loss is general and complementary to existing data augmentation (such as random/adversarial input transform) and regularization techniques such as weight decay, dropout, and batch norm.

This paper proposes a new method for encouraging large margin in the classification problems. In particular, this work is a generalization of SVM to deep models. The idea is interesting, and the method has been tested on many datasets and tasks.   This paper has a few problems.  1. The paper should elaborate the optimization of Eqn 9 and 10, at least in the appendix. The optimization of such problems appears to be nontrivial. The short description in Sec 4.1 is not very easy to follow. Also, what is the time complexity of a forward and backward propagation? Is it much more expensive than optimization the cross-entropy loss?  2. Although I am not sure, I think the proposed large-margin method is very similar to adversarial training. The discussion of the similarity and difference may be interesting.  3. In the experiments, is the large-margin formulation (Eqn 10) applied to every layer or just a few selected layers? Also, Eqn 10 has a parameter $\gamma_l$. If the Eqn 10 is applied to many layers, will the hyper-parameter tuning be expensive?  4. As mentioned by Sec 4.1, the proposed method is expensive if there are many classes. The proposed method does not actually apply to the full ImageNet dataset. I am not sure whether the heuristic of using only one class $i \neq y_k$ is a reasonable approach.  5. Line 160 "... computation of second-order derivatives...": Do the 2nd-order derivatives even exist?


Contextual Pricing for Lipschitz Buyers
We investigate the problem of learning a Lipschitz function from binary feedback. In this problem, a learner is trying to learn a Lipschitz function f : [0,1] d → [0, 1] over the course of T rounds. On round t, an adversary provides the learner with an input x t , the learner submits a guess y t for f (x t ), and learns whether y t > f (x t ) or y t ≤ f (x t ). The learner's goal is to minimize their total loss t (f (x t ), y t ) (for some loss function ). The problem is motivated by contextual dynamic pricing, where a firm must sell a stream of differentiated products to a collection of buyers with non-linear valuations for the items and observes only whether the item was sold or not at the posted price. For the symmetric loss (f (x t ), y t ) = |f (x t ) − y t |, we provide an algorithm for this problem achieving total loss O(log T ) when d = 1 and O(T (d−1)/d ) when d > 1, and show that both bounds are tight (up to a factor of √ log T ). For the pricing loss function (f (x t ), y t ) = f (x t ) − y t 1{y t ≤ f (x t )} we show a regret bound of O(T d/(d+1) ) and show that this bound is tight. We present improved bounds in the special case of a population of linear buyers.

This  paper studies the problem of learning  a  Lipschitz  function  $f(x)$  from  binary  feedback.  Two  types of loss  are  considered,  the  symmetric  ell-1  loss  and  the  unsymmetric  and  discontinuous  pricing  loss.  To  minimize  the  losses,  four  algorithms  are  proposed.      Alg. 1  maintains  a  set  $S  =  {  (X_j,  Y_j)  }$,  where  $X_j$  is  a  partition  interval  of  feasible set  $[0,1]$  and  $Y_j$  contains  the  lower  and  upper  bounds  of  $f(x)$.  At  each  iteration,  an  adversary  queries  a  point  $x_t$.  The  learner  response  with  estimate  $y  =  mid(Y_j)$,  where  $X_j$  is  the  interval  containing  $x_t$.  Base  on  the  feedback,  the  learner  shrinks  $Y_j$  by  intersecting  itself  with  a  set  derived  from  Lipschitz  property  of  $f(x)$.  The  iteration  is  finalized  by  splitting  $X_j$  into  two  new  intervals  when  the  length  of  $Y_j$  can  hardly  be  reduced.  Alg. 2  encodes  the  upper  bound  and  lower  bound  of  $f(x)$  in  sets  $S+$  and  $S-$  respectively.  At each  iteration,  the  learner  receives  a  query  $x_t$,  composing  an  interval  containing  $f(x_t)$  by  solving  a linear  optimization  constrained  on  $S+$  and  $S-$. And it uses  the  mid-point  of  the  interval  as  the  current  guess.  The   $S+$  and  $S-$  are  updated  based  on  the  binary  feedback. Alg. 3  and  Alg. 4  are  extensions  of  Alg. 1  to  the  case  of  multi-dimensional  Lipschitz  functions.  The  author  proves  that  Alg. 3  and  Alg.  4  can  learn  $f(x)$  with  sublinear  regret  for  the  ell-1  loss  and  the  pricing  loss  respectively.  The  two  algorithm  are  optimal  by  giving  the  lower  bound  of  Lipschitz  learning  problem.  The problem is similar to learning a Lipchitz  function  from  binary  feedback  with  ell-1  loss  and  pricing  loss  by  showing  the  optimality  of  their  methods.  The following questions need to be clarified. 1)  All  the  proposed  algorithms  depend  on  the  Lipschitz  constant  $L$.  However,  in  practice,  $L$  may not be satisfied. How  to  adapt  the  proposed  methods  when the condition is not meet ?  2)  Alg.  2  only  supports  the one-dimensional  learning  problem.  How  to  extend  it  to  the  multi-dimension  learning  problem ?  3)  No  empirical  studies  are  performed.  Though  the  author  argues  that  the  model  can  be  used  in  dynamic  pricing  problem,  no  empirical  evidence  validates  the performance.


Overlapping Clustering Models, and One (class) SVM to Bind Them All
People belong to multiple communities, words belong to multiple topics, and books cover multiple genres; overlapping clusters are commonplace. Many existing overlapping clustering methods model each person (or word, or book) as a non-negative weighted combination of "exemplars" who belong solely to one community, with some small noise. Geometrically, each person is a point on a cone whose corners are these exemplars. This basic form encompasses the widely used Mixed Membership Stochastic Blockmodel of networks [1] and its degree-corrected variants [16], as well as topic models such as LDA [9]. We show that a simple one-class SVM yields provably consistent parameter inference for all such models, and scales to large datasets. Experimental results on several simulated and real datasets show our algorithm (called SVM-cone) is both accurate and scalable.

The paper studies the problem of overlapping clustering in relational data under a model that encompasses some other popular models for community detection and topic modeling. In this model, the samples are represented in a latent space  that represents the clustering memberships in which the coordinates are convex combinations of the corners of a cone. The authors propose a method to find these corners using one-class SVM, which is shown to exactly recover the corners in the population matrix, and vectors that are close to the corners in an empirical matrix. The authors establish consistency results for clustering recovery in network and topic models, and evaluate the performance of the method in synthetic and real data, comparing with other state-of-the-art methods.  The problem of overlapping clustering has received increasing attention and is of great interest in different communities. The authors presented a method that can be used in several contexts. The geometric aspects of overlapping clustering models have been studied previously, but the approach followed by the authors is very original and theoretically justified. The method is simple and efficient. I found the method especially relevant for overlapping clustering in network models, as the current approaches in the literature usually require stronger assumptions or make use of combinatorial algorithms. I am not very familiar with the literature in topic modeling, but the method requires to have anchor words on each topic, which seems to be a strong assumption (see Huang et al. "Anchor-Free Correlated Topic Modeling: Identifiability and Algorithm" NIPS 2016). Additionally, there seems to be other work that study the geometrical aspects of topic models, so it would be worth if the authors make more comments in the literature review or numerical comparisons (see for example Yurochkin and Nguyen "Geometric Dirichlet Means algorithm for topic inference", NIPS 2016).  The paper is in general clearly written, and Section 2 is well explained and motivated. However, I found the conditions and results in Section 3 hard to interpret since the authors do not introduce identifiability constraints on the parameters. Maybe this issue can be clarified if the authors are able to obtain closed-forms of the rates for special cases, and compare with other rates in the literature.  The authors make reference to other models which I think should be formally defined (at least in the appendix) for completeness.  UPDATE: Thanks to the authors for clarifying my questions in the rebuttal letter


Parsimonious Bayesian deep networks
Combining Bayesian nonparametrics and a forward model selection strategy, we construct parsimonious Bayesian deep networks (PBDNs) that infer capacityregularized network architectures from the data and require neither cross-validation nor fine-tuning when training the model. One of the two essential components of a PBDN is the development of a special infinite-wide single-hidden-layer neural network, whose number of active hidden units can be inferred from the data. The other one is the construction of a greedy layer-wise learning algorithm that uses a forward model selection criterion to determine when to stop adding another hidden layer. We develop both Gibbs sampling and stochastic gradient descent based maximum a posteriori inference for PBDNs, providing state-of-the-art classification accuracy and interpretable data subtypes near the decision boundaries, while maintaining low computational complexity for out-of-sample prediction.

The paper introduces a new type of (deep) neural network for binary classification. Each layer is in principle infinitely wide but in practice finite number of units is used. The layers are trained sequentially by first training one layer, and then always the next layer after the previous one. The main claim is that the proposed model gives comparable results to the alternative approaches by utilizing fewer hyperplanes that results in faster out-of-sample predictions.   The approach seems somewhat novel and the results support the claim to some extent. However, I found the paper difficult to read in many places (some specific comments below). Furthermore, the accuracy is essentially the same as for the other methods, the computational cost at the training time is not considered and other techniques used to reduce computations at prediction time are not discussed (see below). My overall feeling is that the paper has some potential, but the current exposition of the material requires too much effort from the reader, and hence the paper does not quite meet the standards of NIPS.    Some specific comments/questions:  - I felt the introduction of the iSHM is unnecessarily complicated. The model is actually far more simple than made to appear. It feels that Section 2 could have been both compressed and made clearer simultaneously, as now it contains many not-so-important mathematical details that simply confuse the reader rather than clarify things. The same applies also to Section 3 to some extent.  - All the equations have too little space which makes reading the text and formulae somewhat uneasy.  - I found Fig. 1 extremely hard to follow. Not only are the figures so small that they are barely readable, but the subfigure labeling is also confusing; the outer subfigure names are 1), 2), c) and d) (why not 1-4?), and these are also divided further into 'subsubfigures' named a)-f). It took far too much effort to try to understand the message of these figures. Furthermore, this figure is not even discussed in the main text: "We provide detailed explanations for Fig. 1 in Appendix D." Partly due to these reasons I never got a good intuition about how the model actually works and why it would be better than the traditional ones.  - I would expect the proposed model to be outperformed by convolutional networks (CNN) in image classification. This is not to say that it wouldn't be useful in some other settings, but since the authors claim (in the abstract) that the method achieves "state-of-the-art classification accuracy", I'm simply curious whether the authors agree about this intuition?  - The authors focus on the computational complexity for out-of-sample predictions. What is the computational cost at training time compared to the more traditional networks (e.g. those you used in the comparisons)?  - If the purpose is to reduce the computations at prediction time, there are also other methods specifically designed for this, namely those that attempt to replace the complex architecture by a simpler one, such that the simple model utilizes information from the complex one [e.g. 1, 2]. To my knowledge, these have been quite successful and would be interesting to know how your method compares to these approaches.  - Is it straightforward to extend the proposed model to multiclass problems or regression?   Typos / other minor points:  - 'nueral' page 2, line 63 - abbreviation iSHM introduced twice (page 2, first on line 57 and second time on line 81) - 'refereed' -> 'referred'? page 7, line 252   References:  [1] Cristian Bucilu, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 535–541. ACM, 2006.  [2] Geoffrey Hinton, Oriol Vinyals, Jeff Dean. Distilling the Knowledge in a Neural Network. https://arxiv.org/abs/1503.02531   *** Update *** I thank the authors for submitting a rebuttal which addressed all my questions. After taking some more time to read the paper again I've changed my mind and now think that this could be worthwhile contribution to the conference. I have updated my overall score accordingly. I do, however, encourage the authors to take my initial comments - especially those regarding the readability - into account when preparing the final version of the paper.


Some Considerations on Learning to Explore via Meta-Reinforcement Learning
We interpret meta-reinforcement learning as the problem of learning how to quickly find a good sampling distribution in a new environment. This interpretation leads to the development of two new meta-reinforcement learning algorithms: E-MAML and E-RL 2 . Results are presented on a new environment we call 'Krazy World': a difficult high-dimensional gridworld which is designed to highlight the importance of correctly differentiating through sampling distributions in meta-reinforcement learning. Further results are presented on a set of maze environments. We show E-MAML and E-RL 2 deliver better performance than baseline algorithms on both tasks.
sampling process that maximally informs the meta-learner how it should adapt to new environments. In the context of gradient based algorithms, this means that one principled approach for learning an optimal sampling strategy is to differentiate the meta RL agent's per-task sampling process with respect to the goal of maximizing the reward attained by the agent post-adaptation. To the best of our knowledge, such a scheme is hitherto unexplored.
In this paper, we derive an algorithm for gradient-based meta-learning that explicitly optimizes the pertask sampling distributions during adaptation with respect to the expected future returns produced by it post-adaptation self. This algorithm is closely related to the recently proposed MAML algorithm [7]. For reasons that will become clear later, we call this algorithm E-MAML. Inspired by this algorithm, we develop a less principled extension of RL 2 that we call E-RL 2 . To demonstrate that this method learns more transferable structures for meta adaptation, we propose a new, high-dimensional, and dynamically-changing set of tasks over a domain we call "Krazy-World". "Krazy-World" requires the agent to learn high-level structures, and is much more challenging for state-of-the-art RL algorithms than simple locomotion tasks. 2 .
We show that our algorithms outperform baselines on "Krazy-World". To verify we are not over-fitting to this environment, we also present results on a set of maze environments.

This paper proposes extensions to two popular meta-learning techniques: MAML and RL^2, with the aim of learning an optimal sampling strategy (policy) to maximize expected returns that takes into account the effect of the initial sampling policy on final return. This was more theoretically derived for MAML by explicitly including the dependency in the expectation and differentiating to compute the appropriate gradient, which turns out to have an additional term compared to MAML. This additional term allows the policy to be more exploratory. RL^2 was modified similarly by including multiple rollouts per update, p of which contribute 0 return to the backwards pass (exploratory episodes) and k-p of which contribute the normal return. These models were evaluated on a new environment “Krazy World”, which has dynamically generated features every episode, and also a maze environment.   The authors propose a simple but useful idea, and I think it does make a contribution. Although there are a lot of interesting ideas and results, it feels like a work in progress rather than a complete piece of work. Some technical details are missing, which impact the clarity and quality, as I’ve listed below. This makes me concerned about the reproducibility of their results.  The new environment Krazy World is an interesting benchmark task for meta-learning, and I’d like to see it open-sourced if possible.   More specific comments: -It’s interesting that E-MAML does better initially but E-RL^2 does better in the end. What level of reward is optimal for the two tasks? -For E-RL^2, have you explored tuning the relative contributions of exploration on final return? That is, how is performance impacted by performing more exploratory rollouts per update? How did you settle on 3 and 2? -Are the maze layouts procedurally generated every episode? -There are a lot of typos, please proofread more carefully (e.g. page 7 “desigend” -> “designed”, page 8 “defining an intrinsic motivation signals” -> “signal” and “and the its affect” -> “and its effect”) -In section 2.5, k is both the number of rollouts for each MDP and also the task the MDP represents, so this is confusing -It’s very strange that they don’t cite Duan et al’s RL^2 paper at all, since E-RL^2 is named after it -Figure 8 (in the appendix) is not helpful at all, the axes and legends are impossible to read and axes aren’t even labeled. Why are there 3 columns? The caption only mentions 2. -What do the error shades represent in the plots? Standard deviation? Standard error? Over how many runs?  Update after reading the rebuttal: It seems like all the reviewers had major concerns regarding clarity, and the authors' response does seem to address many of these. There are a lot of things to revise however, and I'm not completely convinced the final product will be ready for publication at NIPS, given how in-progress the initial version felt. Given this, I can't at this time change my assessment, although I still tentatively recommend acceptance and encourage the authors to revise with clarity in mind.


Deep Non-Blind Deconvolution via Generalized Low-Rank Approximation
In this paper, we present a deep convolutional neural network to capture the inherent properties of image degradation, which can handle different kernels and saturated pixels in a unified framework. The proposed neural network is motivated by the low-rank property of pseudo-inverse kernels. Specifically, we first compute a generalized low-rank approximation to a large number of blur kernels, and then use separable filters to initialize the convolutional parameters in the network. Our analysis shows that the estimated decomposed matrices contain the most essential information of an input kernel, which ensures the proposed network to handle various blurs in a unified framework and generate high-quality deblurring results. Experimental results on benchmark datasets with noisy and saturated pixels demonstrate that the proposed deconvolution approach relying on generalized low-rank approximation performs favorably against the state-of-the-arts.

This paper extends the non-blind deconvolution in [1]. The author proposes to use a generalized low-rank approximation method to model a wide range of blur kernels and train the non-blind deconvolution network with the low-rank representation.  Strength: - Unlike the model in [1] that requires a specific training (or fine tuning) for each kernel, the proposed model can handle different blur kernels. - The model can use the low-rank representation results to initialize the model parameters. - Experiments show that the model can achieve good performances on different images and performs better than the compared methods, such as [1].  Weakness: - The most important motivation of this paper is to make the learning-based CNN model being able to handle different kernels. I did not find the reason for using the low-rank approximation-based method. Regarding the extension of the model in [1], I agree the low-rank representation idea is straightforward and clever, but, generally, I am wondering whether building a pre-defined and restricted representation for kernels is necessary. In many optimization-based neural network design, the blur kernels can be directly fed into a learnable neural network. Please check [2,3]. They both can work as a flexible model for different kernels and perform very well for different images. I want to see the discussions and comparisons (if possible) with these neural networks motivated by the conventional optimizer.   - The author may experiment to study the sensitivity of the proposed model to the noise. - More experiments on the standard non-blind deconvolution benchmark datasets are necessary to validate the proposed method.   [1] L. Xu, J. S. Ren, C. Liu, and J. Jia. Deep convolutional neural network for image deconvolution. In NIPS, 2014.  [2] Kruse, Jakob, Carsten Rother, and Uwe Schmidt. "Learning to push the limits of efficient fft-based image deconvolution." IEEE International Conference on Computer Vision (ICCV). 2017. [3] Gong, D., Zhang, Z., Shi, Q., Hengel, A. V. D., Shen, C., & Zhang, Y. (2018). Learning an Optimizer for Image Deconvolution. arXiv preprint arXiv:1804.03368. [4] Zhang, K., Zuo, W., Gu, S., & Zhang, L. (2017, July). Learning deep CNN denoiser prior for image restoration. In IEEE Conference on Computer Vision and Pattern Recognition (Vol. 2).  The authors address my major concerns. I tend to vote for acceptance.


Contextual Stochastic Block Models
We provide the first information theoretic tight analysis for inference of latent community structure given a sparse graph along with high dimensional node covariates, correlated with the same latent communities. Our work bridges recent theoretical breakthroughs in the detection of latent community structure without nodes covariates and a large body of empirical work using diverse heuristics for combining node covariates with graphs for inference. The tightness of our analysis implies in particular, the information theoretical necessity of combining the different sources of information. Our analysis holds for networks of large degrees as well as for a Gaussian version of the model.

UPDATE after authors' rebuttal I am quite happy about the scientific answers provided by the authors. Hence I upgrade my score from 6 to 7. Hoping it will be enough for an acceptance to NIPS (competition is harsh!), and if not, still hoping to read your work somewhere!  This paper is about the clustering of statistical entities for which network data and individual measurements are available. The article reads well and is technically sound. he numerical experiments are disappointing, I don't really understand how they illustrate the performance of the method. I would have expected some accuracy measure for the methods on its ability to identify meaningful clusters. Moreover, I challenge the novelty of the approach presented here. It was easy to find references like https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5890472/ (very similar problem with the same kind of data) or less recently https://ieeexplore.ieee.org/document/4359897/ (with the implementation the authors published in https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3051335/, you could well compare to your approach). I truly feel you should compare the method you present to the state-of-the-art literature. To me, your model resembles a Hidden Conditional Random Fields (or a Markov Random Fields) via Equations (1) and (2). Details:  - l17, the A data is not necessarily a 'similarity' measure, it could just be a graph measure directly defined, e.g. regulatory relationships. A similarity measure is more a representation of similar measures on nodes, hence more a proxy for a B data in your settings.  - l65: you focus on bi-partitioning, which in graph is clearly an easier (yet challenging) special case of the general framework. Why is that? How can your method be applied with K classes and K>2?  - I quite like the 'in Other words...' explanation of our theorems.  - What is a 'Claim' for you? A Proposition? A conjecture?  - Again, your algorithm (Section 4) could be compared to traditional approaches like Variational Inference and MCMC strategies.  - In the reference, make the style uniform: with or without first names?


Does mitigating ML's impact disparity require treatment disparity?
Following precedent in employment discrimination law, two notions of disparity are widely-discussed in papers on fairness and ML. Algorithms exhibit treatment disparity if they formally treat members of protected subgroups differently; algorithms exhibit impact disparity when outcomes differ across subgroups (even unintentionally). Naturally, we can achieve impact parity through purposeful treatment disparity. One line of papers aims to reconcile the two parities proposing disparate learning processes (DLPs). Here, the sensitive feature is used during training but a group-blind classifier is produced. In this paper, we show that: (i) when sensitive and (nominally) nonsensitive features are correlated, DLPs will indirectly implement treatment disparity, undermining the policy desiderata they are designed to address; (ii) when group membership is partly revealed by other features, DLPs induce within-class discrimination; and (iii) in general, DLPs provide suboptimal trade-offs between accuracy and impact parity. Experimental results on several real-world datasets highlight the practical consequences of applying DLPs.

Summary:  This paper describes a series of negative results for "disparate learning processes" (DLPs). These are recent methods to train classification models whose predictions obey a given fairness constraint between protected subgroups (e.g., by solving a convex optimization problem or preprocessing). The key characteristic of DLPs is that they have access to labels for the protected groups at training time, but not at deployment. In practice, this means that a DLP will produce a single model for the entire population.  In this paper, the authors present several arguments against DLPs, including:   1. If the protected attribute is redundantly encoded in other features, then a DLP may result in the same treatment disparity that aims to overcome.  2. If the protected attributed is partially encoded in other features, then a DLP may induce within class-discrimination and harm certain members of the protected group.  3. A DLP provides suboptimal accuracy in comparison to other classifiers that satisfy a specific family of fairness constraints.   The authors present several proofs of the theorems for an idealized setting. They complement the theorems with a series of empirical results on a real-world dataset and UCI datasets showing the shortcomings of DLPs and the superiority of simple alternatives (e.g. training a logistic regression model and applying per-group thresholding).  Comments:  I thought that this is a well-written paper on an interesting and timely topic. I think that it is important to state that the central contribution of this paper is NOT methodological novelty or theoretical novelty, but rather the results against DLPs. I believe that these results will be significant in the long-term as much recent work in this area has focused on tackling fairness-related issues through DLPs without considering their shortcomings.  I believe that the current paper would significantly strengthened if the authors were to consider broader approaches to treatment disparity. For example, many readers would consider Dwork et al's proposed approach to train “separate models for separate groups” as a natural extension to the per-group thresholding used in the numerical experiments. The authors currently cite these methods in the text under the “alternative approaches” in Section 2 and the “fairness beyond disparate impact” in Section 5. However, they should be discussed more thoroughly and included in the experimental results in Section 4.    Other than this, two other major points to address:  - Among the arguments presented against DLPs, 1. deserves some qualification.  It is unlikely that a practitioner would not check for that the protected attribute is redundantly encoded before training the model.  - The results in Section 3 apply to a setting where the authors have access to the optimal classifier d*. This is fine given that the results are in line with their empirical work in Section 4. However, the authors should explicitly state that this does not capture the real-world setting (i.e., where the classifier is learned from data). There should also be a discussion as to when can should expect the results to manifest themselves in the real-world setting  Specific Comments:  - The main findings (e.g. points 1. through 4) at the end of the Introduction are great. I would recommend adding a short example after each point so that readers from a broader audience will be able to understand the points without having to read the papers.  - Proofs in Section 3 could be put into an Appendix for the sake of space.  - The “Complicating matters…” (line 29) in the Introduction is interesting and merits further discussion. I’d suggest placing some of the discussion from lines 318 – 326 here.   - “Alternative Approach” in Section 2 and the “Fairness beyond Disparate Impact” in Section 5 should be merged (see also comments above)  - Section 2: 0-1 loss should be defined in line 70. I'd recommend changing the Section title to Preliminaries.  Minor Comments:  - Table 2: the formatting can be improved. The column labels should be in a different  - Line 6 “\textit{dis}parity”


Multiplicative Weights Updates with Constant Step-Size in Graphical Constant-Sum Games
Since Multiplicative Weights (MW) updates are the discrete analogue of the continuous Replicator Dynamics (RD), some researchers had expected their qualitative behaviours would be similar. We show that this is false in the context of graphical constant-sum games, which include two-person zero-sum games as special cases. In such games which have a fully-mixed Nash Equilibrium (NE), it was known that RD satisfy the permanence and Poincaré recurrence properties, but we show that MW updates with any constant step-size ε > 0 converge to the boundary of the state space, and thus do not satisfy the two properties. Using this result, we show that MW updates have a regret lower bound of Ω(1/(εT )), while it was known that the regret of RD is upper bounded by O(1/T ). Interestingly, the regret perspective can be useful for better understanding of the behaviours of MW updates. In a two-person zero-sum game, if it has a unique NE which is fully mixed, then we show, via regret, that for any sufficiently small ε, there exist at least two probability densities and a constant Z > 0, such that for any arbitrarily small z > 0, each of the two densities fluctuates above Z and below z infinitely often.

************* POST-REBUTTAL ******************* I gave an "accept" recommendation during the first review phase, but I was (negatively) surprised when the authors' rebuttal completely ignored part of my review, namely that Theorem 2 has already been reported in a more general, unilateral setting by Sorin (Mathematical Programming, 2009), Hofbauer, Sorin and Viossat (Mathematics of Operations Research, 2009), and subsequently extended to more general dynamics by Kwon and Mertikopoulos (Journal of Dynamics and Games, 2017) and Mertikopoulos, Papadimitriou, and Piliouras (SODA, 2018). In particular, the first and third papers above discuss extensively the interplay between continuous and discrete time - an interplay which plays an important role in the current submission as well - while the second and last papers also identify constants of motion under the replicator dynamics and discuss Poincaré recurrence.  I still believe the paper should be accepted, but I've downgraded my recommendation to a "weak accept" to indicate my reservations concerning the authors' positioning relative to previous work.   ************* PRE-REBUTTAL ******************* In this paper, the authors study the behavior of the multiplicative-weights algorithm with a constant step-size in zero-sum games (both standard, two-player games, and graphical games played over an interaction graph). The authors' main contributions can be summarized as follows:  1. The continuous-time limit of the multiplicative weights algorithm (known as the replicator dynamics in evolutionary game theory) incurs regret that is at most O(1/T) after time T.  2. The dynamics are "permanent" in the sense that every trajectory thereof is contained in a compact set that is disjoint from the boundary of the game's mixed strategy space (i.e. in the long term, the share of every strategy is bounded away from zero).  3. The above conclusions do not hold if the algorithm is run with a constant, positive step-size: in this case, the algorithm's regret is bounded from below as \Omega(1/(epsilon T)) where epsilon is the algorithm's (constant) step-size, and the dynamics are not permanent (nor Poincaré recurrent).  The paper is well-written, clearly structured and I enjoyed reading it. However, some of the authors' contributions are not new (see below), so the overall novelty is limited. Still, the results that are new are (in my opinion) sufficient and will interest the NIPS community, hence my positive evaluation.   My main criticisms are as follows:  1. The fact that the discrete analogue of the replicator dynamics has a different behavior than the continuous dynamics is not as surprising as the authors claim, see for instance the 2009 paper of Sorin in Mathematical Programming and the monograph of Benaïm (1999) on stochastic approximations, where it is made clear that constant step-size discretizations of dynamical systems could have a wildly different behavior than their continuous-time counterparts.  2. The fact that the replicator dynamics (i.e. the continuous-time limit of the multiplicative weights algorithm) lead to O(1/T) regret was first reported by Sorin (Mathematical Programming, 2009) and Hofbauer, Sorin and Viossat (Mathematics of Operations Research, 2009), and subsequently extended to all mirror descent / follow-the-regularized-leader dynamics by Kwon and Mertikopoulos (Journal of Dynamics and Games, 2017) and Mertikopoulos, Papadimitriou and Piliouras (SODA, 2018). In fact, this O(1/T) bound does not even require a game: it holds for arbitrary (measurable) series of payoffs, even if those are not coming from a game.  2. The permanence result for the replicator dynamics is also alluded to in the classical book of Hofbauer and Sigmund and the more recent paper of Hofbauer, Sorin and Viossat (2009). [Granted, these works do not mention graphical zero-sum games, but the essence of the result is there and the proof strategy is exactly the same] See also the SODA 2018 paper of Mertikopoulos, Piliouras, and Papadimitriou where the authors prove a more general   4. In general, why would one want to employ the multiplicative weights algorithm with a constant step-size? As the authors state themselves, achieving no-regret in an anytime setting requires a decreasing step-size anyway (either due to a doubling trick or a sqrt{t} schedule). It is also well-known that a decreasing step-size is much better suited to stochastic environments (which are the norm in any randomized action selection framework), so it is not clear why one would want to use a constant step-size algorithm in the face of uncertainty (either temporal or informational).  There are some further minor issues (such as the requirement that epsilon ≤ 1/4), but these are not important enough to raise here.


Metric on Nonlinear Dynamical Systems with Perron-Frobenius Operators
The development of a metric for structural data is a long-term problem in pattern recognition and machine learning. In this paper, we develop a general metric for comparing nonlinear dynamical systems that is defined with Perron-Frobenius operators in reproducing kernel Hilbert spaces. Our metric includes the existing fundamental metrics for dynamical systems, which are basically defined with principal angles between some appropriately-chosen subspaces, as its special cases. We also describe the estimation of our metric from finite data. We empirically illustrate our metric with an example of rotation dynamics in a unit disk in a complex plane, and evaluate the performance with real-world time-series data.

* Summary : This paper introduces a general framework to compute a metric between deterministic dynamical systems. This framework relies on the Koopman operator computed in RKHS, which was previously introduced in [Dynamic mode decomposition with reproducing kernels for koopman spectral analysis, Kawahara, NIPS18]. The framework is generic and the authors show that it encompasses previous metrics introduced for linear dynamical systems [Subspace angles between ARMA models, De Cock and De Moor, Systems and Controls Letter 2002] and non-linear ones [Binet-Cauchy kernels on dynamical systems and its application to the analysis of dynamical scenes, Vishwanathan et al.. 2007]. An algorithm is proposed in order to compute this metric from the observation of multiple trajectories generated by these different dynamical systems. Visualization experiments on synthetic and real data show that this metric seems to separate dynamical systems with different properties.  * Review :  This paper is interesting and brings a new framework to study dynamical systems with the Koopman operator, which makes it a significant contribution.  Clarity should be improved. I think that the notations and objects introduced need to be greatly slimmed down to simplify the exposition, just keeping the minimum to reach the main goals of the paper. Further, it would be nice to provide more intuition on the different objects which are introduced in Subsection 3.1.  When it comes to experiments, the visualizations provided are nice, but running a true machine learning algorithm would have been better. For instance, on the real time series, it would have been nice to run a clustering algorithm on top of the metrics computed, to see for instance if it is able to retrieve the original classes, yielding more quantitative measurements.  Overall, I would tend in favor of accepting this paper, provided it is clarified.  * Detailed remarks:  - As far as I understand, $\tilde{A}_m$ is introduced to allow convergence even in case of admissibility and not semi-stability (cf Prop 3.3). This is a nice results, but it is not exploited in the experiments. Why, or why not?  - In Sec 3.1, it would be nice to provide more intuition on the different objects h, L_h, and the subspaces H', H''. What do they represent? Maybe notations could be simplified to introduce only L_h and remove h?  - Is the object $d_m^T$ a positive definite kernel? Or a metric? I have not seen it stated anywhere, or any hint of proof of it (I am not sure it is even named). Since this object is very crucial, it may deserve a definition in its own right and much more explanations or intuitions. What is the role of the parameter $m$ for instance?  - What was the motivation for introducing $A_m^T$ from $d_m^T$? Is it only to take into account all previous works in this framework, or is there something more subtle at work? Is it obvious that $A_m^T$ defines a positive definite kernel?  - Indices and notations need to be slimmed to the minimum over all the paper. For instance, I am not sure that Proposition 2.1 brings much to the main goal of the paper, as it is not used anywhere else. Using multiple $A$ in 4.1 brings confusion.


Neural Edit Operations for Biological Sequences
The evolution of biological sequences, such as proteins or DNAs, is driven by the three basic edit operations: substitution, insertion, and deletion. Motivated by the recent progress of neural network models for biological tasks, we implement two neural network architectures that can treat such edit operations. The first proposal is the edit invariant neural networks, based on differentiable Needleman-Wunsch algorithms. The second is the use of deep CNNs with concatenations. Our analysis shows that CNNs can recognize regular expressions without Kleene star, and that deeper CNNs can recognize more complex regular expressions including the insertion/deletion of characters. The experimental results for the protein secondary structure prediction task suggest the importance of insertion/deletion. The test accuracy on the widely-used CB513 dataset is 71.5%, which is 1.2-points better than the current best result on non-ensemble models.
Needleman-Wunsch algorithm [17] as a differentiable neural network. Next, we show that deep CNNs with concatenations can treat regular expressions without Kleene stars, indicating that such CNNs can capture edit operations including insertion/deletion. Our experiments demonstrate the validity of our approach. The test accuracy of protein secondary structure prediction on the widely-used CB513 dataset (e.g., [26]) results in 71.5% accuracy, the state-of-the-art performance compared to those of previous studies on non-ensemble models. Differentiable Sequence Alignment. In bioinformatics, sequence alignment is key in comparing two biological strings (e.g., DNA, proteins). The NeedlemanWunsch (NW) algorithm [17], a fundamental sequence alignment algorithm, calculates the similarity score between two strings on an alphabet Σ. As shown in Fig.  1, the similarity score is computed via dynamic programming to maximize the total score (illustrated as the double-lined square) by inserting or deleting characters (depicted as the vertical and horizontal arrows, respectively).

This manuscript describes two methods for representing sequence editing operations in neural networks. The authors are motivated by problems in bioinformatics, in which insert, deletion and substitution operations represent basic moves in sequence evolution.  The two techniques are complementary: one replaces the inner loop of the neural net training with a differentiable version of the Needleman-Wunsh dynamic programming algorithm; the other describes a general method for constructing convolutional network architectures that represent a particular class of regular expressions.  Each approach has its drawbacks.  The first method is elegant but computationally expensive; the second method naturally leads to very large networks, requiring some hoop-jumping in order to get the networks down to a more reasonable size. The authors demonstrate the utility of both methods on a small example, and then show that the CNN approach can lead to state-of-the-art results on the prediction of protein secondary structure.  Overall, this manuscript was a pleasure to read, and the core ideas are compelling.  It is certainly the case that deep architectures are being applied more and more widely within bioinformatics, and creative ways to represent sequence operations in this domain are likely to be of very wide interest.  One concern I have is about the experimental results: the authors misrepresent their achievement relative to the state of the art.  In the abstract they explicitly state that they have improved on the state of the art by 1.2% in accuracy.  This is simply false, as pointed out in the footnote on p. 6: they have only improved on non-ensemble models by 1.2%.  The best ensemble method achieves an accuracy only 0.1% less than the method reported here.  The artificial choice to exclude ensemble methods is completely unmotivated. Indeed, there is no reason the authors could not have created ensembles of their own method.  A second considerable caveat to the experimental results is that the authors have clearly and explicitly engaged in data snooping on the CB513 benchmark.  The manuscript reports results from a wide variety of models on this benchmark, and presumably others were tried as well. As such, these results are likely overfit to this benchmark.  To truly show that they have achieved the state of the art, the authors should have used an independent benchmark to do model development and then run only the best-performing method on CB513.  One drawback to the first contribution (EINN) is that the Needleman-Wunsch algorithm is not, actually, widely used at all in bioinformatics. First, any DP algorithm for biosequence alignment needs to be generalized to use affine gap costs (i.e., charge more for introducing a new gap than for extending an existing gap).  Second, only in rare cases is a global alignment required; it is much more common to use a local alignment algorithm like Smith-Waterman. The authors claim on line 211 that they can also apply the EINN method to SW, but this should be done or at least mentioned earlier. Third, in practice, both NW and SW are too slow for most bioinformatics applications, so heuristic approximations like BLAST are much more widely used.  Given these caveats, the description of NW as "one of the most heavily-used sequence alignment algorithms" (line 43) is incorrect and misleading. Note that I don't see any conceptual reason why the proposed method could not be extended to affine gap penalties and local alignment; only that the authors should (1) admit that the method they have implemented falls short of capturing what is used in practice, and (2) at least sketch how the method could be extended to these other settings.  Another concern regarding the empirical results is the "data augmentation" strategy proposed on lines 272-280. The method consists, essentially, of adding noisy versions of the training examples to the input. It is surprising that this method helps, and if this techniques actually accounts for up to 0.8% of the observed improvement, then it is just as important as the star-free regular expression CNN stuff that is purported to be the main advance in this part of the paper.  The authors should discuss the motivation for this technique in more detail.  The writing throughout is clear, though the text exhibits numerous grammatical and usage errors.  I list some of these below, but the authors are encouraged to identify others as well.  The definite or indefinite article is misused (included where not needed, skipped where needed, or wrong type of article) in lines 42, 63, 77, 107, 116, 126, 155, 231, 232.  12: delete "of"  79: Delete "of the."  137: "accepting" -> "accepts"  144: "star" -> "stars"  171, 203: The verb "allow" requires a direct object.  186: Semicolon before "however."  214: "neuron" -> "neurons"  Figure 3: "Concatination" -> "Concatenation"  119: "does" -> "means"  242: "reminder" -> "remainder"  297: "of EINN" -> "for EINNs"  304: "indicates the time complexity" -> "yields a time complexity of"  Other minor comments:  35: Briefly define "star-free regular expression" here.  37: State that CB513 is for protein secondary structure prediction.  101: Delete the sentence beginning "Therefore, ..." (It is obvious, given the previous sentence).  187 "Real biological tasks": Did the authors evaluate the method on tasks other than secondary structure prediction? Mention details here.  I read the author response, and it seemed reasonable. I do think the paper has some problems with the empirical evaluation (as also commented on by Reviewer #2), but they are not fatal flaws. Hence, I did not modify my score.


Efficient Online Portfolio with Logarithmic Regret
We study the decades-old problem of online portfolio management and propose the first algorithm with logarithmic regret that is not based on Cover's Universal Portfolio algorithm and admits much faster implementation. Specifically Universal Portfolio enjoys optimal regret O(N ln T ) for N financial instruments over T rounds, but requires log-concave sampling and has a large polynomial running time. Our algorithm, on the other hand, ensures a slightly larger but still logarithmic regret of O(N 2 (ln T ) 4 ), and is based on the well-studied Online Mirror Descent framework with a novel regularizer that can be implemented via standard optimization methods in time O(T N 2.5 ) per round. The regret of all other existing works is either polynomial in T or has a potentially unbounded factor such as the inverse of the smallest price relative.

This paper introduces a new algorithm (BARRONS) for the online portfolio optimization problem. This algorithm obtains logarithmic regret in time quadratic in the number of rounds without any potentially catastrophic dependence on a gradient norm. This is in contrast to the Cover's universal portfolio algorithm, online newton step, and exponentiated gradient, which each achieve at most two of these three goals.  The algorithm itself is an application of the online mirror descent framework, using the less-classical log-barrier regularizer. The paper provides good intuition for how the algorithm is able to avoid dependence on the gradient norm: the gradient norm is only big if a previously poorly-performing stock starts to perform well. As a result the learner has a kind of "surplus" of regret that it can fall back on while adapting to the new stock.  The algorithm still leaves something to be desired: the total running time T^2N^(2.5) where N is the number of stocks (the dimension of the problem) and T is the number of rounds. Both of these dependencies are rather poor (although much better than the universal portfolio algorithm). Also the regret is N^2log^4(T), rather than the ideal N\log(T).   The authors suggest that the T^2 aspect of the runtime might be improved by using more intelligent optimization algorithms in a subroutine of the main algorithm. This may not bear out, but it suggests that there are at least some obvious points of attack for improving the results.  In a pleasant surprise for a theory paper, Most of a complete proof of the regret bound fits into the main text of the paper.  Quality: This is reasonably clean approach to an old problem that achieves non-trivial new bounds. Overall this seemed a good-quality paper.   Clarity: The background and the contrast between prior approaches was well-explained.  Originality: The main original ideas in this paper seems to be the application of the log-barrier potential to this particular problem, and the intuition used to remove the gradient-norm dependence in the regret. I'm not aware of similar results on the problem.  Significance: The main improvement of this paper is to remove the dependence on the gradient norm in the portfolio optimization problem. Both the regret bound and time complexity have some distressing high exponents, so there is still significantly more work to be done. However, the removal of the gradient norm from the bound is important as it gets rid of a kind of "wishful thinking" aspect of other bounds, and it is significantly faster than the only other algorithm to obtain this (universal portfolio, which is abysmally slow), so I think this paper represents a nice step of progress.


Simple random search of static linear policies is competitive for reinforcement learning
Model-free reinforcement learning aims to offer off-the-shelf solutions for controlling dynamical systems without requiring models of the system dynamics. We introduce a model-free random search algorithm for training static, linear policies for continuous control problems. Common evaluation methodology shows that our method matches state-of-the-art sample efficiency on the benchmark MuJoCo locomotion tasks. Nonetheless, more rigorous evaluation reveals that the assessment of performance on these benchmarks is optimistic. We evaluate the performance of our method over hundreds of random seeds and many different hyperparameter configurations for each benchmark task. This extensive evaluation is possible because of the small computational footprint of our method. Our simulations highlight a high variability in performance in these benchmark tasks, indicating that commonly used estimations of sample efficiency do not adequately evaluate the performance of RL algorithms. Our results stress the need for new baselines, benchmarks and evaluation methodology for RL algorithms.

Post rebuttal:  I agree with the concerns raised about MujoCo environment used for benchmarking RL algorithms. After reading the other reviews and authors' response I remain unconvinced that they make a good case. My concern is with the methodology of the approach, and I cannot see how it can be addressed in the final version.   In the rebuttal, lines 8-12, the authors state: "We offer evidence for the need of new baselines, benchmarks and evaluation methodology for RL algorithms. We show that under current evaluation standards a simple method with linear policies can match or outperform the performance of more complicated methods on current, standard benchmark tasks. Nonetheless, more rigorous evaluation reveals that the assessment of performance on these benchmarks remains overly optimistic because it restricts its attention to too few random seeds."  This argument would hold if the paper showed that vanilla random policies reach near RL performance. Instead, the authors tweak and customize algorithms to specific problems. It is the modification of the algorithms that take away from the argument. Hand-coded and highly-custom algorithms generally perform well for specific problems, and the power of more complex learning methods lies in their generality.   The authors did not address in rebuttal the differences to Choromanski et al., “Structured Evolution with Compact Architectures for Scalable Policy Optimization,” ICML 2018. This paper uses a more generic and simple algorithm on MojoCo tasks. Methods such in Choromanski et al. can be used to make a better case about the complexity of MujoCo. ------------------  This paper argues for simple baselines on reinforcement learning tasks, and proposes a new such baseline. ARS (Augmented Random Search) is a simple, gradient-free algorithm for training linear policies. The authors compare ARS to a number of RL algorithms, presenting results on sample efficiency, sensitivity, and rewards achieved on MuJoCo locomotion tasks. They redefine “sample efficiency” to include hyperparameter search and argue for more sensitivity analysis in the literature.  The paper comes at a good time, with the recent reproducibility issues in RL. The paper’s arguments on sample efficiency and transparency are compelling. The analysis of hyperparameter search and random seed sensitivity sets a good example for the field. The intuition behind the algorithm augmentations is clear. The algorithm, being random search over linear policies, is quite simple and makes me reconsider the need for complex algorithms on these tasks.  It is unclear what the main contribution of the paper is: to show that MuJoCo is too simple, to show how RL algorithms should be evaluated, or to introduce a good baseline algorithm? The paper tries to do all of these, but it would be more effective to focus on one and frame the paper around it. For example, if the goal is to introduce a simple, effective baseline, ARS V2-t should be sufficient, without discussing V1.  Specifically, in Figure 1, it’s strange to present one algorithm ARS V1 on Swimmer-v1 and use ARS V2 and V2-t on other tasks. It’s somewhat acceptable to use both V2 and V2-t, since as mentioned the difference is just tuning hyperparameter b. In a paper contributing a simple, robust baseline to the field, a single algorithm should work on all tasks? A similar argument could be made against choosing the hyperparameters differently for the sample efficiency and maximum rewards results.  In the discussion section, it is argued that “a meaningful sample complexity of an algorithm should inform us on the number of samples required to solve a new, previously unseen task.” Under this definition, it would be nice to see a comparison to metalearning algorithms.  Other smaller comments: It would be great if the sensitivity analysis were presented for other algorithms as well, especially on HalfCheetah-v1, as you say this is often used for evaluation of sensitivity. Table 1: bold the most sample efficient algorithm for each task? Table 1: the humanoid-v1 comparison isn’t compelling. Maybe include more results from other algorithms, or remove all discussion of humanoid-v1? In Table 2, it would be great to show the highest reward encountered in the literature for each task, some of what you have in appendix tables 4 and 5. Paper “Structured Evolution with Compact Architectures for Scalable Policy Optimization,” ICML 2018 should be cited as an evidence that more simple, evolutionary policies solve complex tasks.  Questions:  Why weights initialized to 0, not random?  Why grid of hyperparameters instead of random?


Derivative Estimation in Random Design
We propose a nonparametric derivative estimation method for random design without having to estimate the regression function. The method is based on a variance-reducing linear combination of symmetric difference quotients. First, we discuss the special case of uniform random design and establish the estimator's asymptotic properties. Secondly, we generalize these results for any distribution of the dependent variable and compare the proposed estimator with popular estimators for derivative estimation such as local polynomial regression and smoothing splines.

Paper 1768  This paper considers the univariate regression setting Y_i = r(X_i) + epsilon_i for X_i taking value in [0,1] and independent epsilon_i. The authors study the estimation of the derivative r'(x) of the regression function for x in the interior (0,1). The authors use a weighted finite difference ratio estimator and analyze its behavior in the random design setting where X_i has the uniform density. The main result yields a pointwise rate of convergence for the estimator on the data points X_2, ..., X_{n-1}. Extensions to non-uniform design points are discussed.   I find the problem and the results interesting. I think the paper is close to a state where it can be published but it needs to be developed some more before it is ready.  I like the results but I think they are a bit limited in scope and leave too many immediate natural questions unanswered. The main result characterizes the bias and variance of the estimated derivative at a data point X_i; it is natural to think about the behavior of the estimator at an arbitrary point x in the interior of of (0,1). I would guess that the estimator still behaves well under the same bounded second derivative condition that the authors imposes on the regression function r.  Another natural question is whether the estimated derivative is asymptotically normal. Theory aside, the expression (3) immediately leads one to think about kernel weighing. I hypothesize that using a kernel weighted version of (3) gives an estimator whose behavior is similar to that of one proposed by the authors. Finally, in the case where X is non-uniform but has a density that is bounded away from 0, I would imagine that the exact same method should work with the same, up to a constant, bounds on the bias and variance. It seems excessive in the case of non-uniform X to follow what the authors suggests in section 2.7, which involves estimating the density itself.  I understand it is unfair for a reviewer to speak on and on about what the authors could do. I raise these open questions because I think they are natural and not too difficult to address. I could be wrong of course, in which case, I would be happy if the authors could discuss why they are difficult.   A second issue I have with the paper, which is not a huge issue, is that the mathematical statements are loose. Some small degree of mathematical rigor is needed for a NIPS statistical methodology paper and this paper does not quite reach that standard. In the statement of theorem 1, it does not make sense to condition on the samples U because that yields the fixed design setting. I think what the authors intend is E_U[ \hat{Y}_i - r'(U_i) | U_i], that is, the expected estimate at the point U_i fixing only the value of the point U_i. I think it is clearer to have an unconditional statement, which holds by just taking an iterated expectation. I list a few other issues.  + In expression (4), it is very strange to have the subscript on Y be just ``i'' and to have the subscript on U be ``(i)''. I suggest saying something like, ``by reindexing if necessary, assume that U_1 \leq U_2 \leq ... ``.  + In Section 1.1, it is not technically correct to write m(x) = E(Y) and m(X) = E[Y | X]. The correct expression is m(x_i) = E[Y_i] in the first case and m(x) = E[ Y | X = x] in the second case.  + The authors say that ``we require P(U_(l) = U_(m)) = 0'', but that is already true since the U's are uniformly distributed on [0,1]. I suggest saying something like, ``we observe that because U_i has a density, P(U_(l) = U(m)) = 0 and thus the estimator is well-defined on U_i for all i = 2, ..., n-1''.   + I recommend giving an explicit rate in corollary 1, with k chosen optimally.   + Corollary 2 isn't quite right because the the first term is based on an upper bound of the bias.   + The authors say in line 202 that they need the existence of a density in order for expression (10) to hold but they in fact need the existence of a density earlier. The distribution of X_1 needs to be absolutely continuous with respect to the Lebesgue measure in order for F(X_1) to be uniformly distributed. If the distribution of X_1 has a component singular to the Lebesgue measure, then the quantile function is not the inverse of the distribution function F and F(X_1) is not uniformly distributed.   + I don't know of a result in [1] that describes the optimality of derivative estimators. [1] mostly deals with the estimation of an additive regression function itself. It would be helpful if the authors could provide a reference to the particular theorem in the paper.  [1] C. Stone. Additive regression and other nonparametric models.   EDIT: I have read the authors' response. I agree that the conditional on U statement as it appears in the paper is actually correct; I had misunderstood it before. I also agree that F needs to only be continuous and not absolutely continuous as I had mistakenly stated before. I apologize for these mistakes.   I do not have a reference for using kernel weighing. It just strikes me that the authors' weighing scheme resembles kernel weighing where the kernel is K(u) = 1/u^2 * I(u \leq 1). See an analysis of this kernel in [1]. Bandwidth selection is certainly an issue but so is the selection of "k" in the authors' scheme.   I still do not understand why estimating the density f(X) is necessary. It seems that Lemma 1 would hold if X is non-uniform but has a density that is bounded away from 0. It would be great if the authors could comment on this in the paper.   I don't think that studying the behavior of \hat{r} is "sufficient for another paper", especially since the authors already use the random design setting. I think not being able to say anything about the behavior of \hat{r} except on the sample points is the most significant weakness of the paper.   I have upgraded my assessment of the paper though I still recommend that the authors take some time to further develop the paper before submitting it for publication. But, I don't have very strong objections to accepting the paper as it is.   [1] Belkin, M, Rakhlin, A, and Tsybakov, A. (2018) Does data interpolation contradict statistical optimality?  Arxiv Preprint: 1806.09471


A Game-Theoretic Approach to Recommendation Systems with Strategic Content Providers
We introduce a game-theoretic approach to the study of recommendation systems with strategic content providers. Such systems should be fair and stable. Showing that traditional approaches fail to satisfy these requirements, we propose the Shapley mediator. We show that the Shapley mediator fulfills the fairness and stability requirements, runs in linear time, and is the only economically efficient mechanism satisfying these properties.

This paper studies a game design problem. There are U users and P players. Each player has a set of possible of actions. Each action of a user gives a certain utility to each player. There is a "mediator" that, upon receiving a profile of actions from all the players, will choose which action to display for each user. Thus the mediator algorithm defines the game, and each player gets a utility of 1 when their action is displayed to a user. The question is can we develop a "fair" mediator algorithm that will result in the underlying game to have a pure Nash equilibrium, where fairness includes intuitive properties like for each user, the highest utility action should be chosen with strictly higher probability than the rest of the actions etc. There is another property called completeness, which states that each user should be displayed an action with probability 1 (and not lesser than 1).   The authors show that no game can simultaneously satisfy fairness, completeness and have a pure strategy Nash equilibrium. They propose a game that chooses each action with probabilities proportional to the Shapley value suggested increments and show that this satisfies fairness and has a pure strategy Nash. They then propose to study the price of Anarchy, both for user utility and player utility. The Shapley mediator algorithm has a poa of 2 for the player utility and unbounded for the user utility model.   My concerns about this paper are I am not convinced that in any of the proposed applications, existence of an equilibrium is an essential concern. In particular, the actions in all recommendation systems are suggested content, and the suggested content is going to vary quite frequently --- no content is going to be shown more than a small fixed number of times to any user (I think of news articles, ads etc.). That being the case why is equilbrium a main concern? Even if we are concerned with the same content being shown often, we still need to assume that the same competing contents are also present in all the repeated interactions. If one forgets the equilibrium, the TOP algorithm has fairness and completeness, has a player POA of 1 and user POA unbounded. It's a much simpler algorithm to implement.  Overall, I am not convinced that theoretical analysis of the equilibrium is the right problem to study in this space. The game itself is quite cute though. Also, for a NIPS-like conference, absence of even a tangential connection to any form of Machine Learning is a minus.    ************************************* After author response:  Author  response quote: "In most (and perhaps even all) forms of multi-agent interaction in the Web, mechanism design is carried out to mitigate agent incentives."      ---- Please be specific. I am aware that game theory and equilibrium play a central role in many settings, ad auctions being one of the primary examples. There are many relevant questions for equilibrium's applicability. But one of the most important is:   how often is the *same* game repeated over time, for things like repeated best response to make sense?  First, in personalized offer case, whenever the set of possible actions L_j available to player j is reasonably small (e.g. the set of possible news articles a publisher can promote on any given day is not huge), then player j submitting to the mediator the whole set L_j as their strategy X_j would mean that there are no incentives in the game: for each user i, the mediator algorithm (TOP) will just pick the player j whose best item's satisfaction is highest among all players for user i. There is no room for any incentives here, and the mediator obtains the maximum user welfare as well. The authors mitigate this uninteresting situation by saying that there is a budget k_j on the maximum number of actions that can be a part of the strategy of any player j. Personalization is an important aspect of RSs (as reviewer 2 points out as well), and specifying the regime in which the game is interesting/relevant is essential. For example, the paper begins with Alice and Bob examples, which might suggest that the game is interesting even at a fully personalized level, but it is not.   Second, even as an impersonal game (or a semi-personal game, where we put a limit on the set of usable actions as the authors suggest): the relevant question for equilibrium applicability, has to be discussed.  How often is the same game repeated over time? Same game means, the payoffs are more or less the same for any given strategy profile. For a given strategy profile consisting of the action chosen (or the set of actions chosen as in Appendix C) by every player, will the user payoffs (which ultimately determines how any mediator algorithm will pick the player item to show, and hence determine the player playoffs, which is what is the game's payoffs here) remain the same over multiple interactions? Even across a day, Alice might want not want to see too much of financial news, and may perhaps derive more utility out of knowing about local businesses via their ads. Therefore in the repeated RS game, the user payoffs most likely changes over time. So while encountering Alice the second time, the payoff she derives (which influences what the mediator chooses, and hence influences the player payoffs) is very different from the first round interaction. Hence the planned response for this interaction should also be different. One can counter this by saying that each user provides a content opportunity only a few times a day, but the sum total of all users is huge, and hence it is a repeated game --- this is fallacious though: the sum total of all users together determine one "round" of this game because the combined payoffs from showing content to all users is what determines the player payoffs in the game. With such a model of player payoffs, a repeated game in the RS context necessarily refers to same user visiting multiple times. Therefore the payoff (and hence the game) also necessarily varies over these repetitions. Circumventing this situation by decomposing player payoffs across users (i.e., by making personalized offers for each user) will make the game uninteresting for reasons described in the previous paragraph.   On the other hand, consider the same question for the GSP example. Here, the advertiser payoff can be determined for each user interaction separately (there is ample room for strategizing even when considering a single user, as opposed to the RS game that has no incentives in the personalized user case). Thus for a given keyword, repeatedly interacting with different users, are all repeated plays of the same game, with the same advertiser payoffs (or nearly same, or proportionally same if some users just click less often on ads). Thus, things like repeated best response etc. make direct sense here. And indeed it was in one such repeated best response that the famous saw-tooth bidding behavior happened in the first price auction that was previously used by search engines, prompting the move to the generalized second price auction.   In fact other repeated games like selfish routing also have this property that when all players use the same strategy profile, the payoffs remain the same. It doesn't depend on whether the same routes were already used by all car drivers yesterday! If everyone drives the same route today, the payoffs remain the same.  There are other concerns like how accurately can the players determine their payoffs for any given strategy, or even get a sense of how many players one is competing with.  All these things are hard/vague in the RS game.   ****************************************** Author response quote: "Using a TOP-like approach for search/recommendation engines (the Probability Ranking Principle, a core concept in information retrieval) has lead to the creation of a 60 Billion USD market (in 2015, in the US alone) named Search Engine Optimization (SEO)."  ---- a multi-billion dollar market for SEO might have been created regardless, partly because there are numerous so-called "white hat SEOs" that could legitimately improve the visibility and ranking of a website. There is a multi-billion dollar market just for optimizing bidding strategies in display ad auctions, which have predominantly used second price auctions for a long time (although that is changing now). What is the need for optimizing bidding strategies in a second price auction which is truthful? Perhaps the bidder is not able to predict the click-through-rate themselves, and uses the help of a demand-side-platform. Perhaps, the bidder is not able to optimally manage their budget, etc. Likewise, there are numerous reasons for an SEO market to exist, and saying "TOP-like approach for search/recommendation engines has led to the creation of a 60 Billion USD market" is quite inaccurate.   ******************************************  All that said, I still find the paper game theoretically beautiful. The way the authors design the game deserves to be published. But given the above, and the fact that the connection to learning is quite tenuous, at least NIPS doesn't seem ideal to me. I wouldn't object publication though.


Neural Arithmetic Logic Units
Neural networks can learn to represent and manipulate numerical information, but they seldom generalize well outside of the range of numerical values encountered during training. To encourage more systematic numerical extrapolation, we propose an architecture that represents numerical quantities as linear activations which are manipulated using primitive arithmetic operators, controlled by learned gates. We call this module a neural arithmetic logic unit (NALU), by analogy to the arithmetic logic unit in traditional processors. Experiments show that NALU-enhanced neural networks can learn to track time, perform arithmetic over images of numbers, translate numerical language into real-valued scalars, execute computer code, and count objects in images. In contrast to conventional architectures, we obtain substantially better generalization both inside and outside of the range of numerical values encountered during training, often extrapolating orders of magnitude beyond trained numerical ranges.

The paper proposes a method to extrapolate numbers outside the training range in neural networks. It proposes two models in order to solve this task. The neural accumulator that transforms a linear layer to output additions or subtractions and the neural arithmethic logic unit which can learn more complex mathematical functions.   The tackled problem of extrapolation is an interesting one as neural networks generally perform not so well in such tasks. The paper itself is well motivated in the introduction and the example which compares different existing structures against each other adds additional clarity that this task should be studied.   The method is interesting, novel  and sound. It is well presented through good explanations and the diagrams in figure 2.   The large experiment section further supports the proposed approach. It is compared in different problem settings against many different baselines and even against each other. The promise as well as the limitations are discussed.   The  presentation of the idea overall was well done. However, it does have some gaps. A short background section, for example, could have introduced the reader to extrapolation as a problem and could have reiterated the most important neural network mechanisms as later a deeper knowledge is expected. Furthermore, the mathematical notation (M) used in line 58 onwards is never formally introduced.   Towards the end the authors mention that their work could also be seen as a different principled approach of designing neural networks in that it biases towards certain outcomes. the paper could have discussed this in more detail as this seems wider applicable than to only extrapolation problems and, therefore, more relevant to more readers.    Overall, the paper presented its main contributions clearly and showed in a large result section the importance as well as the success of their method. While the more general underlying principle would have been interesting to explore and a background section might have added additional clarity the paper's novel ideas and impressive results  still outline a significant contribution.