Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation
In this work, we propose to apply trust region optimization to deep reinforcement learning using a recently proposed Kronecker-factored approximation to the curvature. We extend the framework of natural policy gradient and propose to optimize both the actor and the critic using Kronecker-factored approximate curvature (K-FAC) with trust region; hence we call our method Actor Critic using Kronecker-Factored Trust Region (ACKTR). To the best of our knowledge, this is the first scalable trust region natural gradient method for actor-critic methods. It is also the method that learns non-trivial tasks in continuous control as well as discrete control policies directly from raw pixel inputs. We tested our approach across discrete domains in Atari games as well as continuous domains in the MuJoCo environment. With the proposed methods, we are able to achieve higher rewards and a 2-to 3-fold improvement in sample efficiency on average, compared to previous state-of-the-art on-policy actor-critic methods. Code is available at https://github.com/openai/baselines.

The manuscript discusses an important topic, which is optimization in deep reinforcement learning. The authors extend the use of Kronecker-Factored approximation to develop a second order optimization method for deep reinforcement learning. The optimization method use kronecker-factored approximation to the Fisher matrix to estimate the curvature of the cost, resulting in a scalable approximation to natural gradients.
The authors demonstrate the power of the method (termed ACKTR) in terms of the performance of agents in Atari and Mujoco RL environments, and compare the proposed algorithm to two previous methods (A2C and TRPO).
Overall the manuscript is well-written and to my knowledge the methodology is a novel application to Kronecker-factored approximation.
I have the following comments to improve the manuscript:
1- I believe the complexity of the ACKTR, A2C and TRPO algorithms are not the same. The authors compare the performance of these algorithms per iteration in most figures, which shows that ACKTR is superior to the other methods. I suggest that the authors make the same comparison with x-axis representing optimization time instead of number of iterations, thus taking into account the complexity of the different algorithms. This is done in Figure 2 for one of the games. I am proposing here that this time comparison also be performed in the other experiments.
2- I wonder how would a method like ADAM perform relative to ACKTR.
3- The details of the algorithm are distributed at different parts of section 3. I think it is important to include a summary of the algorithm at the end of section 3 or at least in the suppmats to make it clearer.


Premise Selection for Theorem Proving by Deep Graph Embedding
We propose a deep learning-based approach to the problem of premise selection: selecting mathematical statements relevant for proving a given conjecture. We represent a higher-order logic formula as a graph that is invariant to variable renaming but still fully preserves syntactic and semantic information. We then embed the graph into a vector via a novel embedding method that preserves the information of edge ordering. Our approach achieves state-of-the-art results on the HolStep dataset, improving the classification accuracy from 83% to 90.3%.

The paper addresses the problem of premise selection for theorem proving. The objective in this line of work is to learn a vector representation for the formulas. This vector representation is then used as the input to a downstream machine learning model performing premise selection.
The contributions of the paper are: (1) a representation of formulas as graph; (2) learning vector representations for the nodes and ultimately for the formula graphs; and (3) show empirically that this representation is superior to the state of the art on theorem proving data sets.
Pros
* The paper is very well written. A pleasure to read and despite its technical content highly accessible.
* The experimental results show a significant improvement over the state of the art.
* The authors discuss all related work I'm aware of and do it in a fair and comprehensive way.
Cons
* There has been previous work on this topic. The novelty here is the graph representation and the way in which node embeddings are learned. The former is a nice idea. The latter is more interesting from a ML point of view but it would have made the paper much stronger if the authors had compared the averaging of neighboring node embeddings approach to other node embedding approaches such as DeepWalk.
This has been added after the rebuttal:
The main weakness of the paper is in my opinion the (lack of) novelty of the proposed approach. The difference to previous work is the graph representation of the formulas. To represent formulas as graphs is not new. Now, instead of learning a vector representation for the formulas directly, a vector representation of the graphs is learned. The downstream architectures are also more or less what has been done in previous work. It's a nice idea for premise selection that performs better than the SOTA by combining existing ideas (graph representation of formulas + graph vector representation learning). Since there's nothing really new here I would have liked more experimental results and a comparison of different graph embedding methods (many of these are cited by the authors).
Minor comments:
Line 154: "To initialize the embedding for each node, we use the one-hot vector that represents the name of the node." You probably mean that you initialize the embeddings randomly?


End-to-End Differentiable Proving
We introduce neural networks for end-to-end differentiable proving of queries to knowledge bases by operating on dense vector representations of symbols. These neural networks are constructed recursively by taking inspiration from the backward chaining algorithm as used in Prolog. Specifically, we replace symbolic unification with a differentiable computation on vector representations of symbols using a radial basis function kernel, thereby combining symbolic reasoning with learning subsymbolic vector representations. By using gradient descent, the resulting neural network can be trained to infer facts from a given incomplete knowledge base. It learns to (i) place representations of similar symbols in close proximity in a vector space, (ii) make use of such similarities to prove queries, (iii) induce logical rules, and (iv) use provided and induced logical rules for multi-hop reasoning. We demonstrate that this architecture outperforms ComplEx, a state-of-the-art neural link prediction model, on three out of four benchmark knowledge bases while at the same time inducing interpretable function-free first-order logic rules.

Summary of paper
----------------
The paper presents a novel class of models, termed Neural Theorem Provers (NTPs), for automated knowledge base completion and automated theorem proving, using a deep neural network architecture. The recursive construction of the network is inspired by the backward chaining algorithm typically employed in logic programming (i.e., using the basic operations unification, conjunction and disjunction). Instead of directly operating on symbols, the neural network is employed to learn subsymbolic vector representations of entities and predicates, which are then exploited for assessing the similarity of symbols. Since the proposed architecture is fully differentiable, knowledge base completion can be performed using gradient descent. Thus, following the fundamental philosophy of neural-symbolic systems, the paper aims at combining the advantages of symbolic reasoning with those of subsymbolic inference. The paper briefly describes and discusses possible optimizations of the initial, computationally demanding algorithm (i.e., parallel evaluation of proofs and a truncation heuristic to speed up the gradient calculation), and proposes an interesting and apparently very effective training/regularization procedure based on the joint training of NTP together with a state-of-the-art neural link prediction method called ComplEx. Finally, it is shown that the performance of the proposed method on four benchmark knowledge base completion tasks is competitive against ComplEx. Also, it is argued that in contrast to previous approaches, the proposed algorithm is highly interpretable since it is capable of inductive logic programming, i.e., inducing concrete symbolic, first-order logic rules from parameterized templates. The paper concludes with some reasonable and likely worthwile ideas for future investigation.
Short summary of review
-----------------------
This is a solid paper. It aims at solving an important open problem and proposes a creative and effective solution by combining the efforts and thus aggregating the advantages of two very active lines of research. Particularly the dedication to interpretability of the method is commendable. The manuscript is rigorous and at the same time easy to follow. While there are a few issues concerning the quality and clarity, the overall article clearly has merit to be published at a venue like NIPS, and likely is of interest for several of its sub-communities.
Quality
-------
In general, the paper follows a high quality standard. While not being very theoretical in nature, the few technical details provided appear sound. While the experimental section impressively shows how NTPs excell at multi-hop reasoning and inducing interpretable first-order logic rules, it is not particularly successful at convincing the reader of the superiority of NTPs as compared to the state-of-the-art. This is because there is only a single competitor, ComplEx; employing a broader variety of competing algorithms (e.g. from the extensive pool of statistical relational learning methods) would certainly be commendable. However, this might be acceptable in the light of the contribution/focus of the paper being more on interpretability, rather than on significant performance increase. Another central issue with the paper is that while it prominently claims to have designed a system capable of 'theorem proving', the reader is actually never exposed to any evidence that the architecture at hand has indeed the ability to prove mathematical theorems. While this lack is acknowledged in the last sentence of the paper, the fact that said phrase is used in the first sentence of the abstract and even the title may provoke suspicions of deliberatley exploiting its catchiness for marketing purposes. Related work is discussed in commendable detail, providing a solid overview of the field, and extensive references are provided.
Clarity
-------
Most of the paper is well-organized and written in a clear, concise manner, making it rather easy to follow. Figure 1 is very helpful in understanding the general NTP architecture and the chosen running example (i.e., grandfatherOf) is simple enough to enable quick and intuitive understanding and at the same time expressive enough to convey and illustrate the central ideas of the method (also, the subtle reference to popular culture increases the enjoyability of reading the manuscript). However, the structure of section 3 is not quite reasonable: The paragraph directly following the heading '3 Methods' is only refering to subsections 3.1-3.3 (i.e., the modular building blocks of the architecture), but not to subsections 3.4-3.7, which appear to have been placed there rather arbitrarily. Maybe splitting section 3 into two distinct sections would help (e.g. '3 Architecture' and '4 Training & Inference' or something). Also, it would in my view make more sense to put the section on regularization (i.e., 3.7) immediately after the one on training (i.e., 3.4) since these are highly related and the former builds upon the latter (this would also enhance reading flow since one wouldn't have to scroll up to recall the formula for L_ntp, and to identify that the ntp-term and the complex-term share the same structure). Furthermore, the very short background section only defines terminology and notation, but does not provide any details on the fundamental mechanisms of logic programming (e.g. the backward chaining algorithm) that are required to be able to follow the manuscript. Given the tight page limit at NIPS it is evident that extensive coverage of preliminaries is not feasible, but the paper should at least provide pointers to relevant literature for those not familiar with the matter.
Some further, specific issues:
- it is not described how exactly the artificial neural network is created/constructed; it is stated in section 3.1 that 'The module unify [...] creates a neural network [...].' (l. 105-106), the specifics of which are not quite obvious; it should be detailed in what sense the min operation and the RBF kernel correspond to a neural network, and how the recursive construction of the network is to be understood; a graphical representation may be helpful here
- there appear to be inconsistencies regarding the employed notation of data structures: atoms seem to be represented as lists (of three terms), meaning that rule bodies should be represented as lists of lists; however, in the definition of the AND module (section 3.3), sets (denoted by Euler script) are used instead of lists of lists for the first argument, which is also contradicting the signature of AND ('L is the domain of lists of atoms', l. 153); furthermore, the :: operator doesn't make sense on sets; the quickest fix for this issue would be to just define Euler scripts to denote lists of lists instead of sets
- in the example for the unification module (section 3.1), where did the exponentials from the RBF kernel in the proof success score go?
- in Figure 1, it is not clear why S_33, S_313 and S_323 fail; the UNIFY module can only fail if the arity of the atoms doesn't match, which doesn't appear to be the case here; thus, the computation can only fail due to reaching the maximum proof depth; however, as I see it, in that case further paths beyond (for example) S_33 must be explored first, meaning that so the operation cannot fail at S_33 already; this should be clarified
- it would be helpful to have an inference example, i.e. an example of how the success score of a given statement is calculated, given the computational tree/graph in Figure 1; some kind of visualization of this would be helpful as well
Originality
-----------
As is discussed in detail, the paper is built upon the ideas of extensive prior work, and its main contribution essentially boils down to effectively combining two previously independent lines of research: 1) neural-symbolic systems for reasoning in knowledge bases, e.g. as developed in the field of statistical relational learning, and 2) the lately emerging end-to-end differentiable neural network architectures aiming at mimicking some of the capabilities of modern computers. Thus, the paper does not really provide any fundamental novel insight, but rather takes inspiration from said fields and symbiotically combines them in a novel and seemingly effective manner, succeeding at exploiting the advantages of both classes of approaches. The first line of work, i.e. neural-symbolic systems, aims at combining the benefits of subsymbolic similarity-based reasoning (and thus generalization) ability of neural link prediction / statistical relational learning with the symbolic multi-hop reasoning ability of automated theorem provers / logic programming systems. However, previous neural-symbolic approaches are either a) not trainable end-to-end, b) not interpretable and/or c) not capable of incorporating prior domain-specific knowledge. This is where the second line of work, i.e. fully differentiable computing neural network architectures, comes into play, enabling NTPs to handle all the features a), b) and c). Another contribution is the proposal of the recursive construction of the neural network in terms of modules (for unification, conjunction and disjunction operations), similar to dynamic neural module networks, and inspired by the backward chaining algorithm used for example in Prolog.
Significance
------------
The creation and modeling of large knowledge bases/graphs is becoming increasingly important due to their potential support of practical tasks such as information retrieval, question answering etc. Recent efforts in distilling humanities knowledge into large-scale triple-based knowledge graphs (such as Freebase, YAGO, Google KG) are still challenged by difficulties in achieving high amounts of completeness, which is a natural product of the tremendous volume of true facts. Since hand-engineering these systems is clearly not feasible, methods capable of automatically augmenting knowledge bases/graphs based on logical reasoning and statistical inference are highly warranted. The method proposed in this paper has the potential to provide valuable insights - both for scientists as well as for practitioners - into the non-trivial task of knowledge base completion. Especially the ability of NTPs to induce human-readable, logical rules, setting it apart from the majority of previous work, may open up tremendous possibilities, particularly in the light of the recently emerging, significant interest in interpretability in AI / machine learning systems (e.g. as desired by end users, policy makers / governmental bodies and AI critics).
Minor issues
------------
- the explanation of the four rules of the AND module (l. 157-162) is referring to 'line(s)' multiple times; however, the rules aren't formatted in distinct lines, so the formulation is a bit confusing; rather use 'rule(s)' instead of 'line(s)'
- the last sentence of the first paragraph of section 3.4 (i.e., l. 176-177) is gramatically flawed
- in l. 199, it should read 'predicates with indices r, s and t, respectively'


FALKON: An Optimal Large Scale Kernel Method
Kernel methods provide a principled way to perform non linear, nonparametric learning. They rely on solid functional analytic foundations and enjoy optimal statistical properties. However, at least in their basic form, they have limited applicability in large scale scenarios because of stringent computational requirements in terms of time and especially memory. In this paper, we take a substantial step in scaling up kernel methods, proposing FALKON, a novel algorithm that allows to efficiently process millions of points. FALKON is derived combining several algorithmic principles, namely stochastic subsampling, iterative solvers and preconditioning. Our theoretical analysis shows that optimal statistical accuracy is achieved requiring essentially O(n) memory and O(n √ n) time. An extensive experimental analysis on large scale datasets shows that, even with a single machine, FALKON outperforms previous state of the art solutions, which exploit parallel/distributed architectures.

Comments:
- line 19. "... the a variety of ..." -> "... a variety of ..."
- line 25. "... show that there are large class..." -> "... show that there is a large class..."
- line 113. "... that in this case t=sqrt(n)log(n) are needed to achieve..." I guess that the word "iterations" is missing? The same concern is with line 114
- Table 1. The last but one line in the table. There is a problem with a reference for "Nystrom + sgd"
- line 195. "Intuitevely, these conditions measure the size of H, if gamma is small then H is small rates are faster, and the regularity of f, if r is big f is
regular and rates are faster." -> "Intuitevely, these conditions measure the size of H (if gamma is small then H is small and rates are faster) and the regularity of f (if r is big f is regular and rates are faster)."?
- line 204. There should be a space between "O(1/n)." and "By selecting"
Conclusion:
1) In FALCON we need to use the number of inducing points M ~ sqrt(n). Thus in general we are limited in the number of inducing points we can use if n >> 1.
Gaussian Process regression is a probabilistic counterpart of the kernel ridge regression and provides the same regression function, as well as uncertainty quantification, not accessible in case of KRR. Sparse Gaussian Processes (in fact this is the Nystrom approximation), are used to process large samples. Recently, they propose to approximate not the GP model (using Nystrom approximation), but its posterior by optimizing evidence lower bound (ELBO), see
https://www.prowler.io/sparse-gps-approximate-the-posterior-not-the-model/
Thus, by considering inducing points as latent variables, we can optimize ELBO to tune their values. By restricting the family of variational distributions and limiting inducing points to belong to some tensor (see applications of tensor methods for GP e.g. in "A. G. Wilson and H. Nickisch. Kernel interpolation for scalable structured gaussian processes // In International Conference on Machine Learning, pages 1775–1784, 2015", "E. Burnaev, M. Belyaev, E. Kapushev. Computationally efficient algorithm for Gaussian Processes based regression in case of structured samples // Computational Mathematics and Mathematical Physics, 2016, Vol. 56, No. 4, pp. 499–513, 2016") we can use tensor algebra to efficiently perform SGD optimization of ELBO even in case when M ~ n and so we can better approximate the posterior. Could the authors comment on this approach to kernel regression learning? What connections does ELBO optimization by SGD wrt inducing points may have with the proposed learning scheme?
2) The topic of the paper is important. The paper is well written and contains not only efficient algorithm, but also useful theoretical results. Do the authors have available programming code with results of experiments? It could be useful to specify a link to github with this code.


Clustering with Noisy Queries
In this paper, we provide a rigorous theoretical study of clustering with noisy queries. Given a set of n elements, our goal is to recover the true clustering by asking minimum number of pairwise queries to an oracle. Oracle can answer queries of the form "do elements u and v belong to the same cluster?"-the queries can be asked interactively (adaptive queries), or non-adaptively up-front, but its answer can be erroneous with probability p. In this paper, we provide the first information theoretic lower bound on the number of queries for clustering with noisy oracle in both situations. We design novel algorithms that closely match this query complexity lower bound, even when the number of clusters is unknown. Moreover, we design computationally efficient algorithms both for the adaptive and non-adaptive settings. The problem captures/generalizes multiple application scenarios. It is directly motivated by the growing body of work that use crowdsourcing for entity resolution, a fundamental and challenging data mining task aimed to identify all records in a database referring to the same entity. Here crowd represents the noisy oracle, and the number of queries directly relates to the cost of crowdsourcing. Another application comes from the problem of sign edge prediction in social network, where social interactions can be both positive and negative, and one must identify the sign of all pair-wise interactions by querying a few pairs. Furthermore, clustering with noisy oracle is intimately connected to correlation clustering, leading to improvement therein. Finally, it introduces a new direction of study in the popular stochastic block model where one has an incomplete stochastic block model matrix to recover the clusters.

NOTE: I am reviewing two papers that appear to be by the same authors and on the same general topic.  The other one is on noiseless queries with additional side information.
This paper gives upper and lower bounds on the query complexity of clustering based on noisy pairwise comparisons.  While I am familiar with some of the related literature, my knowledge of it is far from complete, so it's a little hard to fully judge the value of the contributions.  One limitation is that the main result would be extremely easy to obtain if the noise were independent when sampling the same pair multiple times.  On the other hand, the case where the same noisy answer is always returned is well-motivated in certain applications.
My overall impression of both papers is that the contributions seem to be good (if correct), but the writing could do with a lot of work.  I acknowledge that the contributions are more important than the writing, but there are so many issues that my recommendation is still borderline.  It would have been very useful to have another round of reviews where the suggestions are checked, but I understand this is not allowed for NIPS.
Here are some of my more significant comments:
- Many of the central results are not written precisely.  The order of quantification is Lemma 1 is unclear (e.g., Pr(for all v) vs. for all v, Pr(...)).  The last step in its proof only holds for sufficiently large k, whereas the lemma is stated for arbitrary k.  The appendix is full of theorems/lemmas/claims with the imprecise phrase "with high probability", and in some cases it seems to be crucial to specify the convergence rate (even if it's done in the proof, it should still appear in the formal statement).
- Near the bottom of p3, the runtime and query complexity are not consistent, with the latter exceeding the former in certain regimes like k = n^(1/6) or p very close to 1/2.  I guess this is because certain dependences were omitted in the runtime (which they shouldn't be without further statement).
- The appendices are often hard to follow.  The strange and somewhat unclear phrases "Suppose if possible" and "Let if possible" are used throughout.  Inequalities like the one at the end of Claim 2's proof appear questionable at first (e.g., when p is close to half), and the reader has to go back and find the definitions of c, c' (and check their dependence on p).  There are a lot of statements along the lines of "similarly to [something earlier]", which I trusted but wasn't certain of.
- The paper is FULL of grammar mistakes, typos, and strange wording.  Things like capitalization, plurals, spacing, and articles (a/an/the) are often wrong. Commas are consistently mis-used (e.g., they usually should not appear after statements like "let" / "we have" / "this means" / "assume that" / "is", and very rarely need to be used just before an equation).  Sentences shouldn't start with And/Or.  There are typos like "entrees", "once crucial".  The word "else" is used strangely (often "otherwise" would be better).  This list of issues is far from complete -- regardless of the decision, I strongly urge the authors to proof-read the paper with as much care as possible.
Here are some comments of a technical nature:
- The paper is so dense that I couldn't check the upper bound proof in as much detail as ideal.  I was reasonably convinced of the proof of the lower bound.
- In Section 2, I don't see why the first "balanced" condition is useful.  Even with a given maximum cluster size, the exact same problem as that described in the first few sentences can still occur, making the problem ill-posed.  I think you should only keep the condition on the minimum cluster size.
- The numerical section does not seem to be too valuable, mainly because it does not compare against any existing works or simple baselines.  Are your methods really the only ones that can be considered/implemented?
Some less significant comments:
- The second half of the abstract can be considerably shortened
- The algorithm descriptions are very wordy.  It might have been useful to accompany them with figures and/or an "Algorithm" environment.
- Sometimes extra citations are needed (e.g., runtime of heaviest weight subgraph, Turan's theorem, and maybe previous use of the TV->KL technique on p5 like Auer et al 1998 bandit paper)
- I would remove "initiate [a rigorous theoretical study]" from the abstract/intro.  Given that there are so many related works, I don't think this is suitable.  It is even less suitable to simultaneously say it in two different papers.
- In the formal problem statement "find Q subset of V x V" makes it sound like the algorithm is non-adaptive.
- The sentence "which might also be the aggregated..." is confusing and should be removed or reworded
- Many brackets are too small throughout the appendix
- In the Unknown k algorithm, l (which should be \ell?) is never updated, so k=2l always assigns the same value
- I would avoid phrases like "information theoretically optimal algorithm", which makes it sound like it is THE algorithm that minimizes the error probability.
[POST-AUTHOR FEEDBACK COMMENTS]
The authors clarified a few things in the responses, and I have updated the recommendation to acceptance.  I still hope that a very careful revision is done for the final version considering the above comments.  I can see that I was the only one of the 3 reviewers to have significant comments about the writing, but I am sure that some of these issues would make a difference to a lot more NIPS readers, particularly those coming from different academic backgrounds to the authors.
Some specific comments:
- The authors have the wrong idea if they feel that "issues like capitalization and misplaced commas" were deciding factors in the review.  These were mentioned at the end as something that should be revised, not as a reason for the borderline recommendation.
- I am still confused about the "balanced" condition and minimum/maximum cluster size.  My understanding is that if you only assume a bound on the maximum size, then the problem is in general ill-posed, by your example (definitions of C_1 and C_2) on lines 149-150.  If you only assume a bound on the minimum size, everything looks fine.  You could also assume both simultaneously, but this doesn't seem to be what you are doing (since on line 155 you write "either of" rather than "both of").   Consider re-wording/explaining more as needed.
- Please revise/re-word the runtime and query complexity on p3 -- if the stated runtime only holds for p = constant, then you should add something like "when p is constant".  I think cases like k = n^(1/6) (where samples seems to exceed runtime) might still look strange to some readers, but since there is no true contradiction, I don't insist on anything.
- It is important that theorems and lemmas are stated precisely (e.g., all assumptions are included like k being large or certain quantities behaving as constants, and convergence rates of probabilities are specified if they play an important role in the later analysis)
- Although it is not a major point, I will mention that I don't recall seeing "Suppose if possible" or "Let if possible" despite reading mathematical papers from many domains.  The latter in particular sounds very strange.  Even for the former, at least for me, a re-wording to something like "Suppose by way of contradiction that..." would be clearer.


Question Asking as Program Generation
A hallmark of human intelligence is the ability to ask rich, creative, and revealing questions. Here we introduce a cognitive model capable of constructing humanlike questions. Our approach treats questions as formal programs that, when executed on the state of the world, output an answer. The model specifies a probability distribution over a complex, compositional space of programs, favoring concise programs that help the agent learn in the current context. We evaluate our approach by modeling the types of open-ended questions generated by humans who were attempting to learn about an ambiguous situation in a game. We find that our model predicts what questions people will ask, and can creatively produce novel questions that were not present in the training set. In addition, we compare a number of model variants, finding that both question informativeness and complexity are important for producing human-like questions.

The authors examine human question asking where answers are K-ary. They define by hand a PCFG for a “battleship” domain, where there are hidden colored shapes in a partially observable grid (i.e., some tiles are revealed as containing part of a ship of a specific color or being empty). The task of the agent is to ask a question with a single word answer that provides as much information about the state of the board. The PCFG served as a prior over questions, which were defined as statements in lambda calculus. Question goodness was defined as a linear function of its informativeness (expected information gain), its complexity (in terms of its length or negative log probability under of it being generated by the PCFG with a uniform distribution over rewrite rules), and “answer type” (e.g., whether it provides a true/false or color as an answer). The human data came from a previously published paper and was hand coded into their grammar. The authors compared lesioned versions of the model via leave one out cross-validation and found that all factors provided a non-negligible contribution (although complexity was clearly the most important). Finally, they presented a series of novel questions.
Question-asking is an important domain within linguistics and hypothesis testing. The authors provide an innovative perspective unifying modern linguistic and cognitive theory to examine how people ask questions, which has the potential to foster novel research in the active learning literature with models asking more sophisticated questions. The paper is extremely clear and well-written. The model is appropriately evaluated from both quantitative and qualitative perspectives.
From the perspective of a computational cognitive scientist, this paper advances the field of self-directed learning. Previous work found that question informativeness was a poor predictor of human question asking. This followup paper resolves part of why this is the case (it is secondary to complexity and may in part be due to the size of possible questions available to the model). Thus, the paper contributes to the cognitive science literature, which may be sufficient to justify its publication in NIPS.
However, there are serious limitations to their approach, which I can see limiting its interest to the more general NIPS audience. Although it may be possible to automate the human question text to statements in some semantic logic, it is unclear how the approach could scale or extend to domains of interest to the broader community. The battleship domain is small with only a few different features and their method is already computationally strained. That being said, the ideas are interesting enough (active learning with more rich questions) to be of interest, though it may be that the authors will need to provide a clear scenario of interest to the broader community where their technique is likely to be applicable and provide novel insights.
Minor comments:
1)	It may not be revealing, but I believe you can do a nested model comparison with the LLs of the lesioned models and get tests of statistical significance.
2)	Moving forward, for sequential question asking, there are other models within the linguistics literature of question-asking in formal semantics and pragmatics. A lot of the work comes out of the Questions under Discussion literature. I listed a few references as pointers to this literature.
Ginzburg, J. (1995). Resolving questions, part I. Linguistics and Philosophy, 18(5), 459-527.
Rojas-Esponda, T. (2013). The roadsigns of communication. Proceedings of Semdial (DialDam), 17, 131-139.


Learning Unknown Markov Decision Processes: A Thompson Sampling Approach
We consider the problem of learning an unknown Markov Decision Process (MDP) that is weakly communicating in the infinite horizon setting. We propose a Thompson Sampling-based reinforcement learning algorithm with dynamic episodes (TSDE). At the beginning of each episode, the algorithm generates a sample from the posterior distribution over the unknown model parameters. It then follows the optimal stationary policy for the sampled model for the rest of the episode. The duration of each episode is dynamically determined by two stopping criteria. The first stopping criterion controls the growth rate of episode length. The second stopping criterion happens when the number of visits to any state-action pair is doubled. We establishÕ(HS √ AT ) bounds on expected regret under a Bayesian setting, where S and A are the sizes of the state and action spaces, T is time, and H is the bound of the span. This regret bound matches the best available bound for weakly communicating MDPs. Numerical results show it to perform better than existing algorithms for infinite horizon MDPs.

The paper proposes TSDE, a posterior sampling algorithm for RL in the average reward infinite horizon setting. This algorithm uses dynamic episodes but unlike Lazy-PSRL avoids technical issues by not only terminating an episode when an observation count doubled but also terminating episodes when they become too long. This ensures that the episode length cannot grow faster than linear and ultimately a Bayesian regret bound of O(HS(AT)^.5) is shown.
Posterior sampling methods typically outperform UCB-type algorithms and therefore a posterior sampling algorithm for non-episodic RL with rigorous regret bounds is desirable. This paper proposes such an algorithm, which is of high interest. The paper is well written and overall easy to follow. I have verified all proofs but Theorem 2 and found no mistakes. The presentation is rigorous and the main insights are discussed sufficiently. This paper is therefore a good contribution to the NIPS community.
A comparison against UCRL2 with empirically tightened confidence intervals would have been interesting. An more extended comparison of TSDE for MDPs with increasing size would have also been interesting to empirically determine the dependency of the regret on the problem quantities. This would have given insights on whether we can expect to prove a tighter regret bound (e.g. with \sqrt(S) dependency) without changing the algorithm. Nonetheless, all of this is somewhat out of the scope of this paper.
Minor detail: The condition on the bottom of page 4 should hold for all s.


Cold-Start Reinforcement Learning with Softmax Policy Gradient
Policy-gradient approaches to reinforcement learning have two common and undesirable overhead procedures, namely warm-start training and sample variance reduction. In this paper, we describe a reinforcement learning method based on a softmax value function that requires neither of these procedures. Our method combines the advantages of policy-gradient methods with the efficiency and simplicity of maximum-likelihood approaches. We apply this new cold-start reinforcement learning method in training sequence generation models for structured output prediction problems. Empirical evidence validates this method on automatic summarization and image captioning tasks.

The paper presents a new method for structured output prediction using reinforcement learning. Previous methods used reward
augmented maximum likelihoods or policy gradients. The new method uses a soft-max objective. The authors present a new
inference method that can be used to efficiently evaluate the integral in the objective. In addition, the authors propose
to use additional reward functions which encode prior knowledge (e.g. to avoid word repetitions).
The paper seems to be promising from the sequence learning point of view as it can
produce sequences of considerably higher quality. However, the authors seem to be ignorant to a large part of the
reinforcement learning and policy search literature, please see [1,2] for good reviews of policy search with information-geometric
constraints.
(1) The soft-max objective is common in the reinforcement learning literature. The given objective is the dual of the following
optimisation problem:
J = E_{p}[R] + KL(p||p_\theta).
If we solve that for p we exactly get the sampling distribution q given in the paper. Plugging the solution back in
the objective J yields the dual problem of the objective which is equivalent to equation 7. I think such insights
reveal much of the structure of the optimisation problem. Moreover, please note that he objective J is commonly
used in policy search, see for example the REPS algorithm, KL control, linearly solvable MDPs, trust region policy optimisation etc... There is an interesting
difference though, that commonly, we want to compute p and p_theta is the sampling distribution. In this algorithm, we optimise for
p_theta, i.e., we change the sampling distribution. I think this connections to standard policy search algorithms need to be revealed
in this paper to provide a better understanding what the algorithm actually does.
(2) The RAML algorithm is a special case of expectation-maximisation based policy search while the new algorithm is an
instance of information-theoretic policy search, see [1].
Clarifying these relations should help to improve the paper significantly.
While in general, I like the approach, I have one technical concern regarding the sampling procedure:
The weight w_i for the reward function is either 0 (with probability p_drop) or W. The authors argue that this is
equivalent to using a weight that is given by p_drop * W. However, this is incorrect as
E_p(drop)[exp(log p(z) + I(drop == 0) * W * reward)] is *not* the same as exp(E_p(drop)[log p(z) + I(drop == 0) * W * reward)] =
exp(log p(z) + p_drop * W * reward)]
In fact, the distribution the authors are sampling from is rather a mixture model:
p_drop * p(z) + (1-p_drop) * exp (W * reward) / Z
This is a very different distribution from the one given in the algorithm so I do not think it is sound to use it.
I think this issue need to be addressed before publishing the paper (maybe its also a misunderstanding on my side,
but the authors need to address my concerns and either proof my wrong or suggest a sound sampling method). Using KL trust
regions similar as in the REPS algorithms could for example be a sounder way of finding the weights w_i).
In summary, this is in interesting paper, but I have a concern regarding the soundness of the sampling procedure which needs
to be addressed.
[1] Deisenroth, Neumann, Peters: A survey on policy search for robotics", FNT 2013
[2] Gergely Neu, Anders Jonsson, Vicenç Gómez, A unified view of entropy-regularized Markov decision processes, 2017


Unbiased estimates for linear regression via volume sampling
Given a full rank matrix X with more columns than rows, consider the task of estimating the pseudo inverse X + based on the pseudo inverse of a sampled subset of columns (of size at least the number of rows). We show that this is possible if the subset of columns is chosen proportional to the squared volume spanned by the rows of the chosen submatrix (ie, volume sampling). The resulting estimator is unbiased and surprisingly the covariance of the estimator also has a closed form: It equals a specific factor times X + X + . Pseudo inverse plays an important part in solving the linear least squares problem, where we try to predict a label for each column of X. We assume labels are expensive and we are only given the labels for the small subset of columns we sample from X. Using our methods we show that the weight vector of the solution for the sub problem is an unbiased estimator of the optimal solution for the whole problem based on all column labels. We believe that these new formulas establish a fundamental connection between linear least squares and volume sampling. We use our methods to obtain an algorithm for volume sampling that is faster than state-of-the-art and for obtaining bounds for the total loss of the estimated least-squares solution on all labeled columns.
Let X be a wide full rank matrix with d rows and n columns where n ≥ d. Our goal is to estimate the pseudo inverse X + of X based on the pseudo inverse of a subset of columns. More precisely, we sample a subset S ⊆ {1..n} of s column indices (where s ≥ d). We let X S be the sub-matrix of the s columns indexed by S (See Figure 1). Consider a version of X in which all but the columns of S are zero. This matrix equals XI S where I S is an n-dimensional diagonal matrix with (I S ) ii = 1 if i ∈ S and 0 otherwise.
We assume that the set of s column indices of X is selected proportional to the squared volume spanned by the rows of submatrix X S , i.e. proportional to det(X S X S ) and prove a number of new surprising expectation formulas for this type of volume sampling, such as

I could go either way on this paper, though am slightly positive. The short summary is that the submission gives elegant expectation bounds with non-trivial arguments, but if one wants constant factor approximations (or 1+eps)-approximations), then existing algorithms are faster and read fewer labels. So it's unclear to me if there is a solid application of the results in the paper. In more detail:
On the positive side it's great to see an unbiased estimator of the pseudoinverse by volume sampling, which by linearity gives an unbiased estimator to the least squares solution vector. I haven't seen such a statement before. It's also nice to see an unbiased estimator of the least squares loss function when exactly d samples are taken. The authors use an inductive argument, with determinants canceling in the induction, and Sherman-Morrison. They give a simple proof of Cauchy-Binet as well. The n^2 speedup the authors give for exact volume sampling over previous work is also important.
On the negative side, since their exact volume sampling is nd^2 time for a single volume sample (as opposed to approximate volume sampling which is faster), it seems there are more efficient solutions for these problems, in particular for standard least squares. The authors don't claim improvement for least squares, but rather they consider the case when reading the labels to the examples is expensive and they only want to read a small number s of labels. The authors are right in that leverage score sampling would require reading d log d labels to get any relative error approximation, and although one could use Batson-Spielman-Srivastava on top of this, I think this would still read d log d labels instead of the d labels that volume sampling uses (even though Batson-Spielman-Srivastava would end up with d sampled columns, to choose these d columns as far as I know would need to read at least d log d labels. This would be worth discussing / fleshing out). But, the authors only get a d-approximation by doing this and need to take d independent volume samples, or d^2 total samples, to get a constant factor approximation to regression. On the other hand, leverage score sampling with d log d samples would get a constant factor approximation.
So regarding applications, if one asks the question - with any amount of time complexity, how can I read as few labels as possible to get an O(d)-approximation, then I think the authors give the first algorithm for this problem. However if one wants a constant approximation or 1+eps-approximation, previous work seems to give faster algorithms and fewer samples.
It would be good if the authors compare their guarantees / running times to others for estimating the pseudoinverse and optimal regression vector, such as the recent ones in:
Price, Song, Woodruff, Fast Regression with an $\ell_\infty$ Guarantee, ICALP, 2017
and the earlier ones for estimating the regression solution vector in:
Sarlos, Improved Approximation Algorithms for Large Matrices via Random Projections, FOCS, 2006
(the running time should be combined with more modern input sparsity time subspace embeddings)
I have read the rebuttal and am still positive on the paper.


An Error Detection and Correction Framework for Connectomics
We define and study error detection and correction tasks that are useful for 3D reconstruction of neurons from electron microscopic imagery, and for image segmentation more generally. Both tasks take as input the raw image and a binary mask representing a candidate object. For the error detection task, the desired output is a map of split and merge errors in the object. For the error correction task, the desired output is the true object. We call this object mask pruning, because the candidate object mask is assumed to be a superset of the true object. We train multiscale 3D convolutional networks to perform both tasks. We find that the error-detecting net can achieve high accuracy. The accuracy of the error-correcting net is enhanced if its input object mask is "advice" (union of erroneous objects) from the error-detecting net.

The paper tackles the important, but relatively unexplored, area of error detection and targeted error correction in connectomics. There is a clear need for this type of methods in the field, primarily in order to optimize the allocation of human proofreading time. The authors make the observation that to a human segmentation errors are much easier to detect than to correct, and propose to automate this process.
Specifically, they propose to build an error detector module in the form a multi-scale 3d CNN, taking as input a binary object mask and predicting whether it is equal to an object in the ground truth segmentation. At inference time, the network is applied on overlapping windows distributed over a grid to identify and localize errors in the segmentation.
The paper also proposes an error correction module -- a 3d CNN reconstructing the object containing the central pixel, similarly to flood-filling networks (FFNs), which the authors cite as related work. Instead of predicting a binary mask like in FFNs, The authors propose to predict a k-dimensional vector for each point of the output, so voxels of the same object have a similar vector, and different objects have not. This vector field is then transformed into a binary mask with an exponential transform. The stated goal of this approach is to soften the dependency on the precise location of the central object. The authors should consider including some additional information about why they chose this particular form of the transform to generate the binary mask, and whether other forms were considered; what value of k was used in the experiments, as well as any experimental data showing that this approach indeed improves the results compared to a direct binary encoding of the output.
The paper is missing information on which losses were used to train the network, which seems particularly important for the error correction module where the k-dimensional vectors are arbitrary and presumably cannot be fully specified based on the training data.
In section 5, the confidence threshold should be explained in more detail -- how was this confidence computed, and what threshold was used? The text also states that the termination condition was an error-free state as predicted by the error detector, or two corrections per iteration. Why was this particular condition used? Would applying the network more than two times result in a better segmentation? Would such a process converge to a stationary state of the segmentation?
It should also be explained explicitly, whether the error corrector deals with both split and merge errors, or only split errors. If merge errors are handled, it should be mentioned what happens in case they are at the supervoxel graph level, and within the supervoxels itself.
In section 6.3, the parameters of the Adam optimizer should be explicitly mentioned.
The paper presents a detailed analysis of performance of the presented systems. The authors should be commended for computing per-object VI scores to make the metrics easier to understand.
Technical comments about the text:
- Fig. 1 is hard to understand, and it is unclear what the nodes represent. The caption mentions horizontal layers, but it does not look like the data flows through the network vertically.
- Line 48 states that the "overhead of agglomeration prohibits the use of deep learning". This does not seem to be the case, as deep nets have been used for agglomeration, see e.g. https://papers.nips.cc/paper/6595-combinatorial-energy-learning-for-image-segmentation.pdf for a recent example.
- Appendix A, Table 4: Why are layer IDs not consecutive, and if this is intentional, what do the IDs mean?


SVD-Softmax: Fast Softmax Approximation on Large Vocabulary Neural Networks
We propose a fast approximation method of a softmax function with a very large vocabulary using singular value decomposition (SVD). SVD-softmax targets fast and accurate probability estimation of the topmost probable words during inference of neural network language models. The proposed method transforms the weight matrix used in the calculation of the output vector by using SVD. The approximate probability of each word can be estimated with only a small part of the weight matrix by using a few large singular values and the corresponding elements for most of the words. We applied the technique to language modeling and neural machine translation and present a guideline for good approximation. The algorithm requires only approximately 20% of arithmetic operations for an 800K vocabulary case and shows more than a three-fold speedup on a GPU.

This paper proposes an efficient way to approximate the softmax computation for large vocabulary applications. The idea is to decompose the output matrix with singular value decomposition. Then, by selecting the most important singular values you select the most probable words and also compute the partition function for a limited amount of words. These are supposed to contribute for the most part of the sum. For the remaining words, their contributions to the partition function is only approximated.
The idea is nice and the experimental results show a significant inference speed up, along with similar performance.
My main concern is that sometimes the paper is poorly written. Just one example: the authors use the term "logit" to denote the exponential of a dot product. This is not correct. The logit function is the inverse of the sigmoidal "logistic" function. This choice of term is therefore incorrect and misleading.
More examples are given below. However the idea is both simple and nice.
- Page 2, line 73: Moreover, self normalized models requires a huge training time.
- Page 3, line 93: It is not "approximately". It's the same. The softmax function outputs the probability distribution over the whole vocabulary.
- Line 102: reword this sentence
- Line 114: for obtaining accurate results -> to obtain accurate results
- Exactly instead of correctly in line 117
Page 5: Why you didn't use the full softmax on the wiki data ? The vocabulary is small and the computational cost is really cheap!
Page 6: The script to compute the BLEU score is included in the Moses released, but it's not the Moses toolkit. However, which one did you use ?
To have a stronger baseline you could use bpe to decompose words. Moreover, it could be more meaningful to have a larger vocabulary. For instance if you translate from English to German, the output vocabulary is really larger, even if you use bpe. I understand the goal is to show that your svd softmax performs as well as the full softmax with a huge gain in time. However, this gain of time could benefit to the NMT system by allowing a longer training in a state of the art setup with the svd softmax.


High-Order Attention Models for Visual Question Answering
The quest for algorithms that enable cognitive abilities is an important part of machine learning. A common trait in many recently investigated cognitive-like tasks is that they take into account different data modalities, such as visual and textual input. In this paper we propose a novel and generally applicable form of attention mechanism that learns high-order correlations between various data modalities. We show that high-order correlations effectively direct the appropriate attention to the relevant elements in the different data modalities that are required to solve the joint task. We demonstrate the effectiveness of our high-order attention mechanism on the task of visual question answering (VQA), where we achieve state-of-the-art performance on the standard VQA dataset.

The paper presents a novel method to model attention on the image and the question for the task of Visual Question Answering (VQA). The main novelty lies in proposing a generic formulation of attention for multiple modalities – a combination of unary, pairwise and ternary potentials for 3 modalities (image, question and answer) in VQA. The proposed approach is evaluated on the VQA dataset and outperforms existing models when ternary potentials are used.
Strengths:
1.	A generic formulation of attention for multimodal problems that is not specific to a particular task is useful because especially if it can be shown to generalize to more than one task.
2.	The ablation study showing the performance without ternary potential is useful to understand how much ternary potential helps.
3.	The paper is clearly written.
Weaknesses:
1.	The approach mentions attention over 3 modalities – image, question and answer. However, it is not clear what attention over answers mean because most of the answers are single words and even if they are multiword, they are treated as single word. The paper does not present any visualizations for attention over answers. So, I would like the authors to clarify this.
2.	From the results in table 1, it seems that the main improvement in the proposed model is coming from the ternary potential. Without the ternary potential, the proposed model is not outperforming the existing models for 2 modalities setup (except HieCoAtt). So, I would like the authors to throw light into this.
3.	Since ternary potential seems to be the main factor in the performance improvement of the proposed model, I would like the authors to compare the proposed model with existing models where answers are also used as inputs such as Revisiting Visual Question Answering Baselines (Jabri et al., ECCV16).
4.	The paper lacks any discussion on failure cases of the proposed model. It would be insightful to look into the failure modes so that future research can be guided accordingly.
5.	Other errors/typos:
a.	L38: mechanism -> mechanisms
b.	L237 mentions that the evaluation is on validation set. However Table 1 reports numbers on the test-dev and test-std sets?
Post-rebuttal comments:
Although the authors' response to the concern of "Proposed model not outperforming existing models for 2 modalities" does not sound satisfactory to me due to lack of quantitative evidence, I would like to recommend acceptance because of the generic attention framework for multiple modalities being proposed in the paper and quantitative results of 3-modality attention outperforming SOTA. The quantitative evaluation of the proposed model's attention maps against human attention maps (reported in the rebuttal) also looks good and suggests that the attention maps are more correlation with human maps' than existing models. Although, we don't know what this correlation value is for SOTA models such as MCB, I think it is still significantly better than that for HieCoAtt.
I have a question about one of the responses from the authors --
> Authors' response -- “MCB vs. MCT”: MCT is a generic extension of MCB for n-modalities. Specifically for VQA the 3-MCT setup yields 68.4% on test-dev where 2-layer MCB yields 69.4%. We tested other combinations of more than 2-modalities MCB and found them to yield inferior results.
Are the numbers swapped here? 69.4% should be for 3-MCT, right? Also, the MCB overall performance in table 1 is 68.6%. So, not sure which number the authors are referring to when they report 68.4%.


Active Bias: Training More Accurate Neural Networks by Emphasizing High Variance Samples
Self-paced learning and hard example mining re-weight training instances to improve learning accuracy. This paper presents two improved alternatives based on lightweight estimates of sample uncertainty in stochastic gradient descent (SGD): the variance in predicted probability of the correct class across iterations of minibatch SGD, and the proximity of the correct class probability to the decision threshold. Extensive experimental results on six datasets show that our methods reliably improve accuracy in various network architectures, including additional gains on top of other popular training techniques, such as residual learning, momentum, ADAM, batch normalization, dropout, and distillation.
other times, as illustrated in Figure 1. This preference is consistent with common variance reduction strategies in active learning [43].
Previous studies suggest that finding informative unlabeled samples to label is related to selecting already-labeled samples to optimize the model parameters [14]. As reported in the previous studies [42,6], models can sometimes achieve lower generalization error after being trained with only a subset of actively selected training data. In other words, focusing on informative samples can be beneficial even when all labels are available.
We propose two lightweight methods that actively emphasize uncertain samples to improve minibatch SGD for classification. One method measures the variance of prediction probabilities, while the other one estimates the closeness between the prediction probabilities and the decision threshold. For logistic regression, both methods can be proven to reduce the uncertainty in the model parameters under reasonable approximations.
We present extensive experiments on CIFAR 10, CIFAR 100, MNIST (image classification), Question Type (sentence classification), CoNLL 2003, and OntoNote 5.0 (Named entity Recognition), as well as on different architectures, including multiple class logistic regression, fully-connected networks, convolutional neural networks (CNNs) [26], and residual networks [16]. The results show that active bias makes neural networks more robust without prior knowledge of noise, and reduces the generalization error by 1% -18% even on training sets having few (if any) annotation errors.

The paper proposes a novel way of sampling (or weighing) data-points during training of neural networks. The idea is, that one would like to sample data-point more often which could be potentially classified well but are hard to learn (in contrast to outliers or wrongly labeled ones). To `find' them the authors propose two (four if split into sampling and weighing) schemes: The first one (SGD-*PV) proposes to weigh data-points according to the variance of the predictive probability of the true label plus its confidence interval under the assumption that the prediction probability is Gaussian distributed. The second one (SGD-*TC), as far as I understand, encodes if the probability of choosing the correct label given past prediction probabilities is close to the decision threshold.
The statistics needed (means and variances of p) can be computed on-the-fly during a burn-in phase of the optimizer; they can be obtained from a forward pass of the network which is computed anyways. The authors test their methods on various architectures and dataset and compare to a range of baselines.
The paper motivates the problem well and the introduction is clearly written (Figure 1 is very helpful, too). The technical sections 3.2 (up to the logReg example) and 3.3 are very dense;  moving table 1 (and or 2) to the appendix and elaborating a bit more (especially on 3.3) would be very helpful.
Major Questions:
- I do not understand how SGD-*PV prefers uncertain example over hard ones, i.e. why should hard samples have lower predictive variance than uncertain ones? Hard ones might be labeled incorrectly, but the probability might still fluctuate.
- In 3.3 P(i|H, S_e, D) is largest around \bar{p}=0.5. Does that mean the method implicitly assumes 2 output classes only? If not, I would like to know how P encodes closeness to a threshold for multiple classes.
- In 3.2 the authors assume that p is Gaussian distributed. Is this meant across occurrences in the mini-batch? I would also like the authors to comment a bit more on this assumption, also since clearly p is not Gaussian distributed (it ranges between 0 and 1). Would it be possible to impose this assumption on log(p) instead to get a better fit?
- how many samples do you typically collect for p? as many as epochs? is this the reason you need a burn in time? or is there another reason why it is good to first burn-in and then start the proposed sampling/weighing schemes?
Further comments:
- since the authors propose a practical algorithm it would be nice to include a sentence about memory requirement or other non-trivial implementation details (in case there are some).
- As mentioned above, I encourage the authors to expand the technical section (3.2 before the logReg example and especially 3.3) since it is very dense.
- It is very nice that the authors compare against many baselines
- It is nice that the authors are transparent about how the learning rates etc. are chosen.
(*after rebuttal*: The rebuttal clarified most of my questions. If the paper gets accepted I encourage the authors to also clarity the points in the paper (to eases the reading process) as they promise in the rebuttal, especially i) the presentation above the logistic regression example in Section 3.2 (maybe a pointer that the assumptions will be discussed below). At that point they are quite ad-hoc and it is hard (and not beneficial for the flow of thoughts) to trace back later ii) closeness to threshold: recap shortly what \bar{p} was (it took me a while to find it again) iii) the meaning of the burn-in phase. iv) A sentence about implementation which points to the Appendix. In good faith that this will be done I increased the score.)


Estimation of the covariance structure of heavy-tailed distributions
We propose and analyze a new estimator of the covariance matrix that admits strong theoretical guarantees under weak assumptions on the underlying distribution, such as existence of moments of only low order. While estimation of covariance matrices corresponding to sub-Gaussian distributions is well-understood, much less in known in the case of heavy-tailed data. As K. Balasubramanian and M. Yuan write 1 , "data from real-world experiments oftentimes tend to be corrupted with outliers and/or exhibit heavy tails. In such cases, it is not clear that those covariance matrix estimators .. remain optimal" and "..what are the other possible strategies to deal with heavy tailed distributions warrant further studies." We make a step towards answering this question and prove tight deviation inequalities for the proposed estimator that depend only on the parameters controlling the "intrinsic dimension" associated to the covariance matrix (as opposed to the dimension of the ambient space); in particular, our results are applicable in the case of highdimensional observations.

The paper provides a truncation estimator for the covariance matrix. The idea is simple yet useful, enjoying very good theoretical properties. I strongly welcome this contribution to the robust statistics literature.
I have only several very minor comments/suggestions.
(1) The authors should discuss relevance to covariance estimation under elliptical distributions, like ECA proposed by Han and Liu (JASA, 2017+) (which is also a robust estimator of the covariance when the eigenvalues are bounded; see, e.g., https://arxiv.org/pdf/1705.06427.pdf). See below for more details.
(2) Can we derive estimation accuracy under other norms (such as L_infty norm and restricted spectral norm, defined in the above ECA paper)? These alternative norms are crucial in high dimensional statistics (e.g., graphical model estimation and sparse PCA). See the ECA paper for an example.
(3) Centeredness is not crucial in covariance matrix estimation as long as the data are iid: We could use the pairwise difference X_i-X_j, which is centered no matter whether X_i is centered or not. This alternative should also be more efficient than the proposed one since it avoids estimating the mean. See again the ECA paper for discussions. Relative comments have also been put in Chen, Gao, and Ren (AoS, 2017+), which the authors cited.


Fast amortized inference of neural activity from calcium imaging data with variational autoencoders
Calcium imaging permits optical measurement of neural activity. Since intracellular calcium concentration is an indirect measurement of neural activity, computational tools are necessary to infer the true underlying spiking activity from fluorescence measurements. Bayesian model inversion can be used to solve this problem, but typically requires either computationally expensive MCMC sampling, or faster but approximate maximum-a-posteriori optimization. Here, we introduce a flexible algorithmic framework for fast, efficient and accurate extraction of neural spikes from imaging data. Using the framework of variational autoencoders, we propose to amortize inference by training a deep neural network to perform model inversion efficiently. The recognition network is trained to produce samples from the posterior distribution over spike trains. Once trained, performing inference amounts to a fast single forward pass through the network, without the need for iterative optimization or sampling. We show that amortization can be applied flexibly to a wide range of nonlinear generative models and significantly improves upon the state of the art in computation time, while achieving competitive accuracy. Our framework is also able to represent posterior distributions over spike-trains. We demonstrate the generality of our method by proposing the first probabilistic approach for separating backpropagating action potentials from putative synaptic inputs in calcium imaging of dendritic spines.

This paper presents a variational auto-encoder-style method for extracting spike-trains from two-photon fluorescence microscopy time traces. This is a task that scientists are interested in for the purposes of interpreting one- two- or three- photon microscopy data, which are becoming a staple method in neuroscience. The authors introduce 3 different spike-to-fluorescence models and use one of two neural networks to create an decoder to recover the (discritized) spike trains from the fluorescence. Overall, the method does show some promise. That said, my biggest concern with this work lies with the characterization of the method. Deep-networks just like any other algorithm, need to have an understanding of their trade-offs and limitations. It is especially important in an applications that will be used to inform scientific conclusion to understand aspects of the method such as the the bias and sensitivity to model mis-match. Unfortunately, I feel that the results section fall short of such a characterization and additional simulations or theory needs to be established to instill confidence in this method. I have detailed below specific comments that I feel the authors should address to improve the manuscript.
Comments:
1) Is this problem even solvable in general? In the linear case (the first, most basic model) it seems that spikes can be inferred, however the nonlinearities used in the following models can make spikes indistinguishable without adequate time between spikes. Is there a limit to the inter-spike times that this method can distinguish in the non-linear case? Is this  a reasonable limit for neural data?
2) In a related note, all the data (and simulations) in this work use 60Hz sampling rates for the fluorescence traces. Much of the fluorescence calcium literature (especially 2-photon imaging, which seems to be becoming dominant) sampling rates as low as 10-15Hz are not unheard of. How does this method perform when the sampling rates fall to 30Hz or 10Hz, when spikes may truly be indistinguishable? Additionally, to have the discrete spike-train output retain a sampling rate on the order of a refractory period, would the proposed networks have an a-symmetry between the input and output rates, or would the network need to be changed to allow for multiple spikes per bins?
3) On line 66, the authors claim to use a "powerful recurrent neural network" model. I assume the authors don't mean 'powerful' in the statistical testing sense, so such embellishment is unnecessary.
4) The authors often claim that the network learned in one sense can generalize well to other settings (e.g. line 61). All the testing and training were done with i.i.d. spikes constant rates chosen from one of three values. I did not see in the results any testing from the network trained with this constrained data-set on fluorescence traces generated from more complex spike-rates. Do the authors have evidence of this generalization, or is this a statement provided based on the use of VAEs in other contexts? If it is the former, the authors should provide these results to validate their claim. If it is the latter, the authors should verify that this generalization holds true for their application, as properties of VAEs do not necessarily translate across domains.
5) The authors present three different models with various levels of complexity, however I do not see any results as to how the proposed network inference methodology changes with respect to each of these networks. It would be informative to see how the results, say on a single V1 trace, change with respect to the model used to train the network.
6) Related to the previous point, sometimes it is unclear which model is being used in which networks for the various results (e.g. what model is used for the V1 results?).
7) In Figure 3, the reconstruction base on the true spike train using (was this using the SCF or SCDF model?) is not as accurate as the results of the spikes inferred using the network trained on that model. How is this possible? This discrepancy warrants at least some discussion. Clearly there is a model-mismatch of some kind. Is the network's ability to achieve better prediction mean that the method has a built-in bias in the estimated spikes that stems from the model mismatch? This seems like an important point that is glossed over. A systematic bias in the inferred spikes can create errors in scientific analyses applied to the resulting spike trains.
8) The work "amortize" is useless in the context of machine learning and only serves to further obfuscate our field further with respect to outside audiences that may seek to use these methods. I very highly recommend omitting this word from the paper.
2) Acronyms should be defined where first used (e.g. in all the sub-sub-section titles for the various models)
3) Most of the figures could do with bigger font sizes


Parameter-Free Online Learning via Model Selection
We introduce an efficient algorithmic framework for model selection in online learning, also known as parameter-free online learning. Departing from previous work, which has focused on highly structured function classes such as nested balls in Hilbert space, we propose a generic meta-algorithm framework that achieves online model selection oracle inequalities under minimal structural assumptions. We give the first computationally efficient parameter-free algorithms that work in arbitrary Banach spaces under mild smoothness assumptions; previous results applied only to Hilbert spaces. We further derive new oracle inequalities for matrix classes, non-nested convex sets, and R d with generic regularizers. Finally, we generalize these results by providing oracle inequalities for arbitrary non-linear classes in the online supervised learning model. These results are all derived through a unified meta-algorithm scheme using a novel "multi-scale" algorithm for prediction with expert advice based on random playout, which may be of independent interest.

The paper introduces a framework for deriving model selection oracle inequalities in the adversarial online learning setting. The framework is based on a new meta-algorithm, called MultiScaleFTPL, which is able to combine base algorithms ("experts"), incurring a regret to the i-th algorithm which is proportional to the range of its losses. This adaptation to the loss range leads to numerous applications of the algorithm in a wide range of settings, in many of which oracle bounds were previously unavailable.
In particular, the authors consider adaptation to the comparator's norm in uniformly convex Banach spaces (which is a novel results, as previously only Hilbert space norms were considered), an algorithm with a bound which holds for any \ell_p norm of the comparator (quite surprising that such a generality in the bound is actually possible), adaptation to the rank for online PCA, adaptation to the norm of positive definite comparator matrix in the Matrix Multipcative Weight setup, combining algorithms in the online supervised learning setup, including online penalized risk minimization and online multiple kernel learning. Many of these results are impressive. The overall contribution is thus very solid.
The central result in the paper is the MultiScaleFTPL algorithm, essentially a "prediction with expert advice" algorithm. The main feature of the algorithm, the reason why it stands out from other previously considered methods, is its adaptation to the loss range: the algorithm guarantees a regret to the expert i (for all i), which scales proportionally to the range of the losses of expert i, rather than to the worst-case range among experts. The algorithms is an unusual version of Follow the Perturbed Leader algorithm, which samples the future and plays with a distribution which minimizes the worst-case (w.r.t. the current gradient) penalized regret.
The technical quality of the paper is high. I verified some of the proofs (not all) in the appendix. I found one confusing aspect in the proof of the main theorem (Thm 1), which I was not able to understand: at the top of page 12 (appendix) what does "choosing p_t to match e_i" mean, given that i is the index over which the supremum is taken? (i.e., it sounds as if p_t matches all all e_i simultaneously...). I would be very to glad to get a feedback from the authors on this issue.
In summary, I really like the new algorithms and numerous impressive novel results the algorithm leads to. I think the paper is definitely worth accepting to the conference.
Minor remarks:
- line 54-55: "all losses ... are all bounded" -> "losses ... are bounded"
- Eq. (3) -- what is w_t^i on the l.h.s.? (i index not explained)
- line 121: "with norm large as" -> "with norm as large as"
- Eq. below line 135: sigma's are denoted differently than in Algorithm 3
- line 187: 2-smooth norm -- this term is not explained (at least give a reference to the definition)
- appendix A.1, last line of page 11: < p_t,g_t >  ->  < p_t, g_t' >
- appendix A.1 (proof of Thm 1): would be good to say what \epsilon is.
- appendix A.1 (proof of Thm 10): domination is defined by saying that w_1 is at least w_2 plus sum of c_1 + c_2, but there is a factor of 2 in front of epsilon sigma_i c_i in the bound. Should I understand it that epsilon is distributed on {-1/2, 1/2}? But then it does not fit to the proof of Thm 1 (top of page 12)
- appendix A.2 (proof of Thm 2): \cal{A} introduced without definition (class of base algorithms?)
-------
The rebuttal clarified my (minor) issues with the proofs. I keep my high score.


Eigenvalue Decay Implies Polynomial-Time Learnability for Neural Networks
We consider the problem of learning function classes computed by neural networks with various activations (e.g. ReLU or Sigmoid), a task believed to be computationally intractable in the worst-case. A major open problem is to understand the minimal assumptions under which these classes admit provably efficient algorithms. In this work we show that a natural distributional assumption corresponding to eigenvalue decay of the Gram matrix yields polynomial-time algorithms in the non-realizable setting for expressive classes of networks (e.g. feed-forward networks of ReLUs). We make no assumptions on the structure of the network or the labels. Given sufficiently-strong eigenvalue decay, we obtain fully-polynomial time algorithms in all the relevant parameters with respect to square-loss. This is the first purely distributional assumption that leads to polynomial-time algorithms for networks of ReLUs. Further, unlike prior distributional assumptions (e.g., the marginal distribution is Gaussian), eigenvalue decay has been observed in practice on common data sets.

This paper considers the problem of square-loss regression for several function classes computed by neural networks. In contrast to what previous hardness results might suggest, they show that those function classes can be improperly learnt under marginal distribution assumptions alone. Specifically, they use previous structural results of approximate RKHS embedding and assume an eigenvalue decay assumption on the empirical gram matrix (namely, it is approximately low-rank). They then apply a recursive Nystrom sampling method to obtain an approximate and compressed version of the gram matrix, which is then used instead in the learning. The error of the resulting regression is then bounded using known compression generalization bounds, hence obtaining
generalization bounds for kernelized regression using Compression Schemes. These in turn are claimed to be tighter than previously used bounds in terms of the required norm regularization (denoted B) in the RKHS. Under the above assumption, this leads to more efficient learning.
Despite the numerous typos and unclear sentences (see below for specific comments), the paper can be well understood for the most part and I found it quite interesting. It non-trivially combines several known results to obtain very nice conclusions about a still not well understood topic.
The main technical contribution is obtaining generalization bounds for kernelized regression using Compression Schemes. As the authors comment, it is indeed interesting how compression schemes can be used to improve generalization bounds for other function classes. One such work is [KSW], showing that compression-based generalization bounds similar to those considered here can be used to obtain a computationally efficient algorithm for computing a Bayes consistent 1-NN classifier in general metric space.
To be more certain about my review please reply to the following:
- It is claimed that the proposed algorithm has fully-polynomial time complexity in all the relevant parameters. Unfortunately, this dependence is not transparent and is implicit in the results, in particular on its dependence on B. It is only claimed in Thm. 1 that the runtime is O(log B). Please also discuss why a linear dependence in B is considered not fully polynomial.
- The dependence on B is Section 7 is hidden, making it very hard to follow as how the new compression bound comes into play. The proof in the supplementary doesn't help much either. As this is the main advantage, it should be much more transparent.
Specific comments:
- Line 38: "fully polynomial-time algorithms are intractable" is self contradicting
- Line 42: "... are necessary for efficient learnability" - I believe you should add "by gradient based algorithms". Otherwise, the next sentence in the paper is a contradiction.
- A better connection to the results in [Zhang] is needed.
- Thm 1.: the role of \eps should be introduced, even if informally.
- Line 62: Paragraph is unclear at first reading. Better to rephrase.
- Line 78: missing "with high probability" at the end of sentence?
- Paragraph starting at line 79 is highly unclear. E.g. "compress the sparsity" makes no sense. "bit complexity" is unclear.
- Definition 1: Unclear to what "with probability 1-\delta" refers to. Is it to property P or the empirical error bound. Missing comma? What is the property P you actually use in your proofs?
- Thm. 2 has many typos. In addition, missing iid assumption. Is there no assumption on the loss function l? More importantly, in the refereed papers, only one direction of the bound is proved, as opposed to the bidirectional (i.e. having absolute value on the lhs of the bound) you give here.
- Definition 2 has many typos.
- Line 74: "Let us say" -> Suppose?; Also eps -> (eps,m)?
- Line 187: "improve the dependence on sqrt(m)". Please add reference.
- The sentence beginning in line 292 is highly unclear.
- Many typos all over.
- The bibliography misses many entries (year, journal, etc.).
References:
[KSW] - Kontorovich, Sabato and Weiss. Nearest-Neighbor Sample Compression: Efficiency, Consistency, Infinite Dimensions. 2017.
[Zhang] - Tong Zhang. Effective dimension and generalization of kernel learning. 2013.


Solving Most Systems of Random Quadratic Equations
This paper deals with finding an n-dimensional solution x to a system of quadratic equations y i = | a i , x | 2 , 1 ≤ i ≤ m, which in general is known to be NP-hard. We put forth a novel procedure, that starts with a weighted maximal correlation initialization obtainable with a few power iterations, followed by successive refinements based on iteratively reweighted gradient-type iterations. The novel techniques distinguish themselves from prior works by the inclusion of a fresh (re)weighting regularization. For certain random measurement models, the proposed procedure returns the true solution x with high probability in time proportional to reading the data {(a i ; y i )} 1≤i≤m , provided that the number m of equations is some constant c > 0 times the number n of unknowns, that is, m ≥ cn. Empirically, the upshots of this contribution are: i) perfect signal recovery in the high-dimensional regime given only an information-theoretic limit number of equations; and, ii) (near-)optimal statistical accuracy in the presence of additive noise. Extensive numerical tests using both synthetic data and real images corroborate its improved signal recovery performance and computational efficiency relative to state-of-the-art approaches.

Update:  I've increased my rating from 6-7 because the authors did a good job addressing my concerns, and the distinction between theoretical and empirical claims seems a bit better upon a fresh read.
One more comment:  Given the empirical nature of the experiments, I think the authors should have compared to state-of-the-art phase retrieval methods that are known to out-perform WF methods (like Feinup's HIO method or Gerchberg–Saxton).  It's likely that such methods would perform at the information theoretic limit as well.
=============================================
This paper presents a (somewhat) new method for solving phase retrieval problems that builds off of other phase retrieval schemes.  The method has 2 phases:
1) Construct an initial guess of the solution using a spectral initialization scheme
2) Refine the guess using a gradient descent method.
The new method is interesting, and empirically the method performs well.  My reservations about the paper stem from the fact that the claims it makes are dramatically over-sold (and possibly untrue, depending on interpretation), and gloss over existing contributions that others have made in this field.   This is a good paper with interesting results.  I rate it as "marginal accept" instead of "accept" because the presentation is at times misleading and ungenerous to other authors (who's work they build upon extensively).
1)  The authors appear to be claiming to be the first method to solve "almost" all quadratic equations.  However, several other methods exists with similar (identical?) theoretical strength.    The authors don't seem to acknowledge this fact until line 266.
2)  In the abstract and intro, the authors make no distinction between empirical and theoretical guarantees.  This paper provides no theoretical guarantees of achieving the optimal number of measurements.  Any observations of this fact are purely empirical.  Empirical claims are fine with me, but the authors should be up-front about the fact that their results are only observed in experiments (the first time I read this paper the abstract and intro led me to believe you have a *proof* of achieving the lower bound, which you do not).
3) Speaking of experiments, how did the authors choose the truncation parameters for the TWF comparison?  Did they tune this parameter for optimal performance?  The parameter should be reported to make experiments repeatable.
4) You claim that recovery takes place in time proportional to MxN.  I'm not sure this is true.  The convex optimization step can definitely be done in NxM time, but what about the (very important) initialization step?  I don't think you can make this claim unless you have a constant upper-bound on the amount of work needed to obtain the leading eigenvector (for example a constant bound on the number of power iterations that does not depend on N or M).  The authors should either justify this claim or remove it.  Or maybe the claim is only based on empirical experiments (in which case this should be made clear)?
   Finally, the authors should make clear that truncated WF flow methods have the same runtime bounds as this method (although I think the truncated WF paper avoids making any claims of linear overall runtime because of the issues mentioned above with the initializer).
Also, you might want to cite the following (optional - I leave this up to the authors):
   - "Phase Transitions of Spectral Initialization for High-Dimensional Nonconvex Estimation"   This paper does a rigorous analysis of spectral initiation methods, and proves tight bounds that don't require constants.  This is relevant because the empirical results you got in Fig1 are predicted by this theory.
   - On line 66:  You mention that convex methods require large storage and computation.  The recent papers "Phase Retrieval Meets Statistical Learning Theory: A Flexible Convex Relaxation" and "PhaseMax: Convex Phase Retrieval via Basis Pursuit" use convexity without the extra storage.  I doubt, though, that these methods can be performed with provably linear time complexity.
   - "Phase Transitions of Spectral Initialization for High-Dimensional Nonconvex Estimation" . This is an alternative formulation for truncated WF that has gotten some attention.  I think it's worth mentioning since this is in the same class of algorithms as the proposed method, and is very closely related.


Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Data
We present a factorized hierarchical variational autoencoder, which learns disentangled and interpretable representations from sequential data without supervision. Specifically, we exploit the multi-scale nature of information in sequential data by formulating it explicitly within a factorized hierarchical graphical model that imposes sequence-dependent priors and sequence-independent priors to different sets of latent variables. The model is evaluated on two speech corpora to demonstrate, qualitatively, its ability to transform speakers or linguistic content by manipulating different sets of latent variables; and quantitatively, its ability to outperform an i-vector baseline for speaker verification and reduce the word error rate by as much as 35% in mismatched train/test scenarios for automatic speech recognition tasks.

Summary
This paper proposes a hierarchical probabilistic generative model for sequential data. The proposed model structure distinguishes between higher-level sequence and lower-level frame latent variables in order to capture multiple scales. A discriminative regularizer is added to the ELBO to encourage diversity across sequence latent variables with the aim of disentangling the latent representations. This work is related to approaches for learning disentangled representations in probabilistic generative models, such as DC-IGN (Kulkarni et al. 2015), InfoGAN (Chen et al. 2016), and Beta-VAE (Higgins et al. 2017). It is also related to hierarchical VAE-style models such as Structured VAEs (Johnson et al. 2016) and the Neural Statistician (Edwards & Storkey 2017) and latent variable models for sequential data including VRNN (Chung et al. 2015), SRNN (Fraccaro et al. 2016) and the hierarchical multiscale RNN of Chung et al. (2017).
Strengths
- Disentanglement in sequential data is not studied sufficiently in current literature
- Seq2Seq model within each frame is sensible
- Paper is overall well-written and clear
Weaknesses
- The proposed hierarchical model makes frame-level independence assumptions
- Discriminative objective is reminiscent of a supervised objective and thus it not clear that the resulting disentanglement of latent representations is in fact unsupervised
- Experiments are somewhat incomplete, both in comparing to baselines and in empirically validating disentanglement
Quality
Two design decisions regarding the graphical model did not make sense to me. First, if z2 is meant to capture sequence-level information, why is it placed inside the frame-level plate? Second, modeling frames independently for a given sequence discards the correlations between frames. Even if this facilitates learning at the frame level, it does not seem to be an entirely appropriate assumption for sequential data. I am wondering if the authors can clarify the motivation behind these choices.
In particular, the first decision seems to necessitate the addition of the discriminative regularizer (equation between ll. 110-111). However such an objective could be naturally incorporated into the model by adding an optional fully observed sequence-level side information variable y^{(i)} which is generated from z2.
For the variational lower bound results, I would suggest adding a comparison to baseline models such as VRNN (Chung et al. 2015) or SRNN (Fraccaro et al. 2016).
The disentanglement properties of the proposed FHVAE seem to be due to the discriminative objective, since the speaker verification error rate drops as alpha increases. However, since in TIMIT the sequences vary primarily on speaker identity (as mentioned in the paper), the discriminative objective is roughly equivalent to a supervised loss in which the goal is to predict speaker identity. In this case, the disentanglement is no longer unsupervised as originally claimed. Adding a speaker verification experiment on a dataset where the sequences vary less based on speaker identity would help clarify the role of the discriminative objective in disentangling the latent representation.
Clarity
The paper is generally well-written and structured clearly. The notation could be improved in a couple of places. In the inference model (equations between ll. 82-83), I would suggest adding a frame superscript to clarify that inference is occurring within each frame, e.g. q_{\phi}(z_2^{(n)} | x^{(n)}) and q_{\phi}(z_1^{(n)} | x^{(n)}, z_2^{(n)}). In addition, in Section 3 it was not immediately clear that a frame is defined to itself be a sub-sequence.
Originality
The paper is missing a related work section and also does not cite several related works, particularly regarding RNN variants with latent variables (Fraccaro et al. 2016; Chung et al. 2017), hierarchical probabilistic generative models (Johnson et al. 2016; Edwards & Storkey 2017) and disentanglement in generative models (Higgins et al. 2017). The proposed graphical model is similar to that of Edwards & Storkey (2017), though the frame-level Seq2Seq makes the proposed method sufficiently original. The study of disentanglement for sequential data is also fairly novel.
Significance
Unsupervised discovery of disentangled latent representations is an important problem. There has been some prior work in learning disentangled representations for image data but to my knowledge not for sequential data. However the proposed discriminative objective sufficiently resembles supervised identity prediction that it is not clear that the proposed method actually addresses the problem.
Chen, Xi, et al. "Infogan: Interpretable representation learning by information maximizing generative adversarial nets." NIPS 2016.
Chung, Junyoung, et al. "A recurrent latent variable model for sequential data." NIPS 2015.
Chung, Junyoung, Sungjin Ahn, and Yoshua Bengio. "Hierarchical multiscale recurrent neural networks." ICLR 2017.
Edwards, Harrison, and Amos Storkey. "Towards a neural statistician." ICLR 2017.
Fraccaro, Marco, et al. "Sequential neural models with stochastic layers." NIPS 2016.
Higgins, Irina, et al. "beta-VAE: Learning basic visual concepts with a constrained variational framework." ICLR 2017.
Johnson, Matthew, et al. "Composing graphical models with neural networks for structured representations and fast inference." NIPS 2016.
Kulkarni, Tejas D., et al. "Deep convolutional inverse graphics network." NIPS 2015.
-----
**Edit:** I have read the authors' rebuttal. The clarification was helpful regarding the importance of the discriminative regularizer in achieving disentanglement. I would recommend emphasizing in the paper (1) the presence of multiple utterances per speaker and (2) the speaker verification results are still good when alpha = 0. I am updating my rating of this paper from 4 to 6.