Generalisation of structural knowledge in the hippocampal-entorhinal system
A central problem to understanding intelligence is the concept of generalisation. This allows previously learnt structure to be exploited to solve tasks in novel situations differing in their particularities. We take inspiration from neuroscience, specifically the hippocampal-entorhinal system known to be important for generalisation. We propose that to generalise structural knowledge, the representations of the structure of the world, i.e. how entities in the world relate to each other, need to be separated from representations of the entities themselves. We show, under these principles, artificial neural networks embedded with hierarchy and fast Hebbian memory, can learn the statistics of memories and generalise structural knowledge. Spatial neuronal representations mirroring those found in the brain emerge, suggesting spatial cognition is an instance of more general organising principles. We further unify many entorhinal cell types as basis functions for constructing transition graphs, and show these representations effectively utilise memories. We experimentally support model assumptions, showing a preserved relationship between entorhinal grid and hippocampal place cells across environments.

It is an interesting study that tackles one of very important questions in computational neuroscience - how generalisation across stimuli and environments is achieved. It is similar to the concept of schemas, which are thought to primarily rely on frontal cortical areas. In this particular case the focus is on entorhinal grid cells and hippocampal place cells, which authors assert code for the environment and conjunction of environment and stimulus respectively. The authors present a computational model that aims to address the question of whether place cells encode a conjunctive outcome of environment and stimulus representations. It is an interesting hypothesis, which if shown convincingly, would be a major breakthrough in neuroscience. However, this study suffers from lack of clarity in methodological descriptions as well as from seemingly arbitrary choices in analyses, which at least in the current version does not convincingly show that the stated hypothesis is supported by data. Both of these aspects should be improved for the paper to be acceptable to NIPS.  More specifically, - it is clear (and supported by data) that grid cells have different frequencies (that go along anatomical gradients); however, for place cells the meaning of frequency is more ambiguous - is there evidence of similar anatomical gradients (based on projections from EC) of place field radii, which may suggest the support of frequencies for place cells as well as the actual hypothesis of conjunctive coding - it's not exactly clear which exactly formulations and equations are used to simulate the model & learning - Bayesian, Hebbian or BPTT - the latter being considerably less biologically realistic. I would suggest to move theoretical background to the appendix and instead present model description and simulation clearly in the main text, including all steps and their justification (including of some seemingly arbitrary choices) - it's not clear how exactly grid cells (and place cells) result from model constraints (or are they more hand-designed?) - considering noisiness of biological place cells, it doesn't seem very reasonable to consider peak activation as their centre - wouldn't it be better to fit a Gaussian and consider its centre? - Why are trials 1 vs 4 selected to evaluate the match between different environments? To avoid bias, it would be a lot better to include all pairwise comparisons of (1,5) vs (2,3,4) - In presenting the "within environment" comparison it would be equally interesting to see if comparing 2 vs 3 vs 4 lead to similar results as 1 vs 5, and if not, why not? - Correlations between measures seem a bit too low to support the claim regarding the relationship between grid and place cells in remapping (even if they are statistically significant). Mindist measure seems considerably smaller between environments than within the same environment (even if marginally significant). - Statistics in 7b seem to indicate a considerable mismatch between the experiment and model prediction - Finally what is meant by "anonymous authors" for ref. 3? - There are also many typos that should be corrected.  --- I thank the authors for their rebuttal. As a result, paper clarity is going to be improved. However, I'm still not very convinced about biological plausibility of network structure and the way it's trained. Furthermore, some of my technical concerns (e.g. why 2-3-4 are not used for within environment comparison, as 1-5 are, or why novelty implies that only #4 should be used) have not been convincingly addressed either. Therefore, I do not change my score. I believe this story holds great potential but I'm not convinced yet.


Non-metric Similarity Graphs for Maximum Inner Product Search
In this paper we address the problem of Maximum Inner Product Search (MIPS) that is currently the computational bottleneck in a large number of machine learning applications. While being similar to the nearest neighbor search (NNS), the MIPS problem was shown to be more challenging, as the inner product is not a proper metric function. We propose to solve the MIPS problem with the usage of similarity graphs, i.e., graphs where each vertex is connected to the vertices that are the most similar in terms of some similarity function. Originally, the framework of similarity graphs was proposed for metric spaces and in this paper we naturally extend it to the non-metric MIPS scenario. We demonstrate that, unlike existing approaches, similarity graphs do not require any data transformation to reduce MIPS to the NNS problem and should be used for the original data. Moreover, we explain why such a reduction is detrimental for similarity graphs. By an extensive comparison to the existing approaches, we show that the proposed method is a game-changer in terms of the runtime/accuracy trade-off for the MIPS problem.

The authors identify the issue in solving MIPS by reducing it to NNS and propose the use of non-metric similarity graphs to solve MIPS in its native form. While the idea and the empirical results are extremely promising, there are a couple of issues that temper my enthusiasm for this manuscript significantly.    Major: - While the theoretical results claim the correctness of the Greedy Walk algorithm (Alg 1) with s-Delaunay graphs, it is hard for me to understand (concretely or even intuitively) how the approximation in the graph construction (so that it is no longer a s-Delaunay graph) affects the accuracy of Alg 1 for MIPS. With existing methods such as LSH/RP-trees+reduction or Greedy-MIPS, the algorithm will find the exact answer(or within some approximation) eventually or drive the probability of not finding the exact answer to really low values (with RP-trees providing stronger guarantees than LSH). However, with Alg 1, there is no guarantee on the approximation of the obtained answer. - Empirical results in terms of actual runtimes are a great way of evaluating the proposed solution. However, it makes it hard for readers to judge whether the improvements of the proposed solution because of the novel/proposed algorithm (which is discussed in detail in the manuscript) or because of implementation details such as programming languages/libraries used, any kind of cache locality/multithreading/register level optimizations used, etc (which are not discussed/elaborated/focused on in the manuscript and hence unknown to the reader). And these implementation details can be quite significant when we are discussing runtimes in the order of milliseconds (unless these numbers are generated by repeating the runs multiple times and ensuring there are no competing programs on the evaluation machine). An accompanying empirical comparison using number of vector operations (for example, dot products) as a surrogate for time would provide a fairer comparison of the algorithms being evaluated, and make the gains from the proposed algorithm more clear.   Minor: - How are the recall vs. time curves generated for the proposed graph based solution in Figures 2,3? What parameters of the algorithm are being varied to span the complete speed-accuracy tradeoff? This is not clear to me from the description of Alg 1 and Section 4. - As per lines 41-42, the authors claim to provide "theoretical analysis to justify the use of similarity graphs". However, I couldn't find any concrete theoretical result supporting this claim (unless I am missing something). - How does the value of the parameters M (graph vertex degree) and L (the size of the priority queue maintained during the search) affect the performance of the proposed MIPS solution? Is the proposed method sensitive to these parameters? How dependent are these parameters on the dataset?  ================ Post author-response edit: I want to thank the authors for their thoughtful responses; I acknowledge that I have read them. The authors have addressed two of my major concerns and I am thoroughly satisfied with one. However, the other concern (re: approximation-accuracy discussion), I appreciate the explanation and intuition -- it definitely helps me understand things better, but it still lacks certain amount of rigour, and the fact that there is no recovery from a local minima (or alternately, no guarantees on the local minima relative to the global minima) still leaves me somewhat unsatisfied. However, my score is still fairly positive so I will leave it at that.


Recurrently Controlled Recurrent Networks
Recurrent neural networks (RNNs) such as long short-term memory and gated recurrent units are pivotal building blocks across a broad spectrum of sequence modeling problems. This paper proposes a recurrently controlled recurrent network (RCRN) for expressive and powerful sequence encoding. More concretely, the key idea behind our approach is to learn the recurrent gating functions using recurrent networks. Our architecture is split into two components -a controller cell and a listener cell whereby the recurrent controller actively influences the compositionality of the listener cell. We conduct extensive experiments on a myriad of tasks in the NLP domain such as sentiment analysis (SST, IMDb, Amazon reviews, etc.), question classification (TREC), entailment classification (SNLI, SciTail), answer selection (WikiQA, TrecQA) and reading comprehension (NarrativeQA). Across all 26 datasets, our results demonstrate that RCRN not only consistently outperforms BiLSTMs but also stacked BiLSTMs, suggesting that our controller architecture might be a suitable replacement for the widely adopted stacked architecture.

Update:  Thanks for your response. I'm glad to hear that you are going to open source the optimizations; I look forward to playing with this. Best of luck with the follow-up work, and I look forward to seeing how the RCRN performs on SNLI in the cross-attention task setting (really hoping to see this in the camera-ready!).  Original Review:  The core idea behind this paper is to use RNNs (LSTMs or GRUs in this work) to compute the gates (input, forget, etc.) for a higher level RNN. They use this idea to show how to improve performance on a large number of tasks (mostly classification) with relatively simple models that have little to no loss in efficiency compared to models that perform similarly. Some of the performances achieved new state-of-the-art results. When implemented well, they can even be faster despite the apparent addition of complexity.   The setup as controller and listener cells seems unnecessarily confusing. It seems clearer to describe this as a collection of three RNNs, two that compute gates for the state produced by the third. If it was presented that way, then it would be clearer that Eq. 15 and 16 are really the meat of the RCRN. The first 14 equations are really just normal (or bidirectional adaptations of normal) RNN equations (7-9 are nice to see for clarity). I understand that it can be helpful to write out all of those equations some times, but I would recommend defining an RNN cell once, and then focus on describing how you use the three different states, rather than duplicating the same equations 2 times in Eq. 1-6 and then again in 10-14. On my first read, I found myself looking for potential difference in all those equations because I assumed they were written out so explicitly for that reason.  In equations 1-3, not sure if it is a typo, but either way: it would be clearer if the W_i, W_f, and W_o used to get i_t^2, f_t^2, and o_t^2 respectively had a numerical superscript like the other matrices. It sounds like it would be a 2, but without the superscript it kind of feels like a typo since all the other matrices have superscripts. U_i^1 is also used for i_t^2, but it seems like that should be U_i^2 instead. Would like a little clarification or a note if these aren’t typos clarifying that some of the weights are indeed shared.  Why not run on SQuAD? That is a pretty standard QA dataset now, so much so that its absence makes me wonder whether that was a failure case for RCRNs. It would be great if you could assuage that concern for future readers. I could almost say the same for MultiNLI over SNLI now as well, but I understand that both of these have external testing systems that can make using them difficult some times.  Why do you only compare with models that do not use semi-supervised learning on external corpora? If you are using CoVe, which is using supervised learning on external corpora and GloVe, then it might seem a bit arbitrary to rule out semi-supervised augmentations in those tables. I think the claims are still effective for RCRN because the BiLSTM and 3l-BiLSTM also use CoVe, POS embeddings, etc.  What exactly does CUDA optimized mean? How hard would these be to make on your own? Or will you be open sourcing these essential optimizations or even better building into something like PyTorch or Tensorflow? If you want maximum impact, definitely help people get that super fast CUDA optimized version. If that existed open source, I would definitely start trying it out.  I’m not convinced about the claim that this is different than ‘going deeper’. This just seems like a different (apparently smarter/better) way of going deeper than by just stacking RNNs as layers.  I wonder whether you could just use a single RNN cell as the Controller and do a simpler transformation of that state to get the different gate values. Perhaps this could be even faster?  Can we have RCRNs controlling RCRNs?  Character level embeddings are using a BiGRU model; is this better than using pertained character embeddings?  Lots of the true SOTA models for entailment do use sentence cross attention, and you chose not to. Why? How do the RCRN’s purported benefits stand up when attention is in the mix for entailment? Or perhaps the Highway networks are particularly important and you don’t get any benefit from the RCRN without them?   It seems insane at first glance that your 3L-BiLSTM is as good as all those other more complex models even without the RCRN. Is that coming from the GRU character embeddings or the POS embeddings or the CoVe embeddings? Embeddings are just really that important it seems, and as we’ve created better representations, simple BILSTMs and stacked versions are getting better.
  Overall is a 7, with a confidence of 4.  I'm reserving very confident for reproducibility of systems that have been declared as being open source in the future. This submission includes a lot of detail, but it seems unlikely that everyone who wants to use this work will implement those custom CUDA ops themselves. The RCRN seems simple enough to reproduce, but at the speeds presented, it seems potentially difficult.


Library Learning for Neurally-Guided Bayesian Program Induction
Successful approaches to program induction require a hand-engineered domainspecific language (DSL), constraining the space of allowed programs and imparting prior knowledge of the domain. We contribute a program induction algorithm called EC 2 that learns a DSL while jointly training a neural network to efficiently search for programs in the learned DSL. We use our model to synthesize functions on lists, edit text, and solve symbolic regression problems, showing how the model learns a domain-specific library of program components for expressing solutions to problems in the domain.

= Summary  A method for learning a DSL for program synthesis together with a search algorithm in that DSL is presented. The method proceeds iteratively, trying to solve tasks with the current DSL, and then extracting new DSL components from the solutions. Experiments show that bootstrapping the method with a DSL made up of trivial primitives is sufficient to discover common high-level constructs present in manually constructed DSLs.  The paper tackles an important problem (DSL design) in an elegant and novel way. The clarity of the paper is not perfect, as the details of the idea require more space than the 8 pages available, but it clearly is stepping stone towards a new generation of program synthesis approaches.  = Quality  The paper precisely defines its main algorithm and the core concepts of the approach. There are a few omissions (e.g., Sect. 2.4 does not explain the exact mechanism to extract new constructs; the supplement gives an algorithm, but omits a clear definition of how "fragments" are found), but overall the paper is precise and explains core concepts well. The experiments are very thorough, comparing the method to interesting ablations and baselines. What I am missing is an analysis of the influence of the initial set of primitives. For example, the list processing set starts with a non-minimal set of primitives (as length is "foldr (\lambda (n) -> 1 + n) 0" and index can be implemented using the others as well, though painfully). What happens, for example, if length is removed?  As an additional data point, I would have appreciated a look at the "evolution" of solutions to a fixed task through the iterations of the main algorithm, i.e., show an initial, early and late solution in the style of Table 1.  = Clarity  Many part of the paper are better understood with the supplement in hand (Sect. 2.3 is not discussing details of how q is modeled, but Supplement Sect. 5 makes clear that is is essentially like DeepCoder; Sect. 2.4 is missing details as discussed above). Overall, the paper is easily understood, but not entirely self-contained and not precise enough to allow re-implementation by third parties.  The supplement is very unpolished (there are many grammatical errors and half-finished sentences, e.g. in line 6, line 33, lines 45/46, ...)  = Originality  While drawing on the program synthesis literature of the last few years (e.g., the guided enumerative search), the core contributions of the paper are completely new and are a substantial step forward for the field.  = Significance  Designing a suitable DSL is often one of the hardest tasks in applying program synthesis to a new domain; the promise of automatically learning a DSL (together with a search algorithm) is obvious. I am not entirely convinced by the authors idea of generalizing to other generative programs, but I am looking forward to being proven wrong.


Re-evaluating Evaluation
"What we observe is not nature itself, but nature exposed to our method of questioning." -Werner Heisenberg Progress in machine learning is measured by careful evaluation on problems of outstanding common interest. However, the proliferation of benchmark suites and environments, adversarial attacks, and other complications has diluted the basic evaluation model by overwhelming researchers with choices. Deliberate or accidental cherry picking is increasingly likely, and designing well-balanced evaluation suites requires increasing effort. In this paper we take a step back and propose Nash averaging. The approach builds on a detailed analysis of the algebraic structure of evaluation in two basic scenarios: agent-vs-agent and agentvs-task. The key strength of Nash averaging is that it automatically adapts to redundancies in evaluation data, so that results are not biased by the incorporation of easy tasks or weak agents. Nash averaging thus encourages maximally inclusive evaluation -since there is no harm (computational cost aside) from including all available tasks and agents.

The paper proposes a new way to evaluate algorithms given their performance either on tasks or in a match against each other. Many papers evaluate new algorithms and tasks in ad-hoc ways, and it is often unclear which algorithm is better than which, and under what circumstances. The authors bring an educated approach to this issue by borrowing techniques from game theory. They consider a meta-game where one player chooses a team / distribution of algorithm and the other a team / distribution of tasks and propose ranking methods based on this game. There are two somewhat different approaches given in the paper. The first describes a decomposition of the algorithm-vs-task (AvT) or algorithm-vs-algorithm (AvA) matrix that lead to a low-rank approximation of sorts, giving ‘latent vectors’ for each algorithm that act as a multi-dimensional ranking. These provide a better representation of the algorithms than a one-dimensional ranking such as Elo, at the cost of being less interpretable. In particular they do not force a total order. The second method provides a ranking that tries to maintain a set of properties that seem very reasonable. This method provides each algorithm a score based on its comparison to the min-max strategy in the above mentioned game. The ranking is done according to the score.     Summary: This is somewhat of a non-standard paper as it doesn’t bring any new technique or develop a new mathematical tool. Rather it adapts known techniques in game theory and attempts to bring order to a somewhat chaotic area of algorithm evaluation. I suspect that the decision of accepting / rejecting the paper could be somewhat subjective but my opinion is to accept the paper.  Writing: The paper is a pleasure to read. The authors take time to explain their approach and the intuition for their decisions is presented clearly.   Related literature: The authors should relate their work to the field of dueling bandits. The dueling bandit problem, motivated by IR where algorithms are compared against each other has very similar issue. It is a common case there that the preference between pairs is not transitive meaning alg1 > alg2, alg2 > alg3, alg3 > alg1. Despite that, the bandit problem requires a notion of the best algorithm and of regret, meaning a sub optimality measure for a non-optimal algorithm. The issue is typically solved there by assuming the existence of a Condorcet winner, meaning an algorithm that is likely to beat any other algorithm in a match w.p. larger than 1/2. Given that, the regret associated with a sub-optimal algorithm is determined by how likely it is to lose to this winner. This definition exactly matches the Nash averaging score given here. In fact, there is a paper for dueling bandits titled “Instance-dependent Regret Bounds for Dueling Bandits” that discusses the case where there is no Condorcet winner, and they propose a notion of regret that is strongly related to the Nash averaging score.  Comments: * Equation at 157: Why are the Eigenvalues assumed to be approximately -i,i? Could you add an explanation?  * Line 165: The norm of P is not one - but depends on the number of players. It would be more helpful to discuss something normalized like |P-P’|/|P|. Also, the objective is not to reconstruct P in the Frobenius norm but with logistic loss. Is it possible to present that instead? Finally, why is there a ‘slight’ improvement? 0.85 to 0.35 seems like a pretty big improvement..  * line 168: Add one line to describe what the algorithms are meant to solve  * Is it true that n_A(i) is the payoff of algorithm i against the maxent NE strategy? This could be mentioned explicitly to demonstrate interpretability   * Theorem 1: The proposed score is the Nash Averaging n_A, but the theorem gives properties of p^*. I understand that the proofs go through p^* but shouldn’t the main statement be written in terms of n_A? Also, the notation \eps-Nash should be defined before used in a theorem.  * Normalizing scores (in AvT) seem like an important issue that is somewhat avoided - is there any educated way of doing that? Perhaps converting to quantiles so outliers will not be an issue?


Global Gated Mixture of Second-order Pooling for Improving Deep Convolutional Neural Networks
In most of existing deep convolutional neural networks (CNNs) for classification, global average (first-order) pooling (GAP) has become a standard module to summarize activations of the last convolution layer as final representation for prediction. Recent researches show integration of higher-order pooling (HOP) methods clearly improves performance of deep CNNs. However, both GAP and existing HOP methods assume unimodal distributions, which cannot fully capture statistics of convolutional activations, limiting representation ability of deep CNNs, especially for samples with complex contents. To overcome the above limitation, this paper proposes a global Gated Mixture of Second-order Pooling (GM-SOP) method to further improve representation ability of deep CNNs. To this end, we introduce a sparsity-constrained gating mechanism and propose a novel parametric SOP as component of mixture model. Given a bank of SOP candidates, our method can adaptively choose Top-K(K > 1) candidates for each input sample through the sparsity-constrained gating module, and performs weighted sum of outputs of K selected candidates as representation of the sample. The proposed GM-SOP can flexibly accommodate a large number of personalized SOP candidates in an efficient way, leading to richer representations. The deep networks with our GM-SOP can be end-to-end trained, having potential to characterize complex, multi-modal distributions. The proposed method is evaluated on two large scale image benchmarks (i.e., downsampled ImageNet-1K and Places365), and experimental results show our GM-SOP is superior to its counterparts and achieves very competitive performance. The source code will be available at http://www.peihuali.org/GM-SOP.

The idea behind a Sparse Gated Mixture (GM) of Expert model has already been proposed in [1]. The main novelty of this paper is in the way the sparse mixture model is applied, namely to modify the 2nd order pooling layer within a deep CNN model to have a bank of candidates. The way GM works is as follow: Given an input sample, the sparsity-constrained gating module adaptively selects Top-K experts from N candidates according to assigned weights and outputs the weighted sum of the outputs of the K selected experts.  Another contribution of this paper is to define a parameterized architecture for pooling: For the choice of expert, the authors use a modified learnable version of matrix square-root normalized second-order pooling (SR-SOP) [2] .  --------------- The idea of using sparsely gated experts is very promising and finding new ways to apply it within model architectures, as it is the case in this work, is important.  The experiments first show that SR-SOP is advantageous over regular SOP, both are prior work, but it's good to justify why use SR-SOP in the first place. It then shows results on the new parameterized version (which slightly improves results) and then results on adding the GM which makes 2+% overall improvements.  While the presented results show advantage of this method, the experiments could be stronger and more convincing. A down-sampled ImageNet is used as dataset and a modified ResNet is used for baseline. Why not experimenting with the standard-size ImageNet, or CIFAR10-100 datasets (especially, if choosing to go with a smaller size image)? Much more research has been done on these 2 datasets and improving over those strong baselines would be a much more convincing result.    Have the authors tried increasing the number of CMs (component modules, aka experts) to multiple hundreds or thousands, to see the limit of adding capacity to the model? The maximum # of CMs in this paper is 32 and it looks like the best overall result is achieved with that many CMs. In a prior work [1], some of the best results are achieved by having 1-2k experts.    [1] N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. V. Le, G. E. Hinton, and J. Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In ICLR, 2017. [2] P. Li, J. Xie, Q. Wang, and W. Zuo. Is second-order information helpful for large-scale visual recognition? In ICCV, 2017


Probabilistic Neural Programmed Networks for Scene Generation
In this paper we address the text to scene image generation problem. Generative models that capture the variability in complicated scenes containing rich semantics is a grand goal of image generation. Complicated scene images contain varied visual elements, compositional visual concepts, and complicated relations between objects. Generative models, as an analysis-by-synthesis process, should encompass the following three core components: 1) the generation process that composes the scene; 2) what are the primitive visual elements and how are they composed; 3) the rendering of abstract concepts into their pixel-level realizations. We propose PNPNet, a variational auto-encoder framework that addresses these three challenges: it flexibly composes images with a dynamic network structure, learns a set of distribution transformers that can compose distributions based on semantics, and decodes samples from these distributions into realistic images.

I have read the other reviews and the authors' rebuttal. My opinion remains that this is a strong submission and should be accepted. I appreciate the authors addressed each of my concerns and questions in sufficient detail. -------------------- This paper proposes a new, modular neural architecture for generating images of scenes composed of attributed objects. The model takes as input a description of the scene to be generated, in the form of a tree-structured 'program' describing the objects, their attributes and their spatial relations. Each elementary concept (object, attribute) is encoded into a distribution over spatial and visual appearance attributes (mu, sigma). These distributions are then composed--following the program tree structure--using a set of operators for combining attributes, describing objects with attributes, and laying out objects in the space. The final result is a final distribution which can be sampled to produce a latent vector representation of the entire scene; this latent representation is then decoded into an image via standard image decoder/generator networks. The model is trained using variational inference. The paper applies this model to generating scenes of colored MNIST digits as well as CLEVR-G scenes, where it often outperforms strong text-to-image baselines under an object-detection-based performance metric.  This paper makes a good contribution: taking ideas from Neural Module Networks and related work, used for analyzing and answering questions about images, and applying them to task of generating the images themselves. The proposed architecture is reasonable (though I wish, space permitting, more justifications for design decisions were given). The evaluation is appropriate, and the proposed model clearly produces better samples than alternative approaches. I have some concerns (see below), but overall I am currently in favor of accepting this paper.  - - - - - - -   I found much of the model to be insufficiently described / under-specified. The level of detail at which it is described is almost sufficient for the main paper. But there are also quite a few details which ought to be provided somewhere (at least in the supplemental) in order for this work to be reproducible:  -- Figure 2: What is w_layout? I infer that this is probably an embedding for a spatial relation concept, but this is never defined (or even mentioned anywhere in the paper text)  -- Figure 2: I believe (according to Equation 3) that *both* inputs should feed into the "Convs" block, yes?  -- Equations 2 and 3: It's not clear to which distributions these operators apply: both z_a and z_s, or just z_a? The way Figure 2 is drawn, it looks like they apply to both. But the f convolution in Equation 3 only makes sense for z_a, so I suspect that this is the case. This could use clearing up, and Figure 2 could be re-drawn to be more precise about it.  -- Equation 4: How are bounding boxes / A sampled? Figure 2 shows the sampler as conditioned on z_s, but how, precisely? What's the functional form of the distribution being sampled from?  -- Transform: I also don't quite follow how the distribution is resized. The bilinear interpolation makes it sound as if the parameter tensors are resampled onto different WxH resolutions--is this correct?  -- Equation 5: The generated bounding boxes don't appear in this equation, nor does w_layout. How is the background canvas placement accomplished? Does this step perhaps use Transform as a subroutine? As it stands, Equation 5 only shows the application of the "Refiner CNN," which is one small part of the overall diagram given in Figure 2.  -- Line 172: "we first map the semantic information into programmatic operations": is this process manual or automatic?  -- Equation 6: You've written the proposal distribution as q(z); I assume you mean q(z | x)? The form of this encoder is not mentioned anywhere.  -- There are other architectural details would would belong in the supplement (e.g. the LSTM encoder in the conditional prior).  The detector-based performance measure is a nice idea. I'm wondering, though--do you detect the 'spatial relation' attributes such as 'top' and 'left?' It's not obvious how to do that, if that is something your evaluation takes into account. If not, that should also be mentioned.  ColorMNIST: It would be good to see some qualitative results to go along with Table 1.  Visualizing the learned concepts: If I understand correctly, your model should also permit you to visualize attribute concepts such as 'shiny?' Do I have that right, and if so, what do these look like?  Figure 5: In the zero-shot experiment, which concept combinations were omitted from the training data? Knowing this would help to assess the errors that the different methods are making in the presented images.  Generalization to complex scenes: If I understand correctly, this experiment the model's ability to train on and generate larger scenes. Can the model generalize to larger scenes than the ones it was trained on? E.g. if you train a model on scenes with at most 4 objects but test it on scenes with up to 8, how does it behave? I would hope to see some graceful degradation, given the modular nature of the model.  Two related works which should probably be cited: (Generating structured scenes from text) "Image Generation from Scene Graphs," Johnson et al. CVPR 2018 (Disentangled representations) "Deep Convolutional Inverse Graphics Network," Kulkarni et al. NIPS 2015


Unsupervised Cross-Modal Alignment of Speech and Text Embedding Spaces
Recent research has shown that word embedding spaces learned from text corpora of different languages can be aligned without any parallel data supervision. Inspired by the success in unsupervised cross-lingual word embeddings, in this paper we target learning a cross-modal alignment between the embedding spaces of speech and text learned from corpora of their respective modalities in an unsupervised fashion. The proposed framework learns the individual speech and text embedding spaces, and attempts to align the two spaces via adversarial training, followed by a refinement procedure. We show how our framework could be used to perform spoken word classification and translation, and the experimental results on these two tasks demonstrate that the performance of our unsupervised alignment approach is comparable to its supervised counterpart. Our framework is especially useful for developing automatic speech recognition (ASR) and speech-to-text translation systems for low-or zero-resource languages, which have little parallel audio-text data for training modern supervised ASR and speech-to-text translation models, but account for the majority of the languages spoken across the world.

This paper investigates unsupervised alignment between text and speech using embedding for both modalities.  Word and speech embedding are first generated, the speech embedding are computed using an unsupervised alignment procedure and averaged. The alignment model is then trained based on adversarial training. The proposed approach is then evaluated on two tasks: spoken word detection and spoken word translation.   Quality: The results of the spoken word detection study are not very appealing: 89% accuracy for the supervised baseline versus 23.7% for the unsupervised proposed approach is rather poor. But the synonym retrieval results are very interesting. Showing that the speech embedding seems to capture semantics information instead of phonetic is significant. The spoken word translation experiment is also interesting. For the spoken word detection task, it's not very clear: what is the input exactly ? is it a full utterance containing several words including the target one, or was the utterance aligned and cropped to contain only the uttered word ? I also think the authors should select a more established task instead of the one they proposed, spoken term detection seems relevant [1].   Clarity: the paper is well-written and easy to read.  Originality: This work is a novel combination of established techniques, such as word embedding and adversarial training. The unsupervised method to compute speech embedding is novel to my knowledge.  Significance: Unsupervised sequence classification, such as ASR and MT, is definitely the next huge milestone in the field, and this paper is significant step towards that long-term objective.   Overall, despite the not-so-great results, I think the paper should be accepted as it presents a good lead toward unsupervised sequence classification.  [1] J. G. Fiscus, J. Ajot, J. Garofolo, and G. Doddingtion, “Results of the 2006 spoken term detection evaluation,”  in Proc. SIGIR, vol. 7. Citeseer, 2007, pp. 51–57.  UPDATE: The A vs A* result is very interesting and is a very good result to demonstrate the potential of the approach. I suggest that this result should be better highlighted in the final version paper


Reward learning from human preferences and demonstrations in Atari
To solve complex real-world problems with reinforcement learning, we cannot rely on manually specified reward functions. Instead, we can have humans communicate an objective to the agent directly. In this work, we combine two approaches to learning from human feedback: expert demonstrations and trajectory preferences. We train a deep neural network to model the reward function and use its predicted reward to train an DQN-based deep reinforcement learning agent on 9 Atari games. Our approach beats the imitation learning baseline in 7 games and achieves strictly superhuman performance on 2 games without using game rewards. Additionally, we investigate the goodness of fit of the reward model, present some reward hacking problems, and study the effects of noise in the human labels.

As the title implies, this paper examines imitation learning that combines human demonstrations and human preferences. The main algorithm builds on DQFD to learn Q-Values from human demonstrations and subsequently fine-tunes the policy using preference elicitation methods. More specifically, preferences are compiled into a surrogate reward function which is then used to further optimize the policy. The resulting algorithm is validated on nine Atari environments and results show that the technique of combining demonstrations with preferences is better than either using either source of feedback alone.  Overall, the paper is clearly written, tackles a well-scoped problem, and presents compelling results. It's great to consider learning from more than a single type of human feedback and the combination of human demonstrations with preferences is a promising one.  My largest concern is how reliant the algorithm is on preference feedback induced from the synthetic oracle. As the authors acknowledge, human preference feedback seems to nearly always perform worse than oracle preferences. It was unclear whether the problem stemmed from inexperienced human demonstrators or the simple fact that humans are bad at providing preference feedback. The latter makes me concerned about the ultimate usefulness of this approach.  While it makes sense that the oracle preference feedback was used to automate the experimentation and remove the need for humans, it really detracts from the argument of learning on environments with no reward. I'm afraid the oracle could simply be helping the agent induce a learned version of the environment rewards. If this is the case, the proper point of comparison should be DQFD with access to true environment rewards.  Another experiment that I would like to see would be a comparison between having human annotators spend time providing extra demonstrations versus preference labels. I wonder if the eventual algorithm performance would be better by directing humans to provide additional demonstrations rather than preferences.  After reading the author rebuttal, my evaluation still remains at weak accept.


Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures
The backpropagation of error algorithm (BP) is impossible to implement in a real brain. The recent success of deep networks in machine learning and AI, however, has inspired proposals for understanding how the brain might learn across multiple layers, and hence how it might approximate BP. As of yet, none of these proposals have been rigorously evaluated on tasks where BP-guided deep learning has proved critical, or in architectures more structured than simple fullyconnected networks. Here we present results on scaling up biologically motivated models of deep learning on datasets which need deep networks with appropriate architectures to achieve good performance. We present results on the MNIST, CIFAR-10, and ImageNet datasets, explore variants of target-propagation (TP) and feedback alignment (FA) algorithms, and examine performance in both fully-and locally-connected architectures. We also introduce weight-transport-free variants of difference target propagation (DTP) modified to remove backpropagation from the penultimate layer. Many of these algorithms perform well for MNIST, but for CIFAR and ImageNet we find that TP and FA variants perform significantly worse than BP, especially for networks composed of locally connected units, opening questions about whether new architectures and algorithms are required to scale these approaches. Our results and implementation details help establish baselines for biologically motivated deep learning schemes going forward.

Summary  Under the objective of finding biologically plausible alternatives to backpropagation that perform well on difficult tasks, variants of difference target propagation are compared to direct feedback alignment and backpropagation. All variants are found to perform significantly worse than backpropagation in digit classification (MNIST) and, in particular, in object classification (CIFAR-10 and ImageNet).   Quality  The study is mostly well done and the results seem plausible. But, to qualify as a solid and reproducible benchmark, I expect more details. Also, some parts feel a bit preliminary.  1. For how many epochs were the different methods trained? Do you report optimal values for a fixed number of training epochs or do you report the test error at the optimal validation value for each method? It seems, for example, that the test classification errors in Figure 2 (right) are still decreasing after epoch 500 for all variants of DTP. 2. With how many different initialisations where the results obtained in table 1 and 2? Did you repeat each experiment so many times that confidence interval are negligible? 3. Can you report the best hyperparameters in addition to the details of hyperparameter optimization? 4. AO-SDTP was tested only for 512 auxiliary outputs and the authors refer to future work (line 268). It would be interesting to know more about the performance of AO-SDTP for different dimensionality of z; potentially even for different ways to obtain the random features. 5. Since DTP alternating was always best on MNIST and CIFAR-10, I think it would be interesting to see it's performance on ImageNet, also to have an estimate of what ideally could be achieved with AO-SDTP.   Clarity  The paper is well written and easy to follow.  One minor point minor point where I stumbled, though, was in table 2, where top-1 and top-5 is not defined. For readers familiar with ILSVRC2012 - I guess this is what you are using - it is probably immediately clear what is meant. For everybody else, one sentence of explanation would help.  Another minor point that I believe could still be improved is the notation of the different losses. For example, why L(h_L) and not L(Theta)? What is L_y in section 5.5 of the appendix?   Originality  The work is original in the sense that the evaluation of DTP variants on MNIST, CIFAR-10 and Imagenet was never published before. The variants themselves are original as well, but minor and straightforward modifications of the original DTP.   Significance  It is important that people working on biologically plausible alternatives to backpropagation know about the limitations of DTP and variants. But since the paper does not present an advancement of state-of-the-art alternatives to backpropagation I think it would be better suited for a more specialized audience, maybe at a NIPS workshop.     Minor Points  - In Figure 2 & 4 the colours for parallel and alternating are hard to distinguish in print. - References 12 & 13 (identical up to title) could be replaced by https://elifesciences.org/articles/22901  - line 47: connections - line 68: to unveil - lines 77-78: I am not a native English speaker but "which eliminate significant lingering biologically implausible features" sounds strange to me.  - lines 109-110: I don't think that the backprop equations themselves imply a mode of information propagation that does not influence neural activity. The neural activity in different phases could corresponds to different quantities of the backprop algorithm. - I don't understand the sentence that starts on line 182. - line 259: perform better - I think the statement starting on line 294 is a bit strong. Reference 37 falls in this period and I am sure one can find further works in computational neuroscience on this topic between the mid 1990s and 2010.   Let me conclude with two questions that go beyond the discussion of this specific paper but address the attempt to find biologically plausible alternatives to backpropagation more broadly. I agree with the authors that biologically plausible methods should also be evaluated on behavioural realism. But what is a behaviourally realistic task? Is the repetitive presentation of static images in minibatches together with corresponding labels a behaviourally realistic task? Shouldn't we rather search for difficult and behaviourally realistic tasks that are less inspired by machine learning challenges but more by actual tasks humans solve?  ==== The author's response answered most questions and concerns. Suboptimal I find the decision to compare the methods on a fixed number of epochs. I can understand it because of limited computation resources. But since some of the curves did not seem to saturate (see point 1 above) I would be curious to know whether the bio-plausible alternatives would become significantly better if the number of epochs were a tunable hyper-parameter. Note that I think optimal performance after a fixed number of epochs is a perfect metric in machine learning but I am not convinced it is the right one to compare bio-plausible alternatives, since our random initializations of the network weights probably don't capture the strong inductive biases that evolutionary processes instantiate in biological neural networks.


Actor-Critic Policy Optimization in Partially Observable Multiagent Environments
Optimization of parameterized policies for reinforcement learning (RL) is an important and challenging problem in artificial intelligence. Among the most common approaches are algorithms based on gradient ascent of a score function representing discounted return. In this paper, we examine the role of these policy gradient and actor-critic algorithms in partially-observable multiagent environments. We show several candidate policy update rules and relate them to a foundation of regret minimization and multiagent learning techniques for the one-shot and tabular cases, leading to previously unknown convergence guarantees. We apply our method to model-free multiagent reinforcement learning in adversarial sequential decision problems (zero-sum imperfect information games), using RL-style function approximation. We evaluate on commonly used benchmark Poker domains, showing performance against fixed policies and empirical convergence to approximate Nash equilibria in self-play with rates similar to or better than a baseline model-free algorithm for zero-sum games, without any domain-specific state space reductions.

2. The paper draws the connection between regret minimization and actor-critic algorithms. Specifically, it shows the connection by defining a new variant of an actor-critic algorithm that performs an exhaustive policy evaluation at each stage (denoted as policy-iteration-actor-critic), together with an adaptive learning rate. Then, under this setting, it is said that the actor-critic algorithm basically minimizes regret and converges to a Nash equilibrium. The paper suggests a few new versions of policy gradient update rules (Q-based Policy Gradient, Regret Policy Gradient, and Regret Matching Policy Gradient) and evaluates them on multi-agent zero-sum imperfect information games. To my understanding, Q-Based Policy Gradient is basically an advantage actor-critic algorithm (up to a transformation of the learned baseline) 3. The authors mention a “reasonable parameter sweep” over the hyperparameters. I’m curious to know the stability of the proposed actor-critic algorithms over the different trials 4. The paper should be proofread again. Many broken sentences and typos (“…be the reach probability of the all policies action choices…”, “…Two transition buffers of are used…”) 5. G_t is not defined. I believe the meaning is cumulative reward R_t, since the authors use them interchangeably.  6. Being at the border between game theory and RL makes notation very confusing. For example, pi that stands for policy (a.k.a strategy in game theory literature) denotes “transition probability” in game theory. Therefore, I encourage the authors to include a table of notations. 7. In line 148 the authors give a fresh intuition about the policy gradient expression (“…so estimate of the gradient must be corrected by dividing by policy probability”). However, that’s counterintuitive. A sampled trajectory with low probabilities would produce a high-intensity gradient (dividing by small numbers), which is the opposite of what is meant to happen. Please clarify your intuition. 8. In line 150 you define Q-based policy gradient, which basically has the form of advantage actor-critic. If this is correct, then please illuminate the relation for the sake of clarity (little or none knows what Q-based PG is vs the popularity of advantage actor-critic methods) 9. Please explain the untraditional containment symbol of ha and z in line 187. Did you mean to indicate set containment? And please clarify what is “ha” (you’ve defined “eta of ha and z” but never “ha” explicitly) 10. Do you share the same critic for all players?  Summary:  Quality - the paper draws a connection between Q values in RL and counterfactual values from the field of game theory. Second, it draws a connection between actor-critic and regret minimization. Although this is true under a very specific setup, I believe that this is an important result that connects two worlds that have been parallel up to now.  Clarity - The paper is hard to follow because it stands on the borderline between disciplines with conflicting notations. In order to remedy the problem, the paper should pay extra attention to notations and their use. Besides that, an extra proofreading of the paper would benefit the paper significantly (many broken sentences and typos)  Originality and significance - I believe that the paper is novel and significant to a fair degree


Third-order Smoothness Helps: Faster Stochastic Optimization Algorithms for Finding Local Minima
We propose stochastic optimization algorithms that can find local minima faster than existing algorithms for nonconvex optimization problems, by exploiting the third-order smoothness to escape non-degenerate saddle points more efficiently. More specifically, the proposed algorithm only needs e O(✏ 10/3 ) stochastic gradient evaluations to converge to an approximate local minimum x, which satisfies krf (x)k 2  ✏ and min (r 2 f (x)) p ✏ in unconstrained stochastic optimization, where e O(·) hides logarithm polynomial terms and constants. This improves upon the e O(✏ 7/2 ) gradient complexity achieved by the state-of-the-art stochastic local minima finding algorithms by a factor of e O(✏ 1/6 ). Experiments on two nonconvex optimization problems demonstrate the effectiveness of our algorithm and corroborate our theory.

This submission is concerned with unconstrained nonconvex stochastic optimization problems, in a setting in which the function to optimize is only available through stochastic estimates. Obtaining points satisfying second-order necessary optimality conditions has been a recent topic of interest at NIPS, as such points can be as good as global minima on several problems arising from machine learning. The authors present new complexity results that improve over the existing complexity bounds for finding an approximate local minimum, identified as a point for which the gradient norm is less than a threshold $\epsilon$ and the minimum Hessian eigenvalue is at least $-\sqrt{\epsilon}$. By assuming that the objective function possesses a Lipschitz continous third-order derivative, the authors are able to guarantee a larger decrease for steps of negative curvature type: this argument is the key for obtaining lower terms in the final complexity bounds and, as a result, lower dependency on the tolerance $\epsilon$ compared to other techniques ($\epsilon^{-10/3}$ versus $\epsilon^{-7/2}$ in previous works).  The authors conduct a thorough review of the related literature, and discuss the main differences between existing algorithms and theirs in Section 2. This literature review appears exhaustive, and identifies key differences between this work and the cited ones.  Section 4 presents the source for the improved results, which lies in the use of the Lipschitz constant of the third-order derivative to scale a negative curvature step. This result is, to the authors' own admission, inspired by reference [9] of the submission, and it even seems that the authors could have used some results from [9, Section 4.1] ``off-the-shelf'' in their analysis, as they have done so for several other lemmas in appendix. I acknowledge that there are differences between their (stochastic) context and that of [9], however I believe the connection with [9] could be better emphasized. In particular, the formula used here for the step size $\eta = \sqrt{3\epsilon_h/L_3}$ already appeared in [9], along with the idea of considering opposite directions for descent. The introduction of a Rademacher variable to cope with stochasticity is an interesting feature, which I would have liked to see discussed more in detail.  Section 5 combines the new negative curvature descent routine with the Stochastically Controlled Stochastic Gradient (SCSG) algorithm. The combination of this first-order method with a negative curvature subroutine cannot be viewed as a fully original contribution, since it already appeared in references [5,34], as described by the authors in Section 2. The main complexity bound is presented in this section, along with several remarks comparing the results with those achieved by similar techniques, and the improvement in terms of dependencies with respect to $\epsilon$ is discussed. The use of the new negative curvature descent procedure clearly improves this dependency, yet the knowledge of the third-order Lipschitz constant is necessary to yield this improvement.  Section 6 provides numerical experiments on two nonconvex problems for the proposed algorithm, called FLASH, compared to variants of Stochastic Gradient Descent (SGD) and SCSG. The figures indicate that FLASH has a better behavior on those problems. Nevertheless, it seems that none of the other schemes used for comparison achieves a second-order guarantee similar to that of FLASH (the noisy stochastic gradient method provides guarantees of converging close to a local minimum, but under a different set of assumptions). Moreover, it seems that none of the other methods relies on negative curvature steps, Hessian-vector or stochastic Hessian-vector products. Without questioning the interest of the comparison (which advocates for the use of negative curvature directions), one can wonder why the authors did not compare with methods that were augmented with negative curvature subroutines. In particular, given that the behavior of SCSG is already good compared to the SGD-based algorithms on the matrix sensing problem, a comparison with other SCSG-based techniques such as those in [5,34] would have been interesting. If FLASH was shown to outperform those methods, it could be an argument in favor of large steps sizes for negative curvature descent.  In summary, I believe the ideas and results of the submission are correct, clearly motivated and described throughout the main body of the submission. As far I could check, the results in the appendices are correct. The originality of the submission is difficult to assess, in that it gather ideas and results for several other works. The main contribution is the use of a new negative curvature subroutine for stochastic optimization, and its analysis is new: however, it is still close to the original idea in the deterministic setting presented in [9]. The hybridization of a negative curvature routine with the SCSG framework is not new, even if the use of this particular routine is. It is unclear whether this submission will have a large impact on the community, especially without comparison with other SCSG-based frameworks.  As a result, I will give this submission an overall score of 5.  Additional comments:  a) Line 280: ``step size $\alpha$ (What is $\alpha$)" is most likely a typo. I would indeed be interested in knowing the chosen initial value for $\alpha$, as it does not seem that information about the Lipschitz constants was used.  b) Lines 404-405: A more precise reference to [32] would be appreciated, to confirm that one indeed obtains $2\sigma^2$ in the variance formula. I was also expecting the norm to be squared. More details would be appreciated.  c) Line 457: ``Applying Markov's inequality [...]" Given that the expectation on the previous equation line 455 does not encompasses the entire left-hand side, the application of Markov's inequality can be unclear at first reading (in particular, one might get confused by the presence of multiple expectations in the sum). I would suggest that the authors add a left-hand side to the last equation on line 455, indicating the expected value on which Markov's inequality will be applied.  d) Lines 472-473: The way the upper bound on $|\mathcal{I}^1_2|$ is derived could in my opinion be better described. In particular, it might be good to indicate which equation involving $|\mathcal{I}^1_2|$ is used as a starting point, to which the various bounds on $B$ are then applied.  Edit following author feedback:  I thank the authors for the efforts they made in addressing my comments during the feedback phase. Their additional numerical experiments are informative, but in my opinion they are not convincing enough to change my review. In particular, I would have liked to be certain that the variants of NEON and NEON2 that are run correspond to a combination of these routines with SCSG, which is not clearly indicated by the authors. Still, assuming that this is the case, these additional experiments show that the performance of FLASH is close to that of NEON/NEON 2 on the MS problem, and it is not clear if the better performance of FLASH is due to longer negative curvature steps. I would have expected the authors to provide evidence that the NEON and NEON2 methods are indeed taking smaller steps in negative curvature directions, which is to be expected from the theoretical analysis.   I thus believe that the authors could have made a stronger case for their method by actually showing that longer negative curvature steps were taken, which is not straightforward from their experiments. Since on the theoretical side, my concerns have not been alleviated by the author feedback nor the other reviews, I maintain my original assessment, and a score of 5 for this submission.


Fast Similarity Search via Optimal Sparse Lifting
Similarity search is a fundamental problem in computing science with various applications and has attracted significant research attention, especially in largescale search with high dimensions. Motivated by the evidence in biological science, our work develops a novel approach for similarity search. Fundamentally different from existing methods that typically reduce the dimension of the data to lessen the computational complexity and speed up the search, our approach projects the data into an even higher-dimensional space while ensuring the sparsity of the data in the output space, with the objective of further improving precision and speed. Specifically, our approach has two key steps. Firstly, it computes the optimal sparse lifting for given input samples and increases the dimension of the data while approximately preserving their pairwise similarity. Secondly, it seeks the optimal lifting operator that best maps input samples to the optimal sparse lifting. Computationally, both steps are modeled as optimization problems that can be efficiently and effectively solved by the Frank-Wolfe algorithm. Simple as it is, our approach has reported significantly improved results in empirical evaluations, and exhibited its high potentials in solving practical problems.

The paper studies the method of hashing a vector inspired by the "fly algorithm" where flies apparently lift an input vector in the olfactory system (after mean shift)  into a higher dimensional space and then sparsify the resulting lifted vector by taking its top k coordinates. Earlier work[ 8] shows that the fly algorithm is a strong alternative to random hyperplane based LSH for similarity search. In this work they provide an algorithm to compute the optimal projection matrix instead of using a random one so that the similarity matrix preserved after the lift. They show experimentally that it outperforms autoencoder, random non optimized fly algorithm, standard LSH for similarity search.  The approach is intriguing and the inspiration from fly visual cortex is very interesting. But the theme of the paper overlaps with the previous work [8] and the main novelty here is the optimization of the projection matrix. The writing and flow of the presentation could  use some improvement.  First to convey the value of the fly algorithm it would be useful to briefly state the exact gains from [8] of this algorithm over standard LSH early on rather than the reader having to reach the experiment section.  Randomized vs optimal lifting: I am also curious if the fly's cortex is believed to use random or "trained" lifting-projection matrix W. If the intent here is to learn a better W to improve upon a random one then you should mention this point up front.  In section 4.2, I found the experimental evaluation metric and the ground truth G to be a bit confusing.  Why is resizing the images using cubic interpolation the right baseline? Shouldn't you instead check something like how well the output hash can predict the digit. Also, what was the value of k that was used?  In comparing autoencoder, LSH and optimal lift/fly, you should also take into account the total number of bits used to represent the final hash values: Besides the number of  hashes you also need to take into account the number of bits required for each hash. For your algorithm it will be log_2(d'). How much was it for LSH.  I couldn't find where \mathcal{Y} was defined: is it the set of k-sparse vectors?


Computing Kantorovich-Wasserstein Distances on d-dimensional histograms using (d + 1)-partite graphs
This paper presents a novel method to compute the exact Kantorovich-Wasserstein distance between a pair of d-dimensional histograms having n bins each. We prove that this problem is equivalent to an uncapacitated minimum cost flow problem on a (d + 1)-partite graph with (d + 1)n nodes and dn d+1 d
arcs, whenever the cost is separable along the principal d-dimensional directions. We show numerically the benefits of our approach by computing the Kantorovich-Wasserstein distance of order 2 among two sets of instances: gray scale images and d-dimensional bio medical histograms. On these types of instances, our approach is competitive with state-of-the-art optimal transport algorithms.

Update after rebuttal & discussion: I am satisfied with the response by the authors and welcome the addition of further experiments; as I now feel confident the notational and other smaller issues will be fixed adequately, I maintain my score and recommend acceptance of this submission. --------- This paper introduces a new approach to solve optimal transport problems by reformulating the computation of d-dimensional Kantorovich-Wasserstein distances w.r.t. separable cost functions as uncapacitated minimum cost flow problems in (d+1)-partite graphs.   The scheme is compared to and contextually embedded into previous works from the literature, and – the main strengths of the paper – the arguments are illustrated by figures and examples, making it easy to follow the contents even if one is only cursorily familiar with optimal transport; numerical experiments are provided to compare the new approach (combined with a network simplex solver for the flow model) with older methods and demonstrate some benefits of the novel formulation. As I found it an interesting read, and somewhat surprising that this simple reformulation has not been thought of before, I believe this work constitutes a worthy contribution to the field and lean toward acceptance of the submission, despite several weaknesses that I shall address point by point below and would like to ask the authors to fix or provide clarification, resp.  1. Please have the paper proof-read for English style and grammar, and typo corrections.  2. I am not sure whether the algorithms you compare to are all still relevant, or if there are better variants that might be more competitive. I got the impression that you are comparing to algorithms that essentially embody the basic principles of one idea or another, but in the Introduction you discuss related works that make it sound like these basic methods have already been improved. Thus, I would appreciate a remark on why you chose the algorithms you did choose in case better/newer variants exist (or if this is not the case and I misread something).  3. Please double-check your notation throughout the paper and consider simplification/clarification where possible. Let me point out some particular issues:  3.a) Why do you introduce X={x_1,...,x_n} (and Y analogously) if it is not really used to simplify/clarify the notation in (1) and (3)-(6)  (by using the indices introduced)? For instance, (4) could be written as \sum_{j=1}^{n} \pi(x_i,y_i) \leq \mu_i  \forall i=1,\dots,n. Similarly for the Earth Mover's Distance later on.  3.b) Right before the EMD formula on p. 3, what is the notation „ (x_i,\mu(x_i))_i “ (and (y_j,\nu(y_j))_j) to mean exactly (i.e., why the extra subscript i or j, resp.)? Similarly, why the extra subscripts i,j at \mu=\{\mu_{i,j}\}_{i,j} and \nu=... in line 98? Those additional subscripts only clutter the notation and make it less readable, in my opinion; they are not really needed.  3.c) In the definition of \nu (line 55), it should by \nu(y_m) at the end (not \nu(x_m)).  3.d) Direct arcs (a,j) to (i,b) are not present in the model (7)--(10), right? Then why are they introduced in Fig. 2 ?  Unless I overlooked something, this does not really make sense and can be quite misleading w.r.t. the tripartite flow model, so please clarify. (I think the direct arcs are from the original bipartite model that the tripartite solutions are mapped back onto via Thm. 1 etc. later... is that so?)  3.e) In Thm. 1, „G \times G“ did not seem right to me; I thought that here, it should be „A^{(1)}\times A^{(2)}“ or „U\times U“? As you can see, I got confused here, and therefore ask to clarify the notation. As a matter of fact, the notation „G“ is overloaded – first, you introduce it to denote an N\times N grid (line 99, p.2) and then to denote the tripartite graph (line 110). Please use distinct letters to avoid mix-ups. (Similarly motivated, I suggest you denote the node subsets in the graph G not by G^{(i)} but rather by V^{(i)}, as the whole node set is called V.)   Similarly, in line 147 (p.5), c should probably also be defined on U\times U (or V\times V), not G\times G. (Furthermore, in Thm. 3, G is reused yet again with a different meaning than before, so please also change notation to a distinct letter here to avoid confusion. Also, „separate“ should be „separable“ in Thm. 3.)  3.f) Proof of Thm.1: the second formula is just a term without connection to anything. I suppose it should be equal to the terms in the first formula? In that case, make it a two-line equation; otherwise, please clarify and fix the second formula.  3.g) In Cor.1, it should be „for any discrete measures“ rather than „for each measures“.  3.h) In the proof of Thm. 3 (p.11) that (25), (26) and (27) are pretty much identical to (17), (18) and (19), respectively. These equations are given new numbers here, but later in the proof the earlier numbers are used. So I suggest either not assigning the new numbers (25)-(27) to the known formulas, or else using the new numbers in the proof instead of reverting to the older ones (17)-(19).   4. I was a bit surprised that combinatorial algorithms for the min cost flow problems are not mentioned, but instead, only the network simplex is advocated (for either bi- or (d+1)-partite formulations). Is the network simplex that much better in practice than, say, the classical strongly-polynomial minimum-mean cycle canceling algorithm (A.V. Goldberg & R.E. Tarjan, „Finding minimum-cost circulations by canceling negative cycles“, JACM 36(4), pp.873-886, 1989)? I would appreciate a brief remark on this in the introduction, where the related of optimal transport and min-cost flow problems is described and the network simplex gets its due credit.  5. It should be mentioned that runtimes of (single-thread) C++ code and (multi-thread) Matlab code are not really comparable. Here, this issue is not critical, since the Sinkhorn algorithm (implemented in Matlab) is not an exact method anyway, and the main take-away message is that the novel (d+1)-partite formulation allows significant performance increase compared to the well-known bipartite model. (What language is the EMD code written in? Is it uncomparable to the C++ network simplex, like Sinkhorn, too?) (Also, what is the CPU speed in your test setup, and how many cores are used for the Matlab parallelized code?)  6. In the discussion of the numerical results, you state that „Indeed, the 3-partite formulation is better essentially because it exploits the structure of the ground distance c(x,y) used, that is, the squared $\ell_2$ norm.“ – This is not entirely accurate. True enough, separability allows to reformulate the problem as a min-cost flow problem on a (d+1)-partite graph. The runtime improves significantly then because the runtime of the network simplex (or any min cost flow algorithm) depends, in particular, on the number of arcs in the network. This is what makes the biggest impact in favor of the new formulations, since, e.g., the 3-partite network has more nodes but far fewer edges than the bipartite graph (3N^2 nodes and 2N^3 arcs vs. N^2 nodes and N^4 arcs, i.e., the node number increases only by a constant factor, but the arc number decreases by an order of magnitude, so the difference becomes more severe the larger the networks are)! Please clarify this aspect in the discussion of your experiments.  7. Please rename Section 5, as it does not contain any word of „future work“. (Also, it may want to clarify that the „much smaller flow network“ is smaller in terms of arc number, but does actually have more nodes.)  8. Please fix capitalization issues in the reference list (Sinkhorn, Kantorovich, Wasserstein, Hitchcock, Mallows, Hellinger, Monge. EMD; also capitalize after a colon (:) and in journal titles). Also, please be consistent in how you format the entries (some contain „pages“, others do not; some contain volume and number in the form v(n), others read „volume v“, spell out authors rather than using „et al.“ in the ref. list, and possibly other such things).   9. This may have been a problem with my printer (low ink), but just wondering: Is the suppl. mat. typeset any differently than the paper itself? It is hard to pinpoint, but somehow looks a bit different on my print-outs.


Deep Homogeneous Mixture Models: Representation, Separation, and Approximation
At their core, many unsupervised learning models provide a compact representation of homogeneous density mixtures, but their similarities and differences are not always clearly understood. In this work, we formally establish the relationships among latent tree graphical models (including special cases such as hidden Markov models and tensorial mixture models), hierarchical tensor formats and sum-product networks. Based on this connection, we then give a unified treatment of exponential separation in exact representation size between deep mixture architectures and shallow ones. In contrast, for approximate representation, we show that the conditional gradient algorithm can approximate any homogeneous mixture within accuracy by combining O(1/ 2 ) "shallow" architectures, where the hidden constant may decrease (exponentially) with respect to the depth. Our experiments on both synthetic and real datasets confirm the benefits of depth in density estimation.
{HM M, T M M } ⊆ LT M ⊆ HT F ⊆ SP N . Moreover, converting from a lower to an upper class can be achieved in linear time and without any increase in size. Our results not only clarify the similarities and subtle differences between these widely-used models, but also pave the way for a unified treatment of many properties of such models, using tools from linear algebra.
We next investigate the consequence of converting a deep mixture model into a shallow one. We first prove that the (nonnegative) tensor rank exactly characterizes the minimum size of a shallow SPN (or LTM or HTF due to equivalence) that represents a given homogeneous mixture. Then, we show that a generic "deep" SPN (with depth at least 2) can be exactly represented by a shallow SPN only when the latter contains exponentially many product nodes. Our result extends significantly those in [7; 26; 10; 18; 8] in various aspects, but most saliently from the restrictive full binary tree [7; 26] to any rooted tree. As a consequence, our results imply that a generic HMM (whose underlying tree is "completely" unbalanced) cannot be exactly represented by any polynomially-sized shallow SPN, which, to our best knowledge, has not been shown before.
From a practical point of view, exact representations are an overkill: it suffices to approximate a given density mixture with reasonable accuracy. Our third contribution demonstrates that under the ∞ metric, we can approximate any homogeneous density mixture within accuracy by combining O(1/ 2 ) shallow SPNs. However, our proof requires the knowledge of the target density hence is not practical. Instead, borrowing a classic idea from [17] we show that minimizing the KL divergence using the conditional gradient algorithm can also approximate any homogeneous mixture within accuracy by combining O(1/
2 ) base SPNs, where the hidden constant decreases exponentially wrt the depth of the base SPNs. Each iteration of the conditional gradient algorithm amounts to learning a base SPN hence can be efficiently implemented. We conduct thorough experiments on both synthetic and real datasets and confirm the benefits of depth in density estimation.
We proceed as follows: In §2 we introduce homogeneous density mixtures. In §3 we articulate the relationships among various popular mixture models. §4 examines the exponential separation in exact representation size between deep and shallow models while §5 turns into approximate representations. We report our experiments in §6 and finally we conclude in §7. All proofs are deferred to Appendix C.

The paper discusses connections between multiple density models within the unifying framework of homogeneous mixture models: tensorial mixtures models [1], hidden Markov models, latent tree models and sum-product networks [2] are discussed. The authors argue that there is a hierarchy among these models by showing that a model lower in the hierarchy can be cast into a model higher in the hierarchy using linear size transformations. Furthermore, the paper gives new theoretical insights in depth efficiency in these models, by establishing a connection between properties of the represented mixture coefficient tensor (e.g. nonnegative rank) and a shallow model. Finally, the paper gives positive and somewhat surprising approximation results using [3].  Strengths: + connections between various models, which so far were somewhat folk wisdom, are illustrated a unifying tensor mixture framework. + depth efficiency is analysed as a simple application of this framework. + approximation results and a greedy algorithm are potentially stimulating for future work.  Weaknesses: - experiment section is rather messy and not insightful - some prior work is not adequately addressed   Quality: The paper is quite timely in discussing and unifying the above mentioned models using homogeneous tensorial mixture models. In some sense, it compiles the gist of multiple research direction in a meaningful manner.  However, in some parts the authors exaggerate the contribution and do not well connect with prior art. For example, while it was perhaps not explicitly mentioned in literature that LTMs are a sub-class of SPNs, this connection is rather obvious, especially in light of well known latent variable semantics in SPNs [4]. I also wonder why SPNs are defined as trees in this paper, when they are usually defined as general DAGs [2]. Furthermore, the strict inclusion claims in Theorem 3.1 are not proven: none of the examples in B.2, B.4, and B.6 rule out the possibility that there exists _some_ polynomial-sized model representing the distribution of the respective "higher" model.  Clarity: The paper is rather clear, but the experimental section is quite messy. One has to actually guess what is going on here; experimental setup and comparison methods are not clearly described. The experiments are, however, not the main contribution in this paper, which is rather on the theoretical side.  Originality. While parts of the paper in isolation would be incremental, in total the paper is original and provides several new insights.  Significance: This paper is clearly of significance and has considerable potential impact.  [1] Sharir et al., "Tensorial mixture models", 2018. arXiv:1610.04167v5 [2] Poon and Domingos, "Sum-Product Networks: A New Deep Architecture", UAI 2011. [3] Li and Baron, "Mixture Density Estimation", NIPS 2000. [4] Peharz et al., "On the Latent Variable Interpretation in Sum-Product Networks", PAMI 2017.   *** EDIT *** I read the authors' rebuttal. I'm happy with it, as they said that they would adequately rephrase Theorem 3.1. Moreover, I'm glad that the authors aim to clean up the experimental section. A good paper, I actually got a new view on the covered material. Definitely worth an accept.


Optimal Algorithms for Non-Smooth Distributed Optimization in Networks
In this work, we consider the distributed optimization of non-smooth convex functions using a network of computing units. We investigate this problem under two regularity assumptions: (1) the Lipschitz continuity of the global objective function, and (2) the Lipschitz continuity of local individual functions. Under the local regularity assumption, we provide the first optimal first-order decentralized algorithm called multi-step primal-dual (MSPD) and its corresponding optimal convergence rate. A notable aspect of this result is that, for non-smooth functions, while the dominant term of the error is in O(1/ √ t), the structure of the communication network only impacts a second-order term in O(1/t), where t is time. In other words, the error due to limits in communication resources decreases at a fast rate even in the case of non-strongly-convex objective functions. Under the global regularity assumption, we provide a simple yet efficient algorithm called distributed randomized smoothing (DRS) based on a local smoothing of the objective function, and show that DRS is within a d 1/4 multiplicative factor of the optimal convergence rate, where d is the underlying dimension.

The paper studies distributed algorithms for non-smooth convex optimization. The authors consider a standard distributed optimization setting in which given a communication graph, where each node has access to its own non-smooth convex function, the goal is to globally minimize the average of functions in little as possible time, where time accounts for both computation on each node and communication between neighboring nodes in the graph. The authors consider two natural assumptions, 1) where the global function (i.e., average of local functions) is Lipschitz continuous and 2) when each local function is Lipschitz continuous. They also consider both distributed master/slave-type algorithms and completely decentralized gossip-based algorithms.  In the centralized master/slave settings, the authors show that under the global Lipschitz assumption, a distributed version of Nesterov's accelerated gradient method (slave nodes only compute gradients and send to master node which runs Nesterov's method), coupled with a standard randomized smoothing technique, gives optimal runtime bounds up to an extra multiplicative dependence on d^{1/4} (d is the dimension), which is caused due to the use of the randomized smoothing. They prove the corresponding lower-bound.  In the decentralized setting, the authors use a known primal-dual algorithm which is distributed via a gossip-based algorithm which overall gives optimal runtime (they prove a corresponding tight lower-bound).  Overall, I think the paper presents very solid contributions on problems of definite interest to the optimization community in NIPS. Both upper bounds and nearly matching lower bounds are given, and hence I support its acceptance.  On the technical side, it seems the results are heavily based on previous results, but still I think the current paper has clear contributions.  One thing is, that I think the writing of Section 4 and in particular Subsection 4.2. could be greatly improved and I encourage the authors to work on the presentation. For one, the authors introduce the saddle-point formulation in Eq. 21 which accounts for the constraint that the solutions on all nodes should be equal. However, their algorithm always returns the average of the solutions on all nodes, so why should we enforce this equality constraint at the first-place? a follow-up question, is then why do we need to use the primal-dual method of [14]? I would be very happy to see more technical explanations here to the algorithmic choices/arguments made in the design of Algorithm 2.  Another question regarding the master/slave algorithm: is it trivial to apply a similar technique when the non-smooth functions admit known DETERMINISTIC smoothing schemes (though not necessarily proximal-friendly)?  *** post rebuttal comments *** I have read the response and it answered my questions.


Mixture Matrix Completion
Completing a data matrix X has become an ubiquitous problem in modern data science, with motivations in recommender systems, computer vision, and networks inference, to name a few. One typical assumption is that X is low-rank. A more general model assumes that each column of X corresponds to one of several lowrank matrices. This paper generalizes these models to what we call mixture matrix completion (MMC): the case where each entry of X corresponds to one of several low-rank matrices. MMC is a more accurate model for recommender systems, and brings more flexibility to other completion and clustering problems. We make four fundamental contributions about this new model. First, we show that MMC is theoretically possible (well-posed). Second, we give its precise information-theoretic identifiability conditions. Third, we derive the sample complexity of MMC. Finally, we give a practical algorithm for MMC with performance comparable to the state-of-the-art for simpler related problems, both on synthetic and real data.

This paper presents mixture matrix completion (MMC) as a novel machine learning tool for learning from low-rank and incomplete data. MMC is a problem that is similar to the problem of subspace clustering with missing data, but is more difficult. Specifically, in MMC the data is assumed to lie in a union of (unknown) low-dimensional subspaces, but the data is not fully observed: only a few entries of each data point are observed, and (unlike subspace clustering with missing data) there is no information as to which entries correspond to the same point. Therefore, one would need to estimate the assignment of entries to data points, the assignment of data points to subspaces, the missing entries, and the subspaces altogether. The major contribution of this paper is the introduction of the MMC problem, a theoretical analysis for when the problem of MMC is well-defined, and an alternating estimation algorithm for solving the MMC problem.  Strengths:  - The paper presents a new machine learning problem formulation that seems natural for addressing practical tasks such as background segmentation. It also has a nice preliminary study of this problem in terms of atheoretical analysis of the identifiability problem, a simple practical algorithm, and experiments on real and synthetic data.   - The paper is well written and the logic flow is clear.   Weaknesses:   - My major concern with the paper is that the theoretical results seem to be stated in vague terms and I don't fully understand them. In Theorem 1, what does it mean to say that Omega^k "contains" disjoint matrices Omega_tau? Does it mean that  Omega^k is a stack of matrices Omega_tau column-wise? Also, what does it mean that it is "possible" to perfectly recover all columns of X^k? Does it mean that the subspaces and the missing entries can be uniquely determined from the observed data? In addition, how is this result compared with the deterministic result for the problem of subspace clustering with missing entries in [4]?  Overall this is a nice paper that brings up a new problem formulation into attention. The study of this problem is still preliminary, though, as one can clearly see that there is no very successful application of the proposed method yet. The experiment on face clustering has a very unrealistic test scenario, and the experiment for background segmentation does not generate results as good as classical robust PCA. Nonetheless, addressing these challenges could be the topic of future study. My reason for not having a higher rating is that I cannot fully appreciate the theoretical studies as mentioned above.  Additional comments:  - It appears that the "cluster" step in the proposed algorithm is very complicated. I'm wondering if this step can be decomposed into the following substeps to make it easier to explain: the estimation of Omega_k is composed of two separate tasks of 1) clustering the entries in each column to different data points, and 2) the assigning data points extracted from all columns to multiple subspaces. In fact, once one have solved task 1) above then the problem reduces to subspace clustering with missing data, which could be solvedby alternating between matrix completion and subspace clustering.  - Since the overall problem is nonconvex, initialization is expected to be very important for the algorithm to achieve good performance.  Can the authors comment on how their algorithm is initialized on real data experiments?  Response to rebuttals:  The updated statement of Theorem 1 in the rebuttal is now much better in terms of clarity and it appears to be very necessary to incorporate it into the final version. Also, given that the conditions in Theorem 1 and 2 are very much similar to those in [4] for a related problem, a more detailed discussion of their connections will help understand the merits of these results. I maintain my overall rating as above and recommend a weak acceptance for this work.


Policy Optimization via Importance Sampling
Policy optimization is an effective reinforcement learning approach to solve continuous control tasks. Recent achievements have shown that alternating online and offline optimization is a successful choice for efficient trajectory reuse. However, deciding when to stop optimizing and collect new trajectories is non-trivial, as it requires to account for the variance of the objective function estimate. In this paper, we propose a novel, model-free, policy search algorithm, POIS, applicable in both action-based and parameter-based settings. We first derive a high-confidence bound for importance sampling estimation; then we define a surrogate objective function, which is optimized offline whenever a new batch of trajectories is collected. Finally, the algorithm is tested on a selection of continuous control tasks, with both linear and deep policies, and compared with state-of-the-art policy optimization methods.

[UPDATE: I have read the author responses and I am satisfied with them]  This paper proposes a novel importance sampling policy search method which accounts for the problem of large variance (low effective sample size) if one just does naive IS and the target distribution differs a lot from the behavior distribution. The authors take a more conservative approach, deriving an update rule based on natural gradient and considering a surrogate which is derived analytically. This is in essence the principle of most of the state-of-the-art algorithms for policy search. They develop both control-based and parameter-based formulations and provide a detailed comparison with some state-of-the-art policy optimization methods.  This is a very well written paper and a solid piece of work. There are several nice contributions in this paper: from a theoretical side, the authors bring to the policy optimization field a set of useful tools from sampling and stochastic calculus which I don't think they are well established in the community. Moreover, they extensively review a vast piece of existing research done in the area. In addition, the proposed algorithms have comparable performance to the state-of-the-art methods.  I have the following comments/questions to the authors:  1- line 185: "unacceptable limitations (...) target variance larger than the behavioral one" and line 109: "we require \sigma_P < 2\sigma_Q".     -1st question: Is this not a contradiction?     -2nd why would one use a target variance larger than the behavioral one? Usually, the behavioral one is more explorative (larger variance) and not the other way around.     -3rd can you relate this with the experiments?  2- While it seems that authors consider general stochastic systems (line 100), they only focus on deterministic systems. In that sense, for example, (line 237) is misleading, since for deterministic systems the probability of a trajectory can be calculated without knowing the model. Isn't it? I strongly suggest to state from the beginning whether they consider deterministic dynamical systems or not.  3 - The numerical experiments are a bit disconnected from the theoretical parts of the paper. For example, given the strong emphasis that is made on the Effective Sample Size, I was expecting some numerical demonstration that showed the increment in the ESS during optimization. Since the bound is directly related with the ESS, why not reporting that as well? It is nice to see performance curves that compare other algorithms, but sometimes is even better to have a theory section well integrated with the empirical one.  4- I would remove the offline/online narrative, which is confusing. For example, in the abstract it looks like the algorithm is just optimizing offline, but in reality it does not.  Other:  - Eq(1): outer integral should be over $\theta$? - line 104: integral again - line 155: "access to set" -> "access to a set" - line 218: swap small o with the plus sign