Computing Higher Order Derivatives of Matrix and Tensor Expressions
Optimization is an integral part of most machine learning systems and most numerical optimization schemes rely on the computation of derivatives. Therefore, frameworks for computing derivatives are an active area of machine learning research. Surprisingly, as of yet, no existing framework is capable of computing higher order matrix and tensor derivatives directly. Here, we close this fundamental gap and present an algorithmic framework for computing matrix and tensor derivatives that extends seamlessly to higher order derivatives. The framework can be used for symbolic as well as for forward and reverse mode automatic differentiation. Experiments show a speedup of up to two orders of magnitude over state-of-the-art frameworks when evaluating higher order derivatives on CPUs and a speedup of about three orders of magnitude on GPUs.

The paper presents an algorithm for computing higher order derivatives of multiple outputs, by translating matrix expressions into Ricci calculus, to generalize modern frameworks for automatic differentiation from first order derivatives of scalar outputs to higher order derivatives of multiple outputs.   Overall comments ---------------- I really enjoyed the core idea of this paper; a generalized form of autograd has immense potential to increase accessibility to many higher order optimization methods by removing the implementation barrier, a benefit that many gradient-based optimization procedures enjoy today. The introduction to Ricci calculus was also easy to read, and the tensor calculus algorithm is easy to understand.   However, I have a number of concerns on the submitted paper, largely regarding the experiments presented in the paper.   1) The paper describes a forward mode, but does not present any sort of evaluation thereof. Jacobians and higher order derivatives are mentioned in the introduction but the experiments focus solely on Hessian computation. The proposed method claims to be able compute derivatives of non-scalar valued functions, but the experiments only demonstrate the approach on scalar-valued functions. The paper would be much stronger if the empirical results matched the described abilities.   2) The main paper claims even better speed up on the GPU. Why do the authors instead present the supposedly worse CPU results up front?  3) I could not find many details regarding the experimental setup, e.g. where the data for the problem came from, whether they are real or synthetic, how the problems are generated if synthetic, etc.  4) There is some strange behavior in Figure 2 of the supplemental material. The authors write that the worse performance of Tensorflow and PyTorch is due to unavoidable data transfer from the main memory and the GPU. However, for some reason PyTorch doesn't suffer from this (but tensorflow does) in only the left-most plot. Why is this the case? Also, within these two frameworks, it should be possible to transfer all data to the GPU before timing the functions, and it is definitely possible to directly allocating all future tensors on the GPU, without needing intermediate data transfers. If the authors believe this is not possible, please elaborate why.   5) The authors point out that one of the drawbacks of existing tools is the creation of enormous computational graphs when computing Hessians. However, the authors explicitly do not report the time for constructing the expression graphs on line 287. For better empirical motivation, the paper should presents timing results that quantify and back up this statement. The results should also compare to the baseline approach of computing the terms based on a formula derived by hand to demonstrate how much (or little) overhead the proposed algorithm has.   Overall, while I enjoyed the exposition on Ricci calculus and automatic differentiation, it is perhaps too verbose, since the paper leaves very little room for experimental justification. This results in a paper that, while well motivated, is experimentally weak. While the few experiments presented do show promise, I cannot recommend acceptance solely based on this, as a much more compelling case can likely be given. Hopefully the authors can shed some light in the rebuttal, as the work here seems to be an important next step in autodifferentiation tools for the machine learning community.   Minor comments -------------- a) Can you elaborate more on the translation to and from Ricci calculus? To actually compute the terms, are expressions translated back into matrix algebra before being evaluated, or are you evaluating the Ricci calculus expressions directly? When you evaluate the Ricci expressions, how do you decide an order of evaluation? For tensor expressions, such choices can greatly effect the actual computational cost of evaluating the expression.   b) Consider include benchmarking results on computing Hessian-vector products, which is also a common use-case when the explicit Hessian cannot be formed or directly decomposed (e.g. due to memory or computational constraints).   c) In the web demo, it would be nice to include more objectives of the mentioned 'classical' machine learning problems (listed in lines 276-277) as examples, as well as examples which compute Hessians as opposed to gradients, and examples which compute non-scalar outputs. It also appears that there is no max operator.   Quality ------- Overall the experiments section is much poorer in quality, as described in this review.   Clarity ------- I found the majority of the main paper to be clear and well-written, with one major exception: the experiments section. Specifically, the `Results' paragraph is extremely poorly written, with most of the sentences not being well-formed.   One other major point of clarity is that there is a mismatch between the statements in the conclusion and the experiments. The conclusion claims that the approach achieves state of the art performance for first order derivatives, while the experiments merely claim that 'basically, all frameworks are equally fast'. The plots in the Appendix do not back up the conclusion.   minor suggestions for main paper:  -On line 115, it may not be clear to readers what it means to 'transform contravariantly' or 'transform covariantly' -The last sentence in line 273-274 does not make sense.  -While 'probably' and 'basically' are not necessarily bad to use in general, their usage in the experimental section is not great  -Figure 3 needs to say how large n is, which I could only found in the Appendix.  -The sentence ending on line 232 is missing a period.  -On line 322, 'in case of' should be 'in the case of'  minor suggestions for appendix:  -Section 4 of the experiments refer to 'badges' of experiments, this should probably be 'batches' -There is a space missing in the description of the second set of experiments -Tables 5,6, the meaning of, e.g., 'k in 1000', is not clear.  -In Figure 1, for logistic regression, it is very difficult to see the line for theano.   Originality ----------- The the originality of the work is to show that a combination of two well-known ideas, Ricci calculus and automatic differentiation, can result in faster, smaller computational graphs that can compute higher order derivatives.   Significance ------------ The core idea here has much potential for the machine learning community at large, but the experiments seem more like preliminary results and could be much stronger.   Post-rebuttal update =============================== While the rebuttal does clear up certain questions, the paper still needs work on its empirical motivation. Though I believe this work has great potential, my assessment of the paper remains the same for the following primary reason:  Any notion of including additional experiments (which is also brought up by another reviewer) to better motivate the work is completely absent in the rebuttal. While the authors argue that their synthetic experiments on three toy problems are enough, they are quite far removed from realistic problems where autodifferentiation with higher order methods would actually be useful. If the authors are going to evaluate on problems that are trivial to solve with typical second order approaches, they need to compare with the typical baseline approach and not just an auto-diff strawman that most people wouldn't use to solve the problem.   In further revisions, I would suggest that the authors seriously consider the following:   1) Formally present the cost of creating the computational graph. While it may seem insignificant to you, the reality is that many second order methods are derived and implemented directly, and not with auto-diff, and so this is the overhead that matters to many people.   2) Include the exact experimental setup for generating the benchmarks. Running time *can* in fact depend on whether the data is synethetic or not; oftentimes real data is sparse or structured, which is often exploited when people implement second order methods. The auto-diff runtime may not be so great if it doesn't exploit various cost-saving structural shortcuts that are common to use in practice.   3) Alternatively, instead of comparing to problems which are trivial to solve without auto-diff, evaluate in a setting where people actually use second order auto-differentiation (e.g. double backprop).


Differentially Private Change-Point Detection
The change-point detection problem seeks to identify distributional changes at an unknown change-point k ⇤ in a stream of data. This problem appears in many important practical settings involving personal data, including biosurveillance, fault detection, finance, signal detection, and security systems. The field of differential privacy offers data analysis tools that provide powerful worst-case privacy guarantees. We study the statistical problem of change-point detection through the lens of differential privacy. We give private algorithms for both online and offline change-point detection, analyze these algorithms theoretically, and provide empirical validation of our results.

This paper studies how to detect a change in a sequence of data while satisfying differential privacy. In this problem, a sequence of n data points is split into points x_1, x_2, . . ., x_{k-1} drawn iid from known distribution P_0 and points x_k, x_{k+1}, . . ., x_n drawn iid from known distribution P_1, and the goal is to recover (a value close to) the unknown k.  The paper first considers the offline case where all n points are known to the algorithm in advance. In the first subcase, they assume P_0 and P_1 have a uniform bound on log(P_1(x)/P_0(x)) from both above and below. Under this assumption, they prove a finite-sample accuracy guarantee for a non-private MLE and a similar guarantee for the private version. In the second subcase, they relax this uniform assumption to one that bounds the log probability ratio everywhere except for a region of small probability mass and obtain slightly worse guarantees. Both algorithms adapt a well-known change-point detection technique and get privacy by adding a very simple (and natural) differentially private protocol on top of it, Report Noisy Max.  The more difficult online case is correspondingly more complex. The algorithmic idea is simple enough -- use an existing sparse vector-type algorithm to track whether a change is likely, then call the offline method to compute an estimate on the data so far. This setting appears to require the same uniform assumption as subcase 1 for the offline version and obtains a similar-looking guarantee except for a log dependence on the number of points seen, and some requirements on the threshold T.  Finally, the paper conducts some simple experiments for the case when P_0 and P_1 are both Bernoulli. These largely bear out the theory.  I did not check the proofs in the Appendix. I'm familiar with the differential privacy literature and not familiar with the change point detection literature.  Positive points (no order): This paper provides, to the best of my knowledge, the first differentially private solution for change-point detection. Its algorithms are simple and intuitive from a differential privacy perspective, and the accuracy guarantees "look right". The fact that these guarantees are (taking the paper's claim here at face value) the first finite-sample accuracy guarantees for change-point detection, even absent privacy, is a nice contribution as well. The paper is generally well-written and easy to follow.  Negative points (in order of decreasing importance): 1. As stated above, I'm not very familiar with the change point detection literature, but looking at a few papers (e.g. https://arxiv.org/pdf/1306.4933.pdf) suggests that "change point detection" often refers to something much more general than the problem presented here, where it is assumed that only one change takes place and both the before distribution P_0 and after distribution P_1 are known. Similarly, the results presented require further assumptions (e.g. the bounds on A) to hold. This is a bit disappointing given the generality of the original claims ("provide the first accuracy guarantees for the MLE from a finite sample"). Signposting these restrictions earlier would help. Similarly, the language in lines 109-111 suggests that the algorithms should be able to identify problem instances where *no* change occurs, but this doesn't seem to be the case.  Along the same lines, I think the paper is a little too blithe about the assumptions on delta(ell) and A_delta. The assumption on delta(ell) almost looks to me like a stealth smoothness assumption (neither distribution is "spiky"). Examples of P_0 and P_1 and their delta(ell) and A_delta (so, an expansion of lines 167-8 to other distributions) would be nice.  2. The online section (4) feels under-explained. In particular, there is little guidance on actually choosing T -- the T_L and T_U in Theorem 6, being defined in terms of k*, don't help. There appears to be some information on this in Appendix D -- a paragraph or two to this effect in the main body would he very helpful. Even if choosing T is actually hard in some way, it's good to be honest about it. Right now it makes me suspicious.  3. The experiments don't seem very useful to me. The Bernoulli case is close to the easiest P_0 and P_1 possible, and the most interesting case (misspecified change) still tests a fairly small error (misspecifying P_1 as Bernoulli 0.4 instead of 0.8, which is even easier to detect). I have similar misgiving about the Gaussian case -- both seem like "toy" distributions" that I would expect good performance on. It would be useful to see distributions for which the algorithms perform poorly, or at least less well-behaved or noisy distributions. I think as is the page this section takes up would be better spent on some of the other negative points mentioned here. This complaint goes hand in hand with my misgivings about bounding delta(ell) and A_sigma -- are these assumptions so strong that only very "nice" distributions satisfy them?  4. Minor points: should be "extend our offline results to the online setting" in lines 79-80, "the a single entry" in line 217.  Overall: This paper offers the first differentially private solution to (a subclass of) change point detection problems. Subject to certain assumptions, it obtains accuracy bounds that depend polynomially on "sensitivity" properties of the generating distributions involved, for both offline and online cases. I think these dependencies are for the most part necessary for a (nontrivial) differentially private solution, but I think the paper suffers from insufficient explanation of what A, C, A_delta, C_M, etc. actually look like (points 1 and 3). Similarly, the online algorithm has a fuzzy dependence (in terms of actually running the thing, not the accuracy guarantee) on T (point 2).   For these reasons, I vote marginal accept. However, I think many of the problems I've noted are not too hard to solve, unless the assumptions are actually very strong and clarifying them reveals the weakness of the paper (which I don't expect to be true). I therefore look forward to the authors' rebuttal. Sample text addressing the concerns above would be a plus.  ---Update After Response--- After reading the authors' response, I have kept my score at a 6. This is not because the response was bad -- it clarified several of my questions. However, I was less convinced by responses to the questions of other authors, particularly Reviewer 1's concerns about how data-dependent the privacy guarantee is, which I seem to have missed on my first read. I think as is the paper is not very clear about this and the response does not help. This is a drawback for a differential privacy paper, even if it's otherwise a promising extension of differential privacy to an interesting problem. I just feel that the current version of the paper is not quite ready enough to merit a clear accept for me.


Improving Simple Models with Confidence Profiles
In this paper, we propose a new method called ProfWeight for transferring information from a pre-trained deep neural network that has a high test accuracy to a simpler interpretable model or a very shallow network of low complexity and a priori low test accuracy. We are motivated by applications in interpretability and model deployment in severely memory constrained environments (like sensors). Our method uses linear probes to generate confidence scores through flattened intermediate representations. Our transfer method involves a theoretically justified weighting of samples during the training of the simple model using confidence scores of these intermediate layers. The value of our method is first demonstrated on CIFAR-10, where our weighting method significantly improves (3-4%) networks with only a fraction of the number of Resnet blocks of a complex Resnet model. We further demonstrate operationally significant results on a real manufacturing problem, where we dramatically increase the test accuracy of a CART model (the domain standard) by roughly 13%.

The authors introduce ProfWeight - a method for transferring knowledge from a teacher model to a student model. A "confidence profile" (taken from classification layers placed throughout the network) is used to determine which training samples are easy and which are hard. The loss function for the student model is weighted to favor learning the easier samples. The authors test this method on CIFAR10 and a real-world dataset.  Quality:  The idea presented by this paper is interesting and well-motivated. The method and results could be presented with more clarity, and the paper could benefit from some additional empirical analysis. However, overall the quality of the experiments are good.  Clarity:  Most of the section are straightforward. However, the method section is confusing and is not strategically organized. Additionally, there are some notational issues (see small notes).  Specifically, the paper is missing a description of ProfWeight in the text. The components of the algorithm (weights, AUC/neural net) are described in Section 3, but there is no overview of the exact steps of the algorithm (learn a simple model, use error of simple model and the AUC/neural net to learn weights, retrain simple model with weights). A reader can only learn this overview from Algorithm 1, which is hard to parse because of all the notation. Similarly, Algorithm 3 should contain more description in the text.  Originality:  The idea of student/teacher transfer with sample weighting is novel and interesting.  Significance:  Model compression and interpretability is an important problem, and this paper presents a promising new method. It will be of interest to the community at large.  Overall:  I vote to accept this paper, since the method proposed is novel and seems promising. I do believe the methods section of this paper should be clarified, and more empirical results on other datasets would strengthen the paper.  Detailed comments:  Section 3.2 is a bit terse. The left-hand side of equation 2 is difficult to parse (what are the parameters \theta referring to? the parameters of the simple model (in which case, how does this differ from M'), or the complex model parameters?) which makes this section confusing.  Additionally, is there a guarantee that Algorithm 3 converges? It would be good to include such a guarantee since (1) frames ProfWeight as an optimization problem.  I wonder if your CIFAR results are influenced by the training set split? A model trained on 30,000 samples or 20,000 samples overfits quite a bit, as you note in the paper. It may be useful to try these experiments on a larger dataset like SVHN or ImageNet where a split dataset would still contain hundreds of thousands of samples.  Finally, I really appreciate the real-world experiment with semi-conductor manufacturing. It is good to see a method tested in real-world settings.  Smaller comments:  - 16-17: some example citations would be good here - "The MAIN criticism of these models has been their lack of transparency" - this is a bold claim. - 31: "Small data settings" - It's unclear how model transfer, as you've described in this paper, would work in such a scenario. - 59: "The first function we consider, area under the curve (AUC) traced by the confidence scores, shows much improved performance." - what is the performance you're talking about? - In the description of the method (and in Algorithm 1), it may be good to use the term "layer" instead of "unit", since "unit" can mean many things in the context of neural networks. - It would be good to re-define all notation in Algorithm 1 (i.e. \beta, the arguments to \mathcal L, etc.) - Labels in 127-129 should be capitalized (e.g. as seen in Algorithm 2) - There is inconsistent notation, which makes Section 3 a bit difficult in some places. For example, the weights are a set (W) in 97, but a vector in Algorithm 1. Additionally, the inputs/labels should probably be notated as vector (e.g. in 1 - as far as I can tell - y represents all labels, and therefore should be a vector (and bolded)?) - 135: there's a reference to Line 7 of Algorithm 3, but no line numbers in Algorithm 3. Also, it would be good to repeat this step in the text, rather than making the reader refer to the algorithm block. - Algorithm 3: should define what N is - Algorithm step 4: missing a \gamma in the update weights equation? - 193: "The set of units U (as defined in Section 2.1) whose representations are used to train the probes are the units in Table 1 of the trained complex model." - I'm not sure what this means? - 196: you should explain here that set 1 is used for the complex model and set 2 is used for the simple model. I was confused by this split until I read further. - Technically, [13] did not invent distillation - [13] is a refinement of the method proposed by [9]. - You should include the following citation on your list of distillation methods: Tan, Sarah, et al. "Detecting bias in black-box models using transparent model distillation." arXiv preprint arXiv:1710.06169 (2017).


Mean-field theory of graph neural networks in graph partitioning
A theoretical performance analysis of the graph neural network (GNN) is presented. For classification tasks, the neural network approach has the advantage in terms of flexibility that it can be employed in a data-driven manner, whereas Bayesian inference requires the assumption of a specific model. A fundamental question is then whether GNN has a high accuracy in addition to this flexibility. Moreover, whether the achieved performance is predominately a result of the backpropagation or the architecture itself is a matter of considerable interest. To gain a better insight into these questions, a mean-field theory of a minimal GNN architecture is developed for the graph partitioning problem. This demonstrates a good agreement with numerical experiments.

A GNN (graph neural network) is a neural network whose input is a graph. This paper studies the problem of using a GNN to detect clusters in a graph drawn from the 2-groups SBM (stochastic block model: a popular model for random graphs with community structure).  Although it is already known how to optimally solve the SBM (i.e. find the hidden clusters), the GNN has the advantage of being a data-driven approach that does not rely on knowing the specific random model. Thus, it is of interest to analyze the performance of GNN on the SBM.  In the GNN architecture studied here, each layer has a node for each vertex in the graph. A node in the GNN is connected to its neighbors (according to the graph adjacency matrix) in the previous layer. Each node of the GNN stores a D-dimensional feature vector.  This paper analyzes the performance of an untrained GNN on the SBM. This means the GNN weights are chosen at random, the GNN runs in order to produce a D-dimensional feature vector for each vertex, and then these vectors are clustered via k-means in order to partition the vertices. The analysis is not rigorous but is a heuristic calculation based on the mean-field approximation from statistical physics. The derivation yields a conjectured threshold (depending on the SBM parameters) at which the GNN is able to detect the presence of clusters and obtain a partition that non-trivially correlates with the truth.  The authors include numerical experiments, showing that the analytically-derived threshold appears to agree reasonably well with the experimental performance of the untrained GNN. They also experimentally compare the untrained GNN with a trained one. They see that the training does not seem to significantly affect the detection threshold (at which point non-trivial recovery becomes possible) but training does significantly improve reconstruction accuracy once you're above the threshold.  I think that this paper constitutes a decent start to the rigorous study of GNN's but I'm unsure whether its contributions are enough to merit acceptance to NIPS. The results are somewhat modest because they only analyze the untrained GNN and the analysis is non-rigorous. Still, the result obtained seems to reasonably approximate the truth, and the experimental results of the paper are of value.  I suspect that the derivation in this paper may not quite give the correct result. To get a rigorously-correct (in the large-graph limit) analysis of the constant-degree SBM typically requires a self-consistency equation simliar to (7) whose state is an entire distribution (see e.g. https://arxiv.org/abs/1611.00814), whereas this paper seems to only keep track of the mean value and make a Gaussian approximation. Regardless, the computations in this paper are at least a crude approximation to the correct result.  EDIT: After reading the other reviews and the author feedback, I have changed my overall score from 5 to 6. The authors make a fair point that deriving a crude approximation to the threshold still has value. I still think it would be good to derive the exact expression for the threshold (in the large graph limit). I would suggest that the authors address whether they believe their derivation gives the exact threshold (it sounds like the answer is no?) and whether deriving the exact threshold (even non-rigorously) is within reach of established techniques from statistical physics (I suspect the answer should be yes). In any case, I have no objections to accepting the paper in its current form.


A Reduction for Efficient LDA Topic Reconstruction
We present a novel approach for LDA (Latent Dirichlet Allocation) topic reconstruction. The main technical idea is to show that the distribution over the documents generated by LDA can be transformed into a distribution for a much simpler generative model in which documents are generated from the same set of topics but have a much simpler structure: documents are single topic and topics are chosen uniformly at random. Furthermore, this reduction is approximation preserving, in the sense that approximate distributions -the only ones we can hope to compute in practice -are mapped into approximate distribution in the simplified world. This opens up the possibility of efficiently reconstructing LDA topics in a roundabout way. Compute an approximate document distribution from the given corpus, transform it into an approximate distribution for the single-topic world, and run a reconstruction algorithm in the uniform, single-topic world -a much simpler task than direct LDA reconstruction. We show the viability of the approach by giving very simple algorithms for a generalization of two notable cases that have been studied in the literature, p-separability and matrix-like topics.

This submission presents an approach to LDA topic reconstruction using word co-occurences.  Exploiting a relationship between LDA and multinomial mixture models (detailed in a separate, included paper), the authors provide a new algorithm for learning (a broad class) of separable topic models.  These results are tested on separable data and compared to Arora et al (2013)’s anchor words algorithm and a  Gibbs sampler.  I will disclaim that I am comparatively weak on the theoretical background for reviewing this paper.  I come from a background quite familiar with LDA and various approaches to estimation.  It may be worth noting for future readers that the abstract isn’t really clear that the goal here isn’t necessarily a practical algorithm as much as a theoretical contribution.  The quality of the paper seems high, although again I’m not particularly well suited to judge many of the theoretical results.   The paper considers two simulation settings, one that is favorable to the anchor words assumptions and one that is not.  In the favorable setting performance is comparable, but STA is faster.  In the unfavorable setting, STA is both faster and has much stronger performance.  From a practical perspective, what is lacking is any sense of whether this works at all on real data.  The anchor words algorithm actually works in real data and recovers semantically coherent topics.  Does the proposed algorithm work for actual documents (particularly those longer than 2 words)?  Does it scale well in the size of the vocabulary and when that vocabulary has a power-law distribution?  It also seems that much of the results of experiment one turns on the runtime of STA being better, but that turns a lot on the implementation of the algorithms (which one was used for anchor words?).  My reaction here may be due to the practical perspective I’m coming from, but it isn’t clear what the experiments would contribute for someone coming at this from a theory angle.  The paper is remarkably clear.  The authors do a nice job of signaling the work that each lemma and theorem will be doing to support the method.  While the build-up of the approach is clear there seems to be a significant leap to the actual algorithm.  It might help in Algorithm 1 to include references to the section where each step is described.  I think it would be difficult to implement without constantly referring back to parts of the paper, but it isn’t always clear where to go (e.g. I struggled down the source of the threshold in step 2, because it wasn’t clear what what would happen at p >=2 where the empirical fraction would be above 1).  Finally, it would really help if the appendix included further details of the experiment.  For example, (1) timing comparisons to Gibbs sampling are always a bit misleading because there is no clear point of termination, so it can be made to look arbitrarily bad by just running it arbitrarily long, (2) it isn’t clear which recover algorithm is used (Recover-KL, Recover-L2 etc.), (3) what is used for topic reconstruction in Gibbs? Is it a single pass of the Gibbs sampler or an average over draws (which is going to be substantially more accurate), (4) there is a curious line which is “Gibbs needs longer documents in order to work so we use lengths \ell \in {10,100}, and we keep the number of total tokens (and thus documents) fixed.”  How can you keep the number of total tokens and the number of documents fixed (I’m assuming relative to STA/Recover) if the length is different (because total tokens = number of documents * average length).    The work seems quite original to me and represents a substantial move beyond other spectral method of moments recovery algorithms in the literature.  It covers the interesting (p,t)-separable case.  I’d be curious to know how it compares to the generalization of the anchor words method provided in Huang, Fu and sidiropoulos (“Anchor-Free Correlated Topic Modeling; Identifiability and Algorithm”)  The significance of this papers substantially by community.  Amongst the community of applied users of LDA, I suspect it will have little significance at the moment because the algorithm as stated throws away essentially all the data in the documents, it makes extremely strong assumptions (like the symmetric hyper parameter which we know from Wallach et al 2009 is important) and there is no evidence it works on real applications.  It seems to represent a really new and interesting direction for theory but unfortunately I’m less well-suited to judge that.  ====================== Response to Authors Reply: Thanks for the reply- this was really helpful for clarifying the details of the evaluations.  I am sympathetic to the difficulties of doing the Gibbs runtime evaluations and I think being transparent about that in the text will help clarify to others why it is so difficult.


A General Method for Amortizing Variational Filtering
We introduce the variational filtering EM algorithm, a simple, general-purpose method for performing variational inference in dynamical latent variable models using information from only past and present variables, i.e. filtering. The algorithm is derived from the variational objective in the filtering setting and consists of an optimization procedure at each time step. By performing each inference optimization procedure with an iterative amortized inference model, we obtain a computationally efficient implementation of the algorithm, which we call amortized variational filtering. We present experiments demonstrating that this general-purpose method improves performance across several deep dynamical latent variable models.

Thank you for your thorough response.  I appreciate the additional analysis and I think it will definitely make for a stronger paper.  I also appreciate the run time discussion --- I was confused about the computational costs of the amortization, and the author response cleared that up for me.  I am still a bit confused about why AVF would be less susceptible to local minima (and more likely to use the latent codes) than the baseline comparisons, and it would be great to include some discussion about this improvement.  I have increased my score.   ------------------ The authors present a method for fitting dynamic latent variable models with a variational inference objective.  They use a variational family factorized sequentially to approximate the posterior over the latent variables. They describe the inference procedure using this posterior sequence approximation, which involves solving T optimization problems at each step of the sequence, approximating filtering.  Learning the global parameters requires performing this as an inner loop.  To reduce computation, they use iterative inference models as an amortized inference scheme.  They present experimental comparisons to baseline methods in a variety of domains involving sequence modeling.    *Clarity*:  The presentation is relatively clear --- the introduction clearly states the problem to be solved and approach they take to solve it. Some aspects were confusing, particularly equations 5 and 13 (and further details of iterative inference mode).  *Quality*: The paper appears to be technically correct, and the proposed method is tested in a variety of domains.  The idea of using a sequentially structured inference network is interesting.  I think the results section would be more compelling if alternate inference network schemes were compared (or if a baseline contains an amortization strategy, how does the structure compare?).  *Originality+Impact*:  The approach showed big empirical gains in some real data examples.  I think the potential impact of this method is contingent on some practical aspects, such as run time.    *Questions and Comments*:  - I am a little confused about the term empirical priors, as used in line 82.  Inference models of the form in Eq (4) have difficulty with which particular distribution?  Is the idea that dynamical priors (or even hierarchical priors) induce some structure that is ignored by a generic inference network?  Is the type of structure that the amortized filtering inference approach exploits?  - Iterative inference models are a big part of the proposed method, however they are not well described in this manuscript.  Could some part of section 2.3 explain Eq (5) in greater detail.    - I am a bit confused about the units used in the experiment log-likelihood reporting.  Do the experiments report the probability of test sequences (having marginalized out the latent variables)?    - How does optimization differ between the using the existing filtering methods vs AVF?  Is it a matter of optimization (e.g. would the existing baselines given more time or tuning rise to the performance of AVF?  Is AVF uniformly better?    - For the video modeling --- what is driving this big improvement?  Is the posterior approximation more accurate?  In general, does the use of the AVF framework enable more expressive posterior approximations than the baselines?  Or does it afford more efficient inference?  Or both?    - How much does the amortization afford you?  If the same variational approximation were used within a stochastic variational inference framework (say for a small dataset), we would expect it to outperform an amortized framework in terms of ELBO objective value.  How does including the amortization structure affect the final model fit?    - For learning global parameters, would a filter + smoothing approach be sensible?  Can this approach be used to update global parameters using information from the entire sequence to do learning?


Heterogeneous Multi-output Gaussian Process Prediction
We present a novel extension of multi-output Gaussian processes for handling heterogeneous outputs. We assume that each output has its own likelihood function and use a vector-valued Gaussian process prior to jointly model the parameters in all likelihoods as latent functions. Our multi-output Gaussian process uses a covariance function with a linear model of coregionalisation form. Assuming conditional independence across the underlying latent functions together with an inducing variable framework, we are able to obtain tractable variational bounds amenable to stochastic variational inference. We illustrate the performance of the model on synthetic data and two real datasets: a human behavioral study and a demographic high-dimensional dataset.

## [Updated after author feedback] Thank you for your feedback. I am happy to see the updated results and I hope you will add them to the paper.   While I agree with the other reviewers that the individual parts of the idea are not new, I find the combination elegant - a whole that is greater than the sum of its parts. I will, therefore, keep my score.  ## Summary The paper presents an extension to multi-output Gaussian processes enabling them to deal with heterogeneous outputs specified by different distributions, thus requiring different likelihood functions. By assuming that each output is completely specified by a distribution, the task is to infer the parameters of the distributions. Each parameter is modelled as the (non-linear transformation of the) output of a latent parameter function f, which itself is a linear combination of Q latent functions u. The main novelty is then to impose a multi-output GP prior on f, allowing the model to learn correlations between all parameters for all outputs. The authors introduce inducing variables and derive bounds allowing for stochastic variational inference, thus making the model applicable to large datasets.  ## Quality The paper is of high technical quality. Some of the derivations are left out or moved to the supplementary material. Is some sense this is justified, as they follow previous work which is adequately cited. However, a pointer to the gradient specification in the appendix would be appreciated.  The authors present four experiments for empirical analysis. I hate to be that reviewer asking for additional experiments, but I think it would be interesting to see how the model performs on a dataset with a large number of outputs. The maximum number of outputs evaluated is three, whereas a high-dimensional input problem is considered. A high-dimensional output problem is, in my opinion, at least as interesting. Using synthetic data as in the first problem would be just fine.  In table 1 and 2, I cannot find information on how the uncertainties were obtained. That should be included. Also, I am unsure what is meant by the "Global" column. How does the covariance function look here?  ## Clarity The paper is clear and well-written. The authors have clearly spent time on the writing and the structure. The main problem and contribution are both clearly stated.  I really like the paragraph headlines in section 2.2 - they provide a structured and easy to follow overview.  One suggestion is to consider making a sketch of the setup. With both latent functions and parameter functions, things quickly get complex with lots of subscripts and indices. Not that the text is unclear, it is just a complex problem to wrap your head around.  ## Originality Related work section is clear and concise. I could not find papers that have been missed.  To my knowledge, a multi-output GP model capable (in principle) of handling any type and number of outputs has not been proposed before.  ## Significance The paper addresses an important issue, namely learning correlations between heterogeneous outputs using GPs. The authors further make it scalable to large datasets by casting it in the stochastic variational inference framework. This method is an important contribution to the field of GPs.  ## Comments line 100: "there is no always" -> "there is not always". line 170: I believe the mean should be \mu_{u_q} instead of u_q? line 291: "slighty difference" -> "slight difference"


Generalized Zero-Shot Learning with Deep Calibration Network
A technical challenge of deep learning is recognizing target classes without seen data. Zero-shot learning leverages semantic representations such as attributes or class prototypes to bridge source and target classes. Existing standard zero-shot learning methods may be prone to overfitting the seen data of source classes as they are blind to the semantic representations of target classes. In this paper, we study generalized zero-shot learning that assumes accessible to target classes for unseen data during training, and prediction on unseen data is made by searching on both source and target classes. We propose a novel Deep Calibration Network (DCN) approach towards this generalized zero-shot learning paradigm, which enables simultaneous calibration of deep networks on the confidence of source classes and uncertainty of target classes. Our approach maps visual features of images and semantic representations of class prototypes to a common embedding space such that the compatibility of seen data to both source and target classes are maximized. We show superior accuracy of our approach over the state of the art on benchmark datasets for generalized zero-shot learning, including AwA, CUB, SUN, and aPY.

POST REBUTTAL COMMENTS: I found the rebuttal effective, as it answered my main concerns. Therefore I upgraded my score from 6 to 7 as I believe the authors would improve the paper based on the rebuttal. I still think the title is overselling, and should be modified. Because the network architecture is standard, and the novelty is in the regularization and temperature calibration scheme.  ---------------------------------------------  The paper deals with zero-shot learning (ZSL). Which is about recognizing target classes with no training samples. Instead a semantic description is provided, which allows to transfer knowledge from source classes, to classify target classes. Specifically this paper focuses on improving a recently introduced “generalized” ZSL framework (Xian, CVPR 2017), where samples from both the source and target classes appear on test time. This framework is challenging, because during inference, the common ZSL models are mostly biased toward predicting source classes, rather than target classes.  This paper proposes a novel idea to tackle this problem, arguing that in many cases the semantic description of the target classes is known during the training phase. Which can be used as additional signal during the training phase.   To exploit this signal, the authors propose to use a network model that outputs a probability distribution (using a softmax). With that, they can calibrate the output distribution using two components: (1) Temperature calibration (Hinton 2015, Guo 2017) (2) A new regularization term for ZSL, which tries to minimize the entropy of target class prediction on training samples (of source classes). This term uses the target-class-semantic-description to produce the target class predictions. The regularization term is motivated by the conjecture that target classes may be similar to some source classes.  The experiments show that using the proposed technique clearly improves the prediction accuracy, with respect to only having a probability distribution output. Comparing to other ZSL approaches show similar or improved accuracy.   #Summary: I like the idea and the proposed regularization term, which are novel. But I gave a marginal score because quite a few details are vague or missing.  Also, some important baselines are missing. I don’t think it should affect acceptance or rejection, because the performance of those baselines are published, and tested with the same datasets this paper uses. Yet, if the paper is accepted, it is important to relate to them.   #Strengths: ##Novel ideas:  (1) Take advantage of the fact that the semantic description of target classes is known during training.  (2) The proposed entropy based regularization term, which appears to be effective. (3) Comparing to other ZSL approaches show similar or improved accuracy.  #Weakness: ##The clarity of this paper is medium. Some important parts are vague or missing.  1) Temperature calibration: 1.a) It was not clear what is the procedure for temperature calibration. The paper only describes an equation, without mentioning how to apply it. Could the authors list the steps they took?  1.b) I had to read Guo 2017 to understand that T is optimized with respect to NLL on the validation set, and yet I am not sure the authors do the same. Is the temperature calibration is applied on the train set? The validation set (like Guo 2017)? The test set?  1.c) Guo clearly states that temperature calibration does not affect the prediction accuracy. This contradicts the results on Table 2 & 3, where DCN-T is worse than DCN.  1.d) About Eq (5) and Eq (7): Does it mean that we make temperature calibration twice? Once for source class, and another for target classes? 1.e) It is written that temperature calibration is performed after training. Does it mean that we first do a hyper-param grid search for those of the loss function, and afterward we search only for the temperature? If yes, does it means that this method can be applied to other already trained models, without need to retrain?  2) Uncertainty Calibration From one point of view it looks like temperature calibration is independent of uncertainty calibration, with the regularization term H. However in lines 155-160 it appears that they are both are required to do uncertainty calibration. (2.a) This is confusing because the training regularization term (H) requires temperature calibration, yet temperature  calibration is applied after training. Could the authors clarify this point? (2.b) Regarding H: Reducing the entropy, makes the predictions more confident. This is against the paper motivation to calibrate the networks since they are already over confident (lines 133-136).  3) Do the authors do uncertainty calibration on the (not-generalized) ZSL experiments (Table 2&3)? If yes, could they share the ablation results for DCN:(T+E), DCN:T, DCN:E ?  4) Do the authors do temperature calibration on the generalized ZSL experiments (Table 4)? If yes, could they share the ablation results for DCN:(T+E), DCN:T, DCN:E ?  5) The network structure:  5.a) Do the authors take the CNN image features as is, or do they incorporate an additional embedding layer? 5.b) What is the MLP architecture for embedding the semantic information? (number of layers / dimension / etc..)  ##The paper ignores recent baselines from CVPR 2018 and CVPR 2017 (CVPR 2018 accepted papers were announced on March, and were available online). These baseline methods performance superceed the accuracy introduced in this paper. Some can be considered complementary to this work, but the paper can’t simply ignore them. For example:  Zhang, 2018: Zero-Shot Kernel Learning Xian, 2018: Feature Generating Networks for Zero-Shot Learning Arora, 2018: Generalized zero-shot learning via synthesized examples  CVPR 2017: Zhang, 2017: Learning a Deep Embedding Model for Zero-Shot Learning   ## Title/abstract/intro is overselling The authors state that they introduce a new deep calibration network architecture. However, their contributions are a novel regularization term, and a temperature calibration scheme that is applied after training. I wouldn’t consider a softmax layer as a novel network architecture. Alternatively, I would suggest emphasizing a different perspective: The approach in the paper can be considered as more general, and can be potentially applied to any ZSL framework that outputs a probability distribution. For example: Atzmon 2018: Probabilistic AND-OR Attribute Grouping for Zero-Shot Learning Ba 2015: Predicting Deep Zero-Shot Convolutional Neural Networks using Textual Descriptions   Other comments: It will make the paper stronger if there was an analysis that provides support for the uncertainty calibration claims in the generalized ZSL case, which is the focus of this paper.  Introduction could be improved: The intro only motivates why (G)ZSL is important, which is great for new audience, but there is no interesting information for ZSL community. It can be useful to describe the main ideas in the intro. Also, confidence vs uncertainty, were only defined on section 3, while it was used in the abstract / intro. This was confusing.  Related work: It is worth to mention Transductive ZSL approaches, which use unlabeled test data during training, and then discriminate this work from the transductive setting. For example: Tsai, 2017: Learning robust visual-semantic embeddings. Fu 2015: Transductive Multi-view Zero-Shot Learning  I couldn’t understand the meaning on lines 159, 160.  Lines 174-179. Point is not clear. Sounds redundant.  Fig 1 is not clear. I understand the motivation, but I couldn’t understand Fig 1.


Visual Reinforcement Learning with Imagined Goals
For an autonomous agent to fulfill a wide range of user-specified goals at test time, it must be able to learn broadly applicable and general-purpose skill repertoires. Furthermore, to provide the requisite level of generality, these skills must handle raw sensory input such as images. In this paper, we propose an algorithm that acquires such general-purpose skills by combining unsupervised representation learning and reinforcement learning of goal-conditioned policies. Since the particular goals that might be required at test-time are not known in advance, the agent performs a self-supervised "practice" phase where it imagines goals and attempts to achieve them. We learn a visual representation with three distinct purposes: sampling goals for self-supervised practice, providing a structured transformation of raw sensory inputs, and computing a reward signal for goal reaching. We also propose a retroactive goal relabeling scheme to further improve the sample-efficiency of our method. Our off-policy algorithm is efficient enough to learn policies that operate on raw image observations and goals for a real-world robotic system, and substantially outperforms prior techniques.

The paper introduces an approach for learning general purpose skills by combining learning a latent representation of images with goal conditioned policies. A visual representation of raw observations is learnt using a VAE model that can also be used to generate arbitrary goals for a goal conditioned policy. The resulting learnt policy can operate with raw image observations in physical control tasks. The framework can also be interpreated as a generalization of Hindisght Experience Replay for control tasks.   A latent variable model is used to learnt a representation of raw image inputs based on VAE framework. The authors motivate that such a model not only learns a good representation of raw inputs, but the prior on the latent variables can also be used to generate random goals for learning a goal conditioned policy. The proposed latent variable model is used for several objectives, specifically to compute arbitrary reward functions based on distances in the latent space between the current state and randomly generated goal. The overall RL objective is trained alongside with a \beta-VAE objective._x000b__x000b_Even though the paper is trying to motivate an interesting latent variable model that can handle raw images for control tasks, there are several aspects of the paper which make the overall research contribution complex. The paper proposes several objectives, and it is not clear why the proposed approach exactly works better. I also do not think the proposed method of learning a general purpose goal conditioned policy, as motivated in the paper, is indeed what it claims to be.   _x000b_The experimental results demonstrated are not fully satisfying. Even if the authors compare to several baselines including HER, they mention modifications being made to the baseline algorithms. It would be been useful to see how the proposed method works on other tasks, instead of simply on the Reacher and Pusher baselines, along with a comparision with the original baselines (e.g HER uses similar experiment setups with Pusher and Reacher).  The range of experimental results is very limited, and the modifications made to the baselines raises questions about reproducibility of these results. _x000b__x000b_Another confusing aspect is the use of TD3 - why is TD3 used in the first place instead of standard off-policy RL algorithms for control tasks? Was there a specific reason to use TD3?_x000b__x000b_Overall, the paper introduces an interesting idea to use a latent variable model to achieve several objectives, but fails to sufficiently demonstrate the proposed approach experimentally on a range of tasks.  The limited set of experimental results are not sufficiently convincing, and it is not exactly clear how the proposed approach can be implemented or can be made to work in practice. The paper is trying to motivate an interesting idea, but may not be useful for practitioners in the field for how to use the proposed approach. More detailed discussion of the algorithm and implementation details are required. _x000b__x000b_I suggest the authors should make it more clear how the experiments with the proposed method can be implemented and reproduced in practice._x000b_  Quality : Good.  Clarity : Average. The paper is at times difficult to follow as it tries to motivate the proposed approach from several perspectives, but does not provide enough intuitions or experimental demonstration of how they are achieved. _x000b_ Originality : Slightly above average_x000b__x000b_Significance : Average.  Overall Rating : Marginally above acceptance level. I think the paper is introducing an interesting approach for raw image based control tasks, but does not still provide sufficient experimental results to demonstrate it. However, I think the overall research contribution is novel and maybe considered for acceptance.


On the Dimensionality of Word Embedding
In this paper, we provide a theoretical understanding of word embedding and its dimensionality. Motivated by the unitary-invariance of word embedding, we propose the Pairwise Inner Product (PIP) loss, a novel metric on the dissimilarity between word embeddings. Using techniques from matrix perturbation theory, we reveal a fundamental bias-variance trade-off in dimensionality selection for word embeddings. This bias-variance trade-off sheds light on many empirical observations which were previously unexplained, for example the existence of an optimal dimensionality. Moreover, new insights and discoveries, like when and how word embeddings are robust to over-fitting, are revealed. By optimizing over the biasvariance trade-off of the PIP loss, we can explicitly answer the open question of dimensionality selection for word embedding.

In this work a Pairwise Inner Product Loss is developed, motivated by unitary invariance of word embeddings. It then investigates theoretically the relationship between word embedding dimensionality to robustness under different singular exponents, and relates it to bias/variance tradeoff. The discovered relationships are used to define a criterion for a word embedding dimensionality selection procedure, which is empirically validated on 3 intrinsic evaluation tasks.  The PIP loss technique is well motivated, clear, and easy to understand. It would be interesting to see this technique applied in other contexts and for other NLP tasks.  The paper is clearly written, well motivated, and sections follow naturally. Only section 5.2 would benefit from some more details (which are deferred to the appendix, presumably due to space constraints), and the Figures are very small. To improve the latter point, I recommend increasing font sizes of axis labels and ticks, as well as legend, and line width.  The work is original and provides a new theoretical view on the robustness of word embeddings. Perhaps the title "On Word Embedding Dimensionality" is a bit too broad as the work chooses a definite route, even though it reveals connections between several existing approaches.  A potential criticism of this work is that it empirically only investigates word embedding dimensionality using intrinsic evaluation (though on 3 tasks), but not extrinsically / in a downstream NLP task. This is even though understanding the effect of word embedding dimensionality on downstream models and applications is one of the major motivations laid out in the Introduction.   Second, it is acknowledged that empirical validation carries "the risk of selecting sub-optimal dimensionalities" (line 37), but a similar caveat is missing for the proposed method "We can explicitly answer the open question of dimensionality selection for word embedding" (line 10-11). It should be kept in mind that also the proposed method (even though theoretically motivated) also only minimises an _expected_ quantity, and that it is also prone to the same risk of choosing sub-optimal dimensionality as a purely empirical selection.   Related to the previous point - another potential criticism is that the PIP Loss metric does not take into account word frequency. Less frequent words will have embeddings with more variance than frequent words. But by Zipf's Law, most words are rare, and giving all words the same contribution in the PIP loss will result in increased variance of the PIP loss when word embeddings are fitted on different corpora. This is speculative, but perhaps it is worth down-weighting infrequent word dimensions in the PIP loss? To investigate this possible phenomenon one could fit vectors either other datasets than text8, or different portions of text8, and observe confidence intervals.  Minor comments: - Line 227: could you explain why the square root term is dominated by (\lambda_k - \lambda_{k-1})^{-1} - Could you expand on line 166 (third line)? - Line 227 (end) -- should the exponent be -2? - to avoid misinterpretation, one could be more explicit in which exact term is plotted in Figure 1a. - line 254: the relative gap to the optimal dimension would be more informative - line 250: Are the chosen dimensionalities really exactly the optimal ones also empirically? It would be better if this was more concrete: dimensionalities chosen with PIP loss minimisation are generally in line with empirical validation, but not exactly identical? - line 267: could you elaborate on this?


Negotiable Reinforcement Learning for Pareto Optimal Sequential Decision-Making
It is commonly believed that an agent making decisions on behalf of two or more principals who have different utility functions should adopt a Pareto optimal policy, i.e. a policy that cannot be improved upon for one principal without making sacrifices for another. Harsanyi's theorem shows that when the principals have a common prior on the outcome distributions of all policies, a Pareto optimal policy for the agent is one that maximizes a fixed, weighted linear combination of the principals' utilities. In this paper, we derive a more precise generalization for the sequential decision setting in the case of principals with different priors on the dynamics of the environment. We refer to this generalization as the Negotiable Reinforcement Learning (NRL) framework. In this more general case, the relative weight given to each principal's utility should evolve over time according to how well the agent's observations conform with that principal's prior. To gain insight into the dynamics of this new framework, we implement a simple NRL agent and empirically examine its behavior in a simple environment.

** update ** I really enjoyed reading this paper, however I do not change my rating since the authors' response as well as the other reviews confirm the problems raised in mine. The first point is shared with reviewer 2: the proofs must be integrated into the paper, and the authors agree. The second point, also present in the second review, concerns how to define players' beliefs in practice: the authors propose to give adequate references dealing with this problem. Finally, I am not convinced by the authors' response to the use of the term "reinforcement learning".  Their answer includes the word "Bayesian" which indeed seems to me more appropriate for the work they propose.  ** summary ** In this paper the authors formalize the situation where two entities choose to cooperate sequentially in uncertainty  without having the same criterion to optimize or the same model for the considered system. The mathematical models of each entity are probabilistic, Markov in time,  partially observable and share the same actions and observations. After an introduction of the type of situation considered using a simple example of cake sharing,  the entities' models of the system are defined as POMDPs.  The definition of an Pareto-optimal policy is also given explicitly.  A first result (Lemma 2.) makes it possible to identify all the optimal policies  based on the expectations of gain of the entities under these policies and arbitrary weights. From this result and the definition of a POMDP having the weights mentioned above  as initial belief on the true model, the main result of this paper (Theorem 4) is derived. This result makes it possible to calculate the probability distribution on the actions  at each stage of the process (the Pareto-optimal strategy). The authors point out that optimization favours the entity whose model best explains  the observations obtained over time, which is verified in the following experimental section. The application proposed in the experimental part sets up an agent on a grid.  Each of the entities wants the agent to reach a different grid cell.  One entity believes that the system is deterministic,  while the other believes that agent transitions are random.  Simulations, governed by each model, are performed, described and explained.  ** quality ** This paper is well introduced, ideas are properly developed and interesting results are obtained. I personally find it interesting that entities have an interest in providing a model  that explains reality well (observations) because it eliminates the problem of cheating  (providing a model that one does not believe in order to maximize one's own utility  knowing the calculations made by the jointly-owned agent).  However, I find it extremely unfortunate that the proofs of the two main results  do not appear in the paper but only in the supplementary materials. As noted in the conclusion, "it employs only basic facts about convexity and linear algebra" thus the authors could at least have provided a sketch of proof  in order to make their work more self-contained.  Moreover it seems that there is a problem of vocabulary with this paper.  Since the models are known in advance and the actions are selected  from calculations based on these models  and not on estimates based on rewards received during tests,  this work is not part of Reinforcement Learning  but part of the domain of sequential decision making in uncertainty.  Finally this paper exceeds the page limit (two extra lines).  These three points must be changed if the paper is accepted.  ** clarity ** I found this paper fairly easy to read and understand. However the following remarks should be taken into account.  page 4:  The formalization of the POMDP model is associated with  (Smallwood "The optimal control of partially observable Markov processes over a finite horizon."  Operations research 1973) rather than  (Russel 2003).  < P( o_i | s_i ) > P( o_{i+1} | s_{i+1} ) Otherwise, no observation for s_{n+1}. Usually, i=0:n, no observation for s_0, but an initial belief b_0)  < P( s_{i+1} | s_i a_i ) > P( s_{i+1} | s_i, a_i )  < e.g. > \textit{e.g.}  page 5,6: The prior distribution is often call "initial belief".  page 7: < Figure > Figure 1.  The design of the point-based algorithms for POMDP solving is associated with  (Pineau "Point-based value iteration: An anytime algorithm for POMDPs." IJCAI 2003) rather than  (Ross 2008).  page 8: < is shown in the top/bottom frame > is shown in the right/left frame  ** originality ** To the best of my knowledge this paper is the first to propose a solution  for calculating optimal policies in the context where competing or cooperating entities  have a different modeling of the system. The authors have done a good bibliographical work to position their study  in the relevant literature. However, work has also been carried out with different models of uncertainty. Indeed entities may not be able to provide precise probabilities  about the system under consideration and this should be mentioned. The study is limited to the case where the entities  have a very precise guess about how works the system.  There are other models taking into account this imprecision  which is unfortunately often present in practice. For example work was produced for planning under Knightian Uncertainty, (Trevizan "Planning under Risk and Knightian Uncertainty." IJCAI 2007), with Dempster-Shafer belief Theory (Merigo "Linguistic aggregation operators for linguistic decision making  based on the Dempster-Shafer theory of evidence." International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems 2010), with Possibility Theory (Drougard "Structured Possibilistic Planning Using Decision Diagrams" AAAI 2014), and with Imprecise Probability Theory (Itoh "Partially observable Markov decision processes with imprecise parameters."  Artificial Intelligence 2007). Some of these references should be cited since this model is intended to work in practice.  ** significance ** The theoretical and application results are very interesting and make this work quite significant. It would be much more so if the proofs were present and the applications more diversified. Indeed the application is a problem with very few states and the entities' models of the system  are only MDPs and not POMDPs.


Zero-Shot Transfer with Deictic Object-Oriented Representation in Reinforcement Learning
Object-oriented representations in reinforcement learning have shown promise in transfer learning, with previous research introducing a propositional objectoriented framework that has provably efficient learning bounds with respect to sample complexity. However, this framework has limitations in terms of the classes of tasks it can efficiently learn. In this paper we introduce a novel deictic objectoriented framework that has provably efficient learning bounds and can solve a broader range of tasks. Additionally, we show that this framework is capable of zero-shot transfer of transition dynamics across tasks and demonstrate this empirically for the Taxi and Sokoban domains.

Summary of paper  This paper introduces Deictic Object-Oriented MDPs, which define MDP dynamics in terms of effects that are relative to a particular "central" object (i.e. deictically defined). A variant of DOORMAX is introduced for the efficient learning of DOOMDPs and their use to guide exploration for sample-efficient MBRL. Finally, experiments demonstrate that the deictic and object-oriented nature of the models can facilitate effective transfer between tasks that share the same dynamics schema.  Technical Quality  As far as I am able to tell the paper is technically sound. DOOMDPs and DOORMAX_D are well-defined and well-motivated. The experiments effectively support the claims that deictic predicates permit the compact representation (and correspondingly efficient learning) of domains with varying numbers of objects.  Clarity  I found the paper to be well-written. The OOMDP framework is highly technical and notation-heavy, and I appreciated the authors' efforts to aid in the reader's comprehension with concrete examples. One place I noticed where this might be improved was in Section 2.2. I found it very difficult to take in the entirety of the framework so quickly and was wishing for a running example to make components concrete. I wonder if the authors would consider interleaving the description of the Taxi domain with the definition of DOOMDPs? I don't think it would require a big change in space usage, but might help the reader make sense of the framework as it is described rather than having to circle back later and map the examples to the general description.  Originality  The paper is fully open about the fact that these findings build heavily upon existing theory and algorithms (particularly POOMDPs and DOORMAX). There is a novel contribution, though. First, conceptually, the paper threads a needle between a purely propositional system, which of course suffers when the number of objects can change, and a fully first-order system, which would negatively impact learnability. The observation that deictic predicates can yield some of the benefits of a first-order framework without incurring its costs is not entirely novel in the history of logic-based systems of course, but it is, as far as I know, novel in the context of OOMDPs. Second, the paper offers a particular instantiation of this idea, working through the details of how deictic predicates would be defined and learned.  Significance  I would say that the practical significance of this paper is rather low. Though appealing in many ways, OOMDPs are not to the point where they can be applied to problems of genuine interest -- they require far too much prior domain knowledge and do not fail gracefully when their biases are not reflected in the environment. Expanding the reach of OOMDPs from small, idealized toy problems to bigger, idealized toy problems is not going to change current practice.  However, I do think there is something to be said about exploring the edges of what a model with strong generalization can achieve. Rmax showed that sample efficient RL was possible, and this work continues a thread extending that analysis to richer and richer state spaces, where the structure is exploited to avoid dependence on the size of the state space, and now can be shown to facilitate generalization from easy problems to much larger and complex problems as the deictic operators allow for the expansion of the number of objects. So from a broader view of MBRL, this paper plants a theoretical flag that says "if you could have a model like this, you could enjoy some massive benefits." To the extent to which the paper teaches us about what we could possibly hope to achieve with MBRL, and what model properties might be necessary to get there, I think it may be of interest to the community.  Overall  I found the paper to be well-written and engaging. It represents a non-trivial extension to existing work. Its contributions are largely conceptual and theoretical and not likely to immediately impact practice. Nevertheless, the findings do expand what is possible in the context of provably efficient MBRL and explore a type of model-structure that may be more generally valuable.  ---After Author Reponse---  Having considered the other reviews and the author response, I still feel that this paper offers a novel and worthwhile contribution in formalizing the deictic setting as a middle ground between fully propositional and fully relational representations. I don't think this result would necessarily have a big impact practically (OOMDPs are hard to scale) or conceptually (I think it is already widely understood that representations like this are desirable) but I do think the results in this paper teach us something about the theoretical boundaries of efficient RL and that's worth having in the literature, so I still tend to vote for acceptance.


A Mathematical Model For Optimal Decisions In A Representative Democracy
Direct democracy, where each voter casts one vote, fails when the average voter competence falls below 50%. This happens in noisy settings when voters have limited information. Representative democracy, where voters choose representatives to vote, can be an elixir in both these situations. We introduce a mathematical model for studying representative democracy, in particular understanding the parameters of a representative democracy that gives maximum decision making capability. Our main result states that under general and natural conditions, 1. for fixed voting cost, the optimal number of representatives is linear; 2. for polynomial cost, the optimal number of representatives is logarithmic.

Summary:  - In this paper, the authors have presented a framework of mathematically exploring the trade-offs of representational democracy, when the decisions are taking in a hierarchical fashion: first a group of some size K selects a representative from among them using some group selection function \mu and then the representatives cast their votes on the issue at hand and the majority wins.  - Their framework assumes that each voter's competence is IID and a common assumption for most "interesting" results presented in the paper is that the group selection function \mu is such that there exists a group size for which the expected vote for the 'correct' side can be made > 1/2.  - The framework allows modeling the benefit of selecting the correct decision (which depends on the size of the population) and the cost of casting a vote (which depends on the size of the set of representatives) and different distributions for the competence.  - In the proofs, the authors first establish bounds for the "homogeneous" setting, where the size of all representative groups, save the last, are the same size and then show that the optimal group size would be "nearly" homogeneous: i.e. would differ by at most one (with some additional assumptions about the representative selection function).  - There are some surprising findings in the paper:       * if we ignore the cost of casting votes, then the optimal group size is a constant which depends on the group selection function and the underlying distribution of competence, but which is independent of the size of the population. This means that the number of representatives is linear in the size of the population.      * if the cost of casting a vote and the benefits are polynomial, then the optimal group size grows roughly as O(n / log n). This means that the number of representatives is logarighmic in the size of the population.  - The authors also generalize their framework to consider representatives which have to cast votes on multiple issues.  - Along the path, they answer an open question about what is the probability of attaining a majority when "competence" of voters is exactly 0.5 and show, somewhat surprisingly, that the probability of majority depends heavily on whether the number of voters is odd or even.  ----  This is an interesting paper which tackles an important problem. It proposes novel mathematical framework for analysis of optimal decisions in Representative Democracy. The paper is a "first-principles" approach to the problem and presents several interesting results which are begging for more exploration.  The presentation of the paper is logical. The examples and the intuitive explanations help in understanding the Theorems. However, such explanations dry up as we advance through the paper. For example, making the intuition behind polynomial costs of voting is found wanting. The results themselves are interesting and look worth exploring further. They involve innovative usage of combinatorial and probabilistic inequalities. However, I believe that the presentation of the proofs paper can be made simpler for the reader to follow.    In particular, abuse of notation in the paper is rampant, which makes the proofs very difficult for me to follow. A few helpful hints to improve the understanding of first time readers:    - Is \mu a function from a set to a number or from a natural number to a number? I understand that because of the IID assumption, it could be stated as either, but defining it as one and using it as the other appears sloppy and causes confusion.    - Why does n < cK_{*} cause Theorem 2 to hold automatically? Is there some property of \mu makes it such that selecting just one group will be optimal? The reason is probably simple, but I had difficulty in establishing it.    - Why is c > (-4/(ln (1 - 4eps^2)) * ln(32/9Aeps^2))? I was not obvious to me that the remaining part of the constant 'c' was negative in value. I do not think that these issues will end up invalidating the proof: I could see that the theorems would likely be true even with corrections (if any). Hence, my support for the paper.  The supplementary material contains interesting results for multi-issue voting, which are not mentioned at all in the main body? Could the proofs be relegated to the appendix and the statement of the results for multiple issues moved to the main text?  The theoretical bounds in the paper seems ripe for simulations which would open another window in the behavior of the algorithms in different settings: e.g. what kind of deviations may one see while moving away from the optimum group sizes/number of representatives. In contrast, the experimental results presented, though interesting and relevent, seem a bit sparse.  Finally, in the abstract, the authors pose this as a problem of creating ensembles of classifiers. However, beyond that, there are no references to machine learning methods. Looking at the list of references too suggests that this paper would perhaps be a better fit for an avenue which focusses on mathematical analysis of social choices.  Though I believe that the paper explores an important problem and presents interesting results and is likely to be of some general interest, due to the caveats with a potentially mismatched scope, somewhat weak empirical analysis, and some issues with the presentation decisions, I would overall recommend rejection.  Minor issues:   - line 228: "capturea" -> "captures"   - Example 5 (supplementary): What does (1/2)@c1 mean?   - Theorem 2: Why is A <= 1 w.l.o.g.? If A > 1 => the inequality \mu(K) <= 1 - A/K^a will never be satisfied for K = 1. Hence, A \in (0, 1] is a requirement.   ----  Update: Reading the other reviews, I am convinced that the paper makes an interesting contribution. Reading the response of the authors, I am convinced that the final version of the paper will address the few presentation issues. I still believe, though, that more motivation and empirical results could improve the paper significantly, but I can see that the issues are not easy to address. Hence, I have chosen to improve the score I give to the paper.


Hybrid Macro/Micro Level Backpropagation for Training Deep Spiking Neural Networks
Spiking neural networks (SNNs) are positioned to enable spatio-temporal information processing and ultra-low power event-driven neuromorphic hardware. However, SNNs are yet to reach the same performances of conventional deep artificial neural networks (ANNs), a long-standing challenge due to complex dynamics and non-differentiable spike events encountered in training. The existing SNN error backpropagation (BP) methods are limited in terms of scalability, lack of proper handling of spiking discontinuities, and/or mismatch between the ratecoded loss function and computed gradient. We present a hybrid macro/micro level backpropagation (HM2-BP) algorithm for training multi-layer SNNs. The temporal effects are precisely captured by the proposed spike-train level post-synaptic potential (S-PSP) at the microscopic level. The rate-coded errors are defined at the macroscopic level, computed and back-propagated across both macroscopic and microscopic levels. Different from existing BP methods, HM2-BP directly computes the gradient of the rate-coded loss function w.r.t tunable parameters. We evaluate the proposed HM2-BP algorithm by training deep fully connected and convolutional SNNs based on the static MNIST [14] and dynamic neuromorphic N-MNIST [26]. HM2-BP achieves an accuracy level of 99.49% and 98.88% for MNIST and N-MNIST, respectively, outperforming the best reported performances obtained from the existing SNN BP algorithms. Furthermore, the HM2-BP produces the highest accuracies based on SNNs for the EMNIST [3] dataset, and leads to high recognition accuracy for the 16-speaker spoken English letters of TI46 Corpus [16], a challenging spatio-temporal speech recognition benchmark for which no prior success based on SNNs was reported. It also achieves competitive performances surpassing those of conventional deep learning models when dealing with asynchronous spiking streams.

[Response to rebuttal: I am very impressed and greatly appreciate the amount of experiments that the authors managed to successfully perform during the rebuttal phase. The additional results strengthen the empirical part of the paper significantly and further improve the experimental rigor of the paper. I am not entirely sure if I would rate the submission in the top 50%, but I would be somewhat upset if it got rejected. I therefore raise my overall score to an 8. I greatly appreciate the additional datasets and analysis in Fig. 1 of the rebuttal. I also want to encourage the authors to take some of the minor issues that were raised, for instance the quality and usefulness of the figures, into account, but I understand that one page of rebuttal is not sufficient to address all issues and I feel like the authors correctly prioritized addressing the issues raised.]  The paper introduces a novel method for formalizing spiking neurons that allows for spiking neural networks which can be trained via stochastic gradient descent. In contrast to previous approaches to address this topic, the proposed model allows to incorporate temporal spiking dynamics while still maintaining differentiability w.r.t. the weights. This is achieved by formulating a rate-model (the macro-level) where the rate depends in a differentiable fashion on (approximate) leaky-integrate-and-fire dynamics (the micro level). In particular, the dependency of the rate on non-differentiable spikes is replaced by expressing the contribution of pre-synaptic spike trains on post-synaptic firing counts (and firing times) as a continuous function, termed “spike-train level post-synaptic potential” (S-PSP). This in turn allows for deriving closed-form gradient update equations for the weights. The method is evaluated on MNIST (fully connected network and convolutional neural network) and neuromorphic MNIST (fully connected network). On these experiments, the method outperforms competitor methods by a small margin. The widespread use of spiking neural networks in machine learning is arguably hindered by a lack of good training algorithms that scale well to large networks of neurons, particularly for the case of supervised learning. Due to the non-differentiable nature of spiking neurons, gradient descent cannot be applied in a straightforward way and several ways to alleviate this shortcoming have been proposed in the literature. The method presented in the paper addresses a timely and significant problem in an original and novel way that works well in the experiments shown. The writing is clear, with a well-written introduction and enough background for a machine learning practitioner to understand the method without having to resort to textbooks on spiking neuron models. I also greatly appreciate that the paper focuses clearly on machine learning applications and does not give in to the temptation of overloading the paper with additionally claiming biological plausibility (in a hand-wavy fashion). The authors also provide code to reproduce their results and also allowing for easy extension to experiment with additional ideas or set up additional experiments, which is definitely a plus. Besides minor issues, my main criticism is that the authors claim superior performance of the method even though they outperform competitors often by less than 0.2% accuracy on MNIST (comparing a single run with random initialization). While the claim of providing state-of-the-art results is technically not wrong, I would rather claim that the method performs on-par with competitor methods and further experiments (on more challenging datasets) are required for conclusive statements. However, I consider the latter overall a tolerable issue and suggest to accept the paper. My comments below are suggestions on how to further improve the quality and significance of the paper.  (1 Decoupling of micro level) Could the authors clarify their particular choice of Eq. (25)? Are there any obvious alternatives (o_j and o_k have a linear effect, but does it necessarily have to be multiplicative or is this a choice motivated by mathematical convenience)? Can the authors provide some quantitative analysis on how sensitive the approximation is w.r.t. the assumption that \hat{\alpha} does not change substantially (e.g. a histogram of how much \hat{\alpha} actually changes in typical experiments, or some comparison of numerical gradients vs. gradients based on the approximation)? (2 Error bars) Even though it has become widely accepted in the neural network community to report results on single runs, without sensitivity-analysis w.r.t. hyper-parameter settings, I want to actively encourage the community to re-introduce more experimental rigor by setting high standards. Whenever possible “with reasonable effort” it would be great if results were reported with “error bars” (e.g. mean/median accuracy over a small number of runs with an appropriate error-bar measure). While I do not strictly require the authors to re-run multiple instances of their experiments for this publication, I would consider a higher score if error bars were provided. (3 Additional datasets) MNIST (and neuromorphic MNIST) seems to have saturated as a data-set to compare different methods for spiking neural network training. It would be an interesting move to start evaluating on more challenging datasets. FashionMNIST should be really straightforward, but it would also be interesting to see more challenging problems on data-sets with temporal / event-driven signals in the long run. (4 Figures) The clarity of figures, particularly of figure captions could still be improved. Given the limited space of the manuscript you might want to push some parts of the derivation or some details of the experiments to the appendix. -) Fig. 1: there are lots of different kinds of arrows in the figure, a bit more space and text in the figure as well as the caption would help. Also, what does the “(+/-) Multiple Spikes” refer to? -) Fig. 2: What does the shaded yellow area correspond to? Why is there no re-set of the middle curve after the second post-synaptic spike? -) Fig. 4: The figure does not add much more clarity compared to the equations, but I think with a bit of polishing (and more space) it could add value to the paper.  (5 Control exp.) For the spiking conv net on MNIST, the results are better than for the MLP (Table 3 vs. 1) which is not too unexpected. However, for Table 3 data augmentation was used as well, but not for Table 1. It would be interesting to see how the result in Table 3 changes without data augmentation – i.e. how much of a performance improvement is gained by using a conv net over an MLP? (6 Convergence speed) One advantage of the proposed method is that it seems to converge faster to high accuracies compared to competitor methods. Since the final accuracies are almost on-par, it would be interesting to have more quantitative analysis on convergence behavior (i.e. convergence curves for the different methods). I understand, of course, that this might not be easily possible since the competitor papers might not report such data and would require re-implementations. If such curves cannot be easily obtained, I do not expect the authors to re-implement other methods during rebuttal, but if possible it would be nice to include convergence curves (i.e. accuracy vs. training epochs) for the proposed method, such that future methods can compare against these results. (7 Supplementary code) The accompanying release of code in the style of an easily extendable and customizable framework is a big plus and could potentially be very beneficial to the community. For the final release of the code please consider adding a bit more documentation and some documented examples. I also highly appreciate that the authors tested their code on Windows and Linux.