Constrained Cross-Entropy Method for Safe Reinforcement Learning
We study a safe reinforcement learning problem in which the constraints are defined as the expected cost over finite-length trajectories. We propose a constrained cross-entropy-based method to solve this problem. The method explicitly tracks its performance with respect to constraint satisfaction and thus is well-suited for safety-critical applications. We show that the asymptotic behavior of the proposed algorithm can be almost-surely described by that of an ordinary differential equation. Then we give sufficient conditions on the properties of this differential equation for the convergence of the proposed algorithm. At last, we show with simulation experiments that the proposed algorithm can effectively learn feasible policies without assumptions on the feasibility of initial policies, even with non-Markovian objective functions and constraint functions.

This paper deals with a safe reinforcement learning problem, where constraints are defined as an expected cost over finite-length trajectories, the goal being to maximize an expected gain (also over finite-length trajectories, so more general than cumulative rewards), while the constraint is greater than a given lower bound. To do so, the authors extend the cross-entropy method applied to classic RL. If sampled policies are feasible (satisfy the constraints), they are sorted according to their gain (as in classic cross-entropy). If not, how much the constraints are violated is also taken into account (by selecting the closer to feasible ones). The proposed contribution is supported by a theoretical analysis and experimental results.  Overall, the paper is well written, deals with an important subject, and provides interesting theoretical and experimental results. However, it could be improved with some clarifications, regarding the approach, as well as theoretical and empirical results. Also, the scalability of the proposed approach could be discussed (a possible work about this is “Evolution strategies as a scalable alternative to reinforcement learning”).  Many (usually small) changes could help clarify the proposed approach, for example: * It would be better to say already in l.110 that J>0 (it is said after, but can be unnoticed, and Eq.1 makes no sense with negative J). * In Alg.1, what are N_l (n_l?), lambda_l? Also, in "ensure" part, there should be NEF and Pi * sec. 4.2 could me made clearer, notably by clarifying how one goes from distributions g to the etas, and especially how lines 11 and 12 in Alg.1 are derived (it is not that clear, even with the appendix). It could maybe also help to instantiate it (say in the appendix), for example for the Gaussian with diagonal matrix case considered practically. * l.230, "by definition of eta", this is really not clear. The definition is eta(t) = m(v(t)), but it is never clearly stated in the paper (can be infered from alg.1, or line 200)  * In the main text (eg l.115), a policy is said to be feasible if H_Z(pi)>d. In Fig. 1, feasible policies are below the dashed lines, that is H_Z(pi)
Reviewer 3
In this paper, the authors studied a safe RL problem in which the constraints are defined over the expected cost over finite length trjectories. Based on the cross-entropy method, the method tracks the reward performance and constratint cost satisfaction and update accordingly the objective function. They further show that the asymptotic behavior of the algorithm can be described by an ordinary differential equation, and give sufficient conditions on the properties of this differential equation to guarantee the convergence of the proposed algorithm.  Simulation shows that this method produces compelling results when compared with CPO, a constrained version of the TRPO algorithm. Furthermore, this algorithm can also effectively solves problem without assumptions on the feasibility of initial policies, even with non-Markovian objective functions and constraint functions.  In general, I think this paper studies a very interesting and practical problem of constrained RL, which also has applications to multi-objective RL. Rather than using gradient-based policy optimization approaches, the authors turn to the evolutionary CE method, where a bunch of policy parameters are sampled from some prior distribution at each iteration, and the best portions of the samples are used to update the distribution of the policy parameter. Specifically, the policy parameter is assumed to have a NEF distribution. Since the RL problem is constrained, the loss function is of a hybrid form.    If the (1−\rho)-quantile of constraint for the current policy distribution is less than the constraint threshold, we select policies by their constraint values in order to increase the probability of drawing feasible policies. On the other hand, if the proportion of feasible policies is higher than \rho, we only select feasible policies with large objective function values. In general, I found the CE procedure intuitive, and the description clearly written.   Furthermore, the authors analyzed the CE algorithm using standard asymptotic analysis of stochastic approximation and ODE, and show that the constrained CE algorithm converges to a local maximum (and feasible) solution almost surely. Although I haven't had a chance to go through the proof details in the appendix, the analysis looks rigorous to me. Then they show that the constrained CE algorithm out-performs CPO in a simple robot navigation problem especially when the cost and constraint functions are non-Markovian, which is very interesting.   However, I have the following technical questions/concerns: 1) Unlike Lagrangian approaches, CPO is a policy gradient method that is derived to guarantee feasibility at every update iteration. Although it's true that in function approximations, without engineering tricks such a guarantee in general might fail to hold, this algorithm is still developed based on principled arguments on constraint satisfaction versus policy improvement. But to the best of my knowledge, for evolutionary methods such as CE, while it can work surprisingly well on some examples, they have the disadvantage of updating parameter values stochastically, without following any explicit gradient that carries information about what changes to parameter values are likely to improve performance. In this case, I suspect the constrained CE can guarantee constraint satisfaction at every update.  2) Since convergence of the proposed algorithm is in the asymptotic sense, why don't we use a Lagrangian approach to solve the constraint problem (i.e., for fixed \lambda, we solve for the unconstrained problem using CE, and update \lambda in a slower time-scale with rewards/constraint cost sampled from the black box)? In this case, we can directly re-use existing algorithms and analysis of unconstrained CE. How does this approach numerically compare with the proposed constrained algorithm?   3) While I appreciate the positive results reported in this paper, (especially the non-Markovian ones) since CE is an evolutionary algorithm, testing only on one domain might not be convincing enough to conclude that constrained CE will always out-perform CPO. Have you compared the proposed method with CPO on the Mujoco benchmarks reported in the CPO paper?   4) Can we extend the constrained CE method to handle infinite horizon problems?


Distributed Learning without Distress: Privacy-Preserving Empirical Risk Minimization
Distributed learning allows a group of independent data owners to collaboratively learn a model over their data sets without exposing their private data. We present a distributed learning approach that combines differential privacy with secure multiparty computation. We explore two popular methods of differential privacy, output perturbation and gradient perturbation, and advance the state-of-the-art for both methods in the distributed learning setting. In our output perturbation method, the parties combine local models within a secure computation and then add the required differential privacy noise before revealing the model. In our gradient perturbation method, the data owners collaboratively train a global model via an iterative learning algorithm. At each iteration, the parties aggregate their local gradients within a secure computation, adding sufficient noise to ensure privacy before the gradient updates are revealed. For both methods, we show that the noise can be reduced in the multi-party setting by adding the noise inside the secure computation after aggregation, asymptotically improving upon the best previous results. Experiments on real world data sets demonstrate that our methods provide substantial utility gains for typical privacy requirements.

The paper is about learning in a decentralized setting where parties have their own dataset and learn privately a global model. Privacy is usually guaranteed either by cryptographic procedures (secure multi party computation, MPC) or by introducing noise and theoretically analyzed in the framework of differential privacy (DP). Following Pathak et al [39], the paper combines MPC and DP. But the main contribution is to reduce the noise by a factor sqrt(m) using two different techniques: one from Chauduri et al [10] and one from Bun and Steike [6]. The paper considers two settings, output perturbation and gradient perturbation.   In the output perturbation scheme, the parties securely send their model learned locally to a central server, the server aggregates (averages) the models to build a joined model and then adds noise before releasing it. The average is privately computed in the MPC protocol. The approach is therefore very similar to the one of Pathak. The contribution in this part of the paper is to use a bound proved in Chauduri et al [10] in the proof that evaluates the needed noise to be \epsilon-DP.  In the gradient perturbation scheme, noise is introduced at every gradient step, after averaging the gradients, and before sending a new model back to the parties. Therefore, a general result is obtained using composition results in DP. The paper relies on zero concentrated DP for a more efficient combination step (Bun and Steike [6]).  The new bounds improves also the impact of noise in the true and empirical risks (and therefore it is important for associated ERM problems). Most of the proofs are not original (adapted from Chauduri and Monteleoni [9]) but  easy to follow and clearly exposed.     The paper is clearly written. The authors have correctly explained the motivations and have honestly indicated at the core of their contribution, to whom this or that result was attributed. Detailed proofs are given in the appendix.   The authors claim in different points of the paper that the benefit is explained by the fact that they add noise inside the MPC. This is not clear for me because in each scheme, the noise is introduced after model aggregation (so it could be "after" rather than "inside").   In conclusion, the paper improves the known bounds and shows theoretically and experimentally the benefits of the proposed algorithms. The significance is moderated by the fact that the results are obtained from small adaptations of the known techniques using each time another work. But I think that the paper is clearly written and the results are interesting.    Technical:  - l133-134 is difficult to understand (almost false). I think that the authors want to  say that Z is concentrated around 0 and then it becomes unlikely to distinguish D from D'.  - I am not sure that the comparison with Pathak et al is fair enough because they use the l1 norm to bound the difference between two models learned from closeby by datasets     UPDATE:  "The key difference is whether the noise is added before or after the models are aggregated." Indeed, you're right and it is more clear to say after r before aggregation than "inside/within" (because the MPC is not really modified).


Model Agnostic Supervised Local Explanations
Model interpretability is an increasingly important component of practical machine learning. Some of the most common forms of interpretability systems are example-based, local, and global explanations. One of the main challenges in interpretability is designing explanation systems that can capture aspects of each of these explanation types, in order to develop a more thorough understanding of the model. We address this challenge in a novel model called MAPLE that uses local linear modeling techniques along with a dual interpretation of random forests (both as a supervised neighborhood approach and as a feature selection method). MAPLE has two fundamental advantages over existing interpretability systems. First, while it is effective as a black-box explanation system, MAPLE itself is a highly accurate predictive model that provides faithful self explanations, and thus sidesteps the typical accuracy-interpretability trade-off. Specifically, we demonstrate, on several UCI datasets, that MAPLE is at least as accurate as random forests and that it produces more faithful local explanations than LIME, a popular interpretability system. Second, MAPLE provides both example-based and local explanations and can detect global patterns, which allows it to diagnose limitations in its local explanations.

Paper summary  This paper focuses on interpretable machine learning (IML) and proposes an approach (denoted SLIM) for achieving interpretability through example-based and local explanations. The approach combines ideas from local linear models and random forests. It builds on existing work on feature selection and supervised neighborhood selection for local linear modeling. The paper uses recent work in IML to define interpretability and shows via experiments on eight UCI datasets how to maintain high accuracy (through the use of random forests and similar methods) while being able to provide local explanations.  Summary of strengths and weaknesses  - Strengths: the paper is well-written and well-organized. It clearly positions the main idea and proposed approach related to existing work and experimentally demonstrates the effectiveness of the proposed approach in comparison with the state-of-the-art.  - Weaknesses: the research method is not very clearly described in the paper or in the abstract. The paper lacks a clear assessment of the validity of the experimental approach, the analysis, and the conclusions.  Quality  - Your definition of interpretable (human simulatable) focuses on to what extent a human can perform and describe the model calculations. This definition does not take into account our ability to make inferences or predictions about something as an indicator of our understanding of or our ability to interpret that something. Yet, regarding your approach, you state that you are “not trying to find causal structure in the data, but in the model’s response” and that “we can freely manipulate the input and observe how the model response changes”. Is your chosen definition of interpretability too narrow for the proposed approach?  Clarity  - Overall, the writing is well-organized, clear, and concise. - The abstract does a good job explaining the proposed idea but lacks description of how the idea was evaluated and what was the outcome.  Minor language issues  p. 95: “from from” -> “from” p. 110: “to to” -> “how to” p. 126: “as way” -> “as a way” p. 182 “can sorted” -> “can be sorted” p. 197: “on directly on” -> “directly on” p. 222: “where want” -> “where we want” p. 245: “as accurate” -> “as accurate as” Tab. 1: “square” -> “squared error” p. 323: “this are features” -> “this is features”  Originality  - the paper builds on recent work in IML and combines two separate lines of existing work; the work by Bloniarz et al. (2016) on supervised neighborhood selection for local linear modeling (denoted SILO) and the work by Kazemitabar et al. (2017) on feature selection (denoted DStump). The framing of the problem, combination of existing work, and empirical evaluation and analysis appear to be original contributions.  Significance  - the proposed method is compared to a suitable state-of-the-art IML approach (LIME) and outperforms it on seven out of eight data sets. - some concrete illustrations on how the proposed method makes explanations, from a user perspective, would likely make the paper more accessible for researchers and practitioners at the intersection between human-computer interaction and IML. You propose a “causal metric” and use it to demonstrate that your approach achieves “good local explanations” but from a user or human perspective it might be difficult to get convinced about the interpretability in this way only. - the experiments conducted demonstrate that the proposed method is indeed effective with respect to both accuracy and interpretability, at least for a significant majority of the studied datasets. - the paper points out two interesting directions for future work, which are likely to seed future research.


GumBolt: Extending Gumbel trick to Boltzmann priors
Boltzmann machines (BMs) are appealing candidates for powerful priors in variational autoencoders (VAEs), as they are capable of capturing nontrivial and multimodal distributions over discrete variables. However, non-differentiability of the discrete units prohibits using the reparameterization trick, essential for low-noise back propagation. The Gumbel trick resolves this problem in a consistent way by relaxing the variables and distributions, but it is incompatible with BM priors. Here, we propose the GumBolt, a model that extends the Gumbel trick to BM priors in VAEs. GumBolt is significantly simpler than the recently proposed methods with BM prior and outperforms them by a considerable margin. It achieves state-of-theart performance on permutation invariant MNIST and OMNIGLOT datasets in the scope of models with only discrete latent variables. Moreover, the performance can be further improved by allowing multi-sampled (importance-weighted) estimation of log-likelihood in training, which was not possible with previous models.

# Response to author feedback  My thanks to the authors for their detailed feedback to the comments in my review. The clarifications about the GumBolt-nW experiment and further baseline results provided comprehensively addressed my queries about this issue and instil confidence in the authors' conclusion that there is a significant gain to using a RBM prior over simpler factorial choices. After reading the authors responses to my and the other reviews I continue to feel this would be a worthy inclusion in the conference and vote for acceptance.  ---  # Summary and relation to previous work  This submission builds upon a series of previous works proposing methods for including discrete latent variables in variational autoencoder models while retaining the ability to train the models with low-variance reparameterisation trick based gradient estimates of the variational objective by relaxing the discrete latent variables with associated continuous valued variables. In particular it extends the dVAE (Rolfe, 2016) and dVAE++ (Vahdat et al., 2018) models which use a Boltzmann machine (BM) prior on the discrete latent variables by using an analogue of the 'Gumbel trick' relaxation (Maddison et al., 2016; Jang et al., 2016) applied to the BM prior. The resulting model and training approach is argued to be implementationally simpler than the dVAE and dVAE++ approaches while also allowing the use of a tighter importance-weighted variational bound (Burda et al., 2015) which has been found to often improve training performance. The authors empirically demonstrate the efficacy of their proposed 'GumBolt' approach compared to dVAE and dVAE++ in terms of significant improvements in test set log likelihoods on two benchmark binarized image generative model datasets (MNIST and OMNIGLOT) across a range of different architectures.  # Evaluation  Although the proposed approach is a largely incremental improvement of existing work, I still believe it would be a worthwhile contribution to the conference with the particular synthesis of ideas from various previous pieces of works being used to produce an approach for training VAE models with discrete latent variables which leads to significant gains in empirical performance. The approach is clearly described, with the relevant background on the various ideas being combine (VAEs, Gumbel trick, BMs) succinctly explained and the relationship of the proposed approach to the most closely related previous work (dVAE and dVAE++) well summarised in Figure 1 and the accompanying text.  The issue that even with a bipartite connectivity, sampling from BM distributions is challenging is largely glossed over, with block Gibbs often still mixing quite slowly and therefore requiring many iterations to get a close to independent sample - the original dVAE paper illustrated improved test-set log-likelihoods as the number of Gibbs iterations used during training for sampling from the (R)BM prior was increased and it would have been helpful to have at least a short discussion of this issue here and justification for the choice 200 Gibbs updates during training in the experiments.  The illustration of the relationship between the various objectives discussed in Figure 2a is helpful, and the inclusion of a plot of the sensible sanity check of whether the proposed relaxed proxy training objective is tracked by the discrete objective is reassuring.  In the experiments with results summarised in Table 2 it is unclear if the 'GumBolt-nW' entries correspond to a GumBolt model trained with non-zero weights and then W set to zero *after* training, or a GumBolt model with W set to zero throughout training - could the authors clarify which of these is the case? If the latter then the results do indeed seem to support that the use of a BM prior gives a significant improvement in performance over a simpler factorial prior, though if the removal of the coupling matrix W parameters in the GumBolt-nW case significantly decreases the overall number of parameters in the model it would be interesting to also include an additional baseline in which e.g. a deeper and/or wider networks are used in the decoder model to see if this can counteract the loss of flexibility in the prior.  # Minor comments / typos / suggestions  L20: 'VAEs tend to ignore the latent code in the presence of powerful decoders' - its not clear exactly what is meant here by 'latent code' and by 'ignore'. It seems implausible for VAE to 'ignore' the latent code in the sense of the latent variables not having a strong influence on the generated output when mapping through the decoder, for example in the case of a Gaussian decoder this would surely correspond to the mean and variance networks outputting fixed values independent of z and so the learned marginal distribution on x being Gaussian. L28: Not clear why that Boltzmann machine represent intractable distributions make them 'ideal for priors in VAEs' L38/39: 'auto-encoding term' is not explicitly defined making it a bit difficult to parse this L51: Citation on this line should use \citet not \citep as directly reference in text L96: 'to facilitates sampling' -> 'to facilitate sampling' - would also be more accurate to make explicit this is MCMC sampling L97: Would be useful to here to state bipartitie connectivity assumed in Boltzmann machine prior often termed restricted Boltzmann machine (as RBM which is used later is never explicitly defined) Eq 5: nitpick but it would more correct for the subscripted distribution density the expectation is indicated to be with respect to to be a product of densities over the sample index $i$ (i.e. joint over independent samples) Eq 6: Looks like there are negatives missing in the expression after the second and third equalities L116: 'probability distribution' -> 'probability density function' would be better here L123: 'theorom' -> 'theorem' Table 1: The wider gap between $k=1$ and $k=5$ GumBolt columns is a bit confusing as makes it appear as if the $k=1$ column is grouped with those to left rather than right. Also need to make cleared in caption and/or main text that these are (presumably) estimated *test-set* log-likelihoods. L199: 'reduced by 0.3' is ambiguous - would be better to be more explicit e.g. is the learning rate multiplied by 0.3 or 1 - 0.3 = 0.7?


LinkNet: Relational Embedding for Scene Graph
Objects and their relationships are critical contents for image understanding. A scene graph provides a structured description that captures these properties of an image. However, reasoning about the relationships between objects is very challenging and only a few recent works have attempted to solve the problem of generating a scene graph from an image. In this paper, we present a method that improves scene graph generation by explicitly modeling inter-dependency among the entire object instances. We design a simple and effective relational embedding module that enables our model to jointly represent connections among all related objects, rather than focus on an object in isolation. Our method significantly benefits main part of the scene graph generation task: relationship classification. Using it on top of a basic Faster R-CNN, our model achieves state-of-the-art results on the Visual Genome benchmark. We further push the performance by introducing global context encoding module and geometrical layout encoding module. We validate our final model, LinkNet, through extensive ablation studies, demonstrating its efficacy in scene graph generation.

Summary -------------  This paper proposes an attentional graph neural network approach for scene graph classification (classifying the objects in a scene, as well as the relationships between those objects) and applies it to the Visual Genome dataset, achieving state-of-the art performance with significant improvements particularly on the scene graph classification tasks.  Overall, I think this approach seems very promising and the results are quite impressive (though I should say I am less familiar with the literature on scene graph classification specifically). The architecture used in the paper is not particularly novel as graph neural networks have (1) previously been tried and (2) the paper mostly just borrows its architecture from the Transformer architecture (see below), but I think the impressive results can make up for a lack of novelty. However, I did find the paper somewhat difficult to read and it took me several passes through Section 3 to understand exactly what was going on. Additionally, I found the experiments to be a bit weak and some of the architectural choices to be somewhat arbitrary and not justified.  Quality ---------  Some of the choices of the architecture seem somewhat arbitrary and it's not clear to me their significance. For example, why is it necessary to turn O_4 into one-hot vectors via an argmax? Or, why is O_3 used in constructing E_0? Does these choices really change performance?  I appreciate the ablation experiments which show that the Global Context Embedding, Geometric Layout Encoding, and Relational Embedding are all important aspects, but I feel these experiments could have gone further. For example, the relational embedding is used four times: twice for node classification and twice for edge classification. Is it needed for both node and edge classification, or is it sufficient to just use it e.g. for node classification? And how much of an impact is there in using two stacked relational embeddings rather than one (or three)? Moreover, there are no ablation experiments which completely remove the relational embedding (but e.g. leave in the global context or the geometric layout encoding), making it difficult to judge how much that is improving things over a baseline.  Clarity --------  I found Section 3 hard to follow, with inconsistent and sometimes confusing notation. The following points were the hardest for me to follow:  - Sometimes the text specifies a variable name for a parameter matrix (e.g. W_1, U_1), and sometimes it uses an fc layer (e.g. fc_0, fc_1). I take it the difference is that the parameter matrices don't include bias terms, but I think it would be far easier to understand if the paper just used fc functions everywhere and specified somewhere else (e.g. in a footnote, or in the appendix) which fc terms used a bias and which did not. - I would strongly recommend a revision include (for example) an algorithm box which makes it clearer exactly how the computation is structured (this could even be in the appendix). - I found Figure 1 to be confusing, especially since there are terms used in the figure which aren't defined anywhere but the text (ideally, the figure should be interpretable in isolation). For example, O_3 and O_4 are not defined in the figure. I think the figure could be redone to be more explicit about where the different terms come from (I think there is a disproportionate amount of whitespace in the relational embedding and FC boxes, so these could be shrunk to make room for more detail). - Another detail which wasn't clear to me is where the features f_{i,j} (which are used in Eq. 12) come from. Are these the same ROI features introduced in 3.3.1? If so, how are they indexed by j? I thought f_{i} was of dimensionality 4096.  Originality -------------  There are two forms of novelty in this paper: first, applying the idea of attentional graph neural networks to scene graph classification, and second, including auxiliary losses and computations in the global context embedding and the geometric layout encoding. However, the architecture of the relational embedding that is used (Eqs. 1-3 and 4-5) is almost exactly that from the Transformer architecture (Vaswani et al., 2017). Moreover, the idea of using (non-attentional) graph neural networks in the context of scene graph classification was previously tried by Xu et al. (2017). Thus, while the results are impressive, there does not seem to be much innovation in the architecture.  Some other recent work has also used graph neural networks for just region classification and may be worth discussing and comparing to, see: Chen, X., Li, L., Fei-Fei, L., and Gupta, A. (2018). Iterative visual reasoning beyond convolutions. CVPR 2018.  Significance ----------------  As far as graph neural networks go, it is very interesting to see them applied more widely to more naturalistic data like raw images and I think this paper is significant in that regard (even if it is not the absolute first to do so). More significant are the results that the method achieves on the Visual Genome dataset, which seem to significantly push the state of the art.  Edit after author response: I have read the other reviews and the author response and am quite satisfied with the addition of several new ablation experiments. I do wish the authors had attempted to explain why they made the choices that they did about E_0 and O_4 in the first place (and hope that they will in the final revision), but it is good to see the empirical experiments showing those choices do matter. I also wish that the authors had explained a bit further how they see their work as different from previous approaches in the rebuttal, as suggested by R3, and hope that they will do so in the final version. All that said, I think the results are strong enough and the experiments now thorough enough that I have increased my score to a 7.


Near Optimal Exploration-Exploitation in Non-Communicating Markov Decision Processes
While designing the state space of an MDP, it is common to include states that are transient or not reachable by any policy (e.g., in mountain car, the product space of speed and position contains configurations that are not physically reachable). This results in weakly-communicating or multi-chain MDPs. In this paper, we introduce TUCRL, the first algorithm able to perform efficient exploration-exploitation in any finite Markov Decision Process (MDP) without requiring any form of prior knowledge. In particular, for any MDP with S C communicating states, A actions and
C is the diameter (i.e., the length of the longest shortest path between any two states) of the communicating part of the MDP. This is in contrast with existing optimistic algorithms (e.g., UCRL, Optimistic PSRL) that suffer linear regret in weakly-communicating MDPs, as well as posterior sampling or regularised algorithms (e.g., REGAL), which require prior knowledge on the bias span of the optimal policy to achieve sub-linear regret. We also prove that in weaklycommunicating MDPs, no algorithm can ever achieve a logarithmic growth of the regret without first suffering a linear regret for a number of steps that is exponential in the parameters of the MDP. Finally, we report numerical simulations supporting our theoretical findings and showing how TUCRL overcomes the limitations of the state-of-the-art.

UPDATE: I have read the authors' response and it was very helpful and addressed my concerns well and I have adjusted my score.  I now see the flaw in the regal reasoning and encourage the authors to add the explicit callout of the backwards inequality in the supplementary material (unless I missed it).  I also agree with the reasons for not comparing to regal empirically.  The point about algorithm mechanics is well thought out, though I still see some benefit to calling it out.  The wording correction you mentioned for line 159 makes sense as well.   Summary:  The paper considers the problem of RL exploration/exploitation in weakly communicating MDPs where many states (S_T) may not actually be reachable.  Such states cause an overestimate of values in algorithms like UCRL, which consider such states in their contribution to the exploration bonus.  The authors propose a new algorithm, TUCRL, that performs an online estimate of the reachability of states, and discards states that are below a threshold based on visits and the number of steps so far.  The algorithm is shown to have improved regret in such MDPs.  Furthermore, a new lower bound is proven showing that no algorithm  in this setting can achieve logarithmic regret without first incurring  exponential regret.  Review:  Overall, the paper makes a strong case for TUCRL.  The proofs are thorough and convincing.  I appreciate having the sketches actually in the paper in addition to the complete results in the supplementary material.  The idea appears to be sound, since UCRL was not making use of the tighter bounds that can be derived on state reachability, and the empirical results verify that the real-world advantages of the improved bound are significant.  The lower bound is also an important result outside of the TUCRL analysis.  My only concerns are in the comparison to Regal, some of the terminology in Section 2, and the description of the algorithm mechanics but I think these can be easily addressed.  The paper’s claim of a flaw in the proof of Regal’s regret was surprising and the logic laid out in the supplementary material appears correct, but the explanation is worded in a confusing way that makes it unclear what exactly the flaw is.  On line 417 of the supplemental material, we see the flawed inequality -C \Sum{v} <= -C 2^j.  The authors then state: “[that inequality] is not true since -C := -c / sqrt(2^j).  However, that definition of C does not actually invalidate the inequality since C appears on both sides.  The authors later point out that what is needed is a lower bound on v, but they are unable to find one so they declare the proof incorrect.  All the evidence here is pointing to there being a flaw and the authors being right, but we’re missing an actual example where the inequality fails to hold.  The absence of a proof does not mean the statement is necessarily wrong (even though I believe it is).  Can the authors give us an actual MDP or even just plausible numerical instantiations of this inequality that make it untrue?  If so, that would seem to be a much cleaner refute of Regal.  In addition to the unclear terminology above, I was surprised that Regal did not appear in the experiments since it seems to be built for these kinds of environments, even if the theory is flawed.  Would an empirical result actually show TUCRL outperforming Regal?  If all we have is questions about its proof then an empirical result seems needed.  Most of the paper’s terminology is rigorously defined, but I found the definition of S^C and S^T in the beginning of Section 2 surprisingly informal.  The initial definitions make it unclear what cases are allowed under these conditions – are they hard partitions where only one is visited – can there be many “island” MDPs that don’t communicate but an episode can start in any one of them?  In that case, what counts as S^C?  Many of these questions are cleaned up laterin the text, particularly in the multi-chain MDP subsection later in section 2, but they should really be addressed with a formal and rigorous definition of S^C and S^T right at the start.  One of the most interesting parts of the proof was the bounding of the number of times a poorly visited (s,a) is visited.  This is reminiscent of escape probabilities in the PAC literature but in TUCRL, we see a UCB/UCRL like behavior where “jumps” in the exploration occur periodically (though with longer and longer periods).  This is also borne out in the empirical results.  I would have liked to have seen this intuition presented when the algorithm was introduced, maybe as a set of “key properties” of the algorithm so that users will better understand the behavioral characteristics of the algorithm.  Right now those details are scattered in various sections.  I realize space is tight, but you could add a few lines to the supplemental material covering this.  The main result is a good mathematical derivation but the significance of the empirical results put the theory in a better light.  The lower bound is also a strong contribution and an important component for a theory paper.   Minor notes:   Line 159: poorly visited so far are enabled -> This is unclear.  What does enabled mean?  Shouldn’t poorly visited states be discarded?


Optimistic Optimization of a Brownian
We address the problem of optimizing a Brownian motion. We consider a (random) realization W of a Brownian motion with input space in [0, 1]. Given W , our goal is to return an ε-approximation of its maximum using the smallest possible number of function evaluations, the sample complexity of the algorithm. We provide an algorithm with sample complexity of order log 2 (1/ε). This improves over previous results of Al-Mharmah and Calvin (1996) and Calvin et al. (2017) which provided only polynomial rates. Our algorithm is adaptive-each query depends on previous values-and is an instance of the optimism-in-the-face-of-uncertainty principle.

This paper considers a classic problem in Bayesian optimization, where we seek to maximize a one-dimensional function evaluated without noise with a Brownian motion prior.  Using a novel algorithm and analysis that builds on Munos 2011 but goes substantially beyond it, it shows that an epsilon-optimal point can be found using log^2(1/epsilon) samples.  This provides over the previous best-known rate for this problem.  The results are quite nice from the point of view of the theoretical foundations of Bayesian optimization.  Indeed, optimizing with a Brownian motion prior has been studied since the 1970s by Calvin and Zilinskas especially, and this result improves on all of those results.  Thus, I feel the work is of high quality.  The work is also original --- although the paper builds on work by Munos and Calvin, there is a lot that is new.  The paper is also quite clear.  The only problem is with the paper's significance for the NIPS community, which often wants to see meaningful empirical results and practical applicability.  Unfortunately one-dimensional problems are much less common in practice than multi-dimensional ones, and Wiener process priors often don't fit the functions we optimize very well.  I feel that this work *is* significant from a theory perspective, given the dearth of results on convergence rates for Bayesian optimization, and is a stepping stone to understanding rates for more practical settings.  Beyond this, I only have some minor advice on clarity:  Lipschitz discussion: When I read the initial parts of the paper, I found the discussion of Lipschitz continuity confusing, since the path of a Brownian motion is not differentiable, and I believe is not Lipschitz continuous in the classical sense.  Here we are looking at finite meshes of decreasing size, and so after reading into the body of the paper I understood what was meant, but perhaps there is a way to write the initial discussion in a way that does not confuse readers familiar with the infinite roughness of Brownian motion's paths.   Throughout, I find the word "Brownian" without "motion" or "bridge" strange.  I recommend talking about a "Brownian motion" rather than a "Brownian".  If you like, you could abbreviate this at BM.  Typos: line 117 typo: "The full proofs the auxiliary lemmas" line 139 typo: "achieving such PAC guarantee is low." line 150 and 151: it would be more clear to write the necessary condition for splitting as max(W(a),W(b)) >= M - eta*(b-a).


Chaining Mutual Information and Tightening Generalization Bounds
Bounding the generalization error of learning algorithms has a long history, which yet falls short in explaining various generalization successes including those of deep learning. Two important difficulties are (i) exploiting the dependencies between the hypotheses, (ii) exploiting the dependence between the algorithm's input and output. Progress on the first point was made with the chaining method, originating from the work of Kolmogorov, and used in the VC-dimension bound. More recently, progress on the second point was made with the mutual information method by Russo and Zou '15. Yet, these two methods are currently disjoint. In this paper, we introduce a technique to combine the chaining and mutual information methods, to obtain a generalization bound that is both algorithm-dependent and that exploits the dependencies between the hypotheses. We provide an example in which our bound significantly outperforms both the chaining and the mutual information bounds. As a corollary, we tighten Dudley's inequality when the learning algorithm chooses its output from a small subset of hypotheses with high probability.

The paper begins with musings about deep learning and SGD. It then makes this claim    "In particular, classical generalization bounds do not allow to explain this phenomenon."  which requires citation as it is not established by this paper, and otherwise it's hearsay. In fact, it's arguably false. On the surface, Bartlett's 1998 bounds say that controlling the norm of the weights suffices and it could have been the case that overparametrized networks allowed for small norm solutions... enough to establish generalization.  But, in fact, these bounds imply only that the population classification error is .... less than one. That is, vacuous. Dziugaite and Roy demonstrate this by studying margin bounds based on the path-norm. (Zhang et al, in fact, do not demonstrate that Rademacher bounds cannot explain generalization... no one ever seriously proposed to explain overparametrized neural network generalization by UNIFORM convergence alone!!! Ridiculous.)  This papers contributions are more fundamental than recent hype, and so I would suggest cutting the hearsay and sticking to fundamentals as well. I find the conclusion odd as well.  I have two concerns:  1. The authors do not seem to be aware that PAC-Bayes bounds relate to mutual information by taking P = E[Q(S)] for S ~ i.i.d. and Q : Z^m \to M(H) the randomized learning algorithm. Then the KL(Q||P) part of the PAC-Bayes bound is  equal to the mutual information. While PAC-Bayes bounds control the risk of Gibbs classifiers, taking expectations leads one to bounds on the expected generalization error of a classifier drawn at random, which overlaps with the setting here!  This connection with mutual information can be found in Catoni (2007), I believe. It's certainly mentioned by McAllester in one of his PAC-Bayes tutorials too.   2. No citation is made to Audibert and Olivier (2004) "PAC-Bayesian Generic Chaining", which, combined with 1, would seem to have already delivered on the promised combination here, albeit for randomized classifiers. Raginsky et al seem not to be aware of these connections either.  I need to hear a concrete plan for how the paper will be modified to address this prior work, which seems to overlap significantly.  Other comments:  I really liked the example. Extremely clear. Regarding Table 1:  the CMI is impressive compared to the chaining and MI bounds on their own, but the bounds seem to be off by 20 or so. What explains that gap? Could a better sequence of partitions have fixed it? Generic chaining? A 20x gap will completely blow any chance of teasing apart the generalization performance of state of the art versus JUNK algorithms.   I also like the proof sketch, which is also very very clear.  Some issues:  179 The order of operations here is ambiguous. I am parsing this as           (argmax_phi X_phi) \oplus Z (mod 2pi)      which seems to be the only sensible parsing, but it's annoying I had to      spend 20 seconds on this, and manage this uncertainty going forward.  204 strictly speaking, it doesn't fail. it's just vacuous. a failure sounds like the bound is violated, which it isn't.  205 these side facts about LIL and random tournaments are irrelevant.   minor typographical issues:  122 hyphens should be an "endash" in Azuma--Hoeffding and other pairings of names.  130 typo: increasing sequence   136 text is bouncing back and forth between "the metric space" and "a (bounded) metric space". Are we now assuming (T,d) is bounded? Or only for this paragraph. If you stick with "the", consider writing "Assume now that (T,d) is a bounded space."  136 unnecessary to recall this soon after making these definitions.  204 the union bound


Thermostat-assisted continuously-tempered Hamiltonian Monte Carlo for Bayesian learning
We propose a new sampling method, the thermostat-assisted continuously-tempered Hamiltonian Monte Carlo, for Bayesian learning on large datasets and multimodal distributions. It simulates the Nosé-Hoover dynamics of a continuously-tempered Hamiltonian system built on the distribution of interest. A significant advantage of this method is that it is not only able to efficiently draw representative i.i.d. samples when the distribution contains multiple isolated modes, but capable of adaptively neutralising the noise arising from mini-batches and maintaining accurate sampling. While the properties of this method have been studied using synthetic distributions, experiments on three real datasets also demonstrated the gain of performance over several strong baselines with various types of neural networks plunged in.

This paper presents a sampling method that combines Hamiltonian Monte Carlo (HMC), mini-batches, tempering, and thermostats, to more efficiently explore multimodal target distributions. It is demonstrated on a number of substantial neural network problems using real data sets.  This is an interesting method, and the empirical results are quite substantial. Figure 2 does a nice job of demonstrating how the omission of any of the ingredients (e.g. the tempering, or the thermostat) is detrimental to the overall result, which is a nice illustration of how the combination works together well. This is followed by some substantial image classification examples.  I have one major, and one minor issue with the work, which perhaps the authors can address in their feedback:  1. It is not clear to me that the work contributes any new ideas beyond those already introduced in the following paper (cited as [5]):  Ding et. al. (2014). Bayesian Sampling Using Stochastic Gradient Thermostats.  Unless I am missing something, the above work introduces a sampler that combines the same components as listed above, and it, too, offers substantial empirical results. The authors might like to clarify what they see as the contribution of their work.  2. Given that these are Bayesian methods, I would have liked to have seen comparison of the posterior distributions or empirical convergence metrics, and not merely predictive performance, as given in Table 1. Presumably there is an "honest" predictive performance that is obtained by the exact posterior distribution for the model and data under study, and it could be obtained (hypothetically, at least) with any of the sampling methods (not the optimizations methods obviously) with a sufficient number of iterations. If a method has not converged sufficiently close to the posterior distribution, the predictive performance may be higher or lower than the honest value. It would not necessarily be lower. So these results are not really definitive as to which method is working best at obtaining the posterior distribution.  To show that, it would be useful to see some conventional convergence diagnostics such as autocorrelation plots, effective sample size, etc. Or, if the argument is that the Bayesian approach offers a regularization against overfitting, rather than us caring about having a good estimate of the posterior distribution, some uncertainty range could be reported in Table 1 (posterior variance of accuracy, or 95% credibility over posterior samples, etc), to calibrate the significance of the increase in predictive performance for the method that is presented. (I presume the predictive performance reported is the mean accuracy over the posterior samples, given the comment at the bottom of page 7, "For the sampler baselines, the final accuracy is calculated from the Monte Carlo integration on all sampled models...")  AFTER AUTHOR FEEDBACK  I thank the authors for their detailed response. The authors have addressed my concerns regarding novelty. Having had a better look at Ding et al. now I am satisfied that the addition of continuous tempering is indeed novel, and have increased my score accordingly. The authors have suggested that they will look at some additional metrics, which I would also very much like to see.


Answerer in Questioner's Mind: Information Theoretic Approach to Goal-Oriented Visual Dialog
Goal-oriented dialog has been given attention due to its numerous applications in artificial intelligence. Goal-oriented dialogue tasks occur when a questioner asks an action-oriented question and an answerer responds with the intent of letting the questioner know a correct action to take. To ask the adequate question, deep learning and reinforcement learning have been recently applied. However, these approaches struggle to find a competent recurrent neural questioner, owing to the complexity of learning a series of sentences. Motivated by theory of mind, we propose "Answerer in Questioner's Mind" (AQM), a novel information theoretic algorithm for goal-oriented dialog. With AQM, a questioner asks and infers based on an approximated probabilistic model of the answerer. The questioner figures out the answerer's intention via selecting a plausible question by explicitly calculating the information gain of the candidate intentions and possible answers to each question. We test our framework on two goal-oriented visual dialog tasks: "MNIST Counting Dialog" and "GuessWhat?!". In our experiments, AQM outperforms comparative algorithms by a large margin.

Summary: The current work proposes a new framework, Answerer in Questioner’s mind (AQM), for goal-driven visual dialog. Here, questioner has a model of the answer, which it uses to pick a question from a pool of candidate questions, based on information theoretical heuristics. Experiments are shown on (a) MNIST Counting (toy dataset), and (b) GuessWhat?!, showing improvements over competing baselines.  Strengths: (S1) The approach is well motivated by the obverter technique, based on the theory of mind. The paper does a good job of putting forth these arguments.  Weaknesses: (W1) My primary concern is regarding the limited natural language complexity of the Answerer the Questioner is trying to model. From the datasets used and examples shown, Answerer either utters single word answers (classification in case of MNIST Counting) or binary answers (GuessWhat?!). In my opinion, the proposed approach benefits the most when there is richness in language used by Answerer, to which Questioner adopts itself via this internal modeling. (C1) explores another dimension to this.  (W2) The manuscript has a lot of writing errors/typos (few listed at the end). Even though it does not hinder the understanding, it does break the flow of the reader. I urge the authors to carefully proofread their manuscript.  Comment: (C1) Modeling Answerer seems more impactful if Answerer has quirks. For instance, in any application, different human users have different preferences, personas, language styles, etc. Such experiments are missing, which would potentially make this paper stronger. As an example, a color blind Answerer (in case of MNIST counting) should drive the Questioner away from asking color based questions, and so on.  (C2) From my understanding, the model seems to benefit from both YOLO object candidates and the selection of question using proposed approach. On the other side, competing methods seem to be doing this visual inference themselves. Perhaps, include a baseline that also conditions on these proposals.  (C3) Less of a comment and more of a curious question. L141-155 mentions that this approach needs training for a given Answerer. Any thoughts on online training or a low-shot adaptation for a novel answerer?  (C4) L117: Functionality of Q-Sampler has not been introduced so far.  (C5) L122, Eq2: Typo in the equation. Are the bounds \Pi_{j=1}^{t} for the product?  (C6) L169-170: How is 85% -> 46.6%? These two sentences are unclear.  (C7) L212: What does consecutive questions mean here? Aren’t these just pool of questions that make up Q-Sampler, not in any particular sequence?  (C8) L191: Is the class label from YOLO also used?  Typos: L14: Swap closing mark and full stop L44: split -> splits L67: Not sure what the sentence is trying to convey, perhaps a typo somewhere? L102: Typo in the sentence. L115: substituted to -> substituted with / by L115: the equation -> an equation L120: to -> as L137: selects -> select L235: to -> with L235: task training -> task of training


Learning to Play With Intrinsically-Motivated, Self-Aware Agents
Infants are experts at playing, with an amazing ability to generate novel structured behaviors in unstructured environments that lack clear extrinsic reward signals. We seek to mathematically formalize these abilities using a neural network that implements curiosity-driven intrinsic motivation. Using a simple but ecologically naturalistic simulated environment in which an agent can move and interact with objects it sees, we propose a "world-model" network that learns to predict the dynamic consequences of the agent's actions. Simultaneously, we train a separate explicit "self-model" that allows the agent to track the error map of its worldmodel. It then uses the self-model to adversarially challenge the developing world-model. We demonstrate that this policy causes the agent to explore novel and informative interactions with its environment, leading to the generation of a spectrum of complex behaviors, including ego-motion prediction, object attention, and object gathering. Moreover, the world-model that the agent learns supports improved performance on object dynamics prediction, detection, localization and recognition tasks. Taken together, our results are initial steps toward creating flexible autonomous agents that self-supervise in realistic physical environments.

The work presents a new approach on exploration yielding towards self-supervising agents. The proposed idea relies on learning two models, a world-model and a self-model. The former predicts the world dynamics while the latter predicts errors of the world-model and is used to select actions that challenge the current world-model. This enables the agent to develop a good world model.   The overall approach is interesting but as far as I understood, it delivers in the end another approach for exploration.   The applied metrics are not further specified than in the paragrapg starting in line 219, so the actual evaluation is unclear to me, e.g. I do not understand how exactly the appearence of interesting behaviors is quantified.  It is not clear to me how exactly the different phases are determined i.e. the onsets of 'Emergence of object attention', 'Object interaction learning' etc. It would also have been nice to see a video / simulation of the behavior over the whole period (or at least plots from important phases, similar to Figure 5).  Due to the two points mentioned above (and brevity in explanation), the experiments and the respective evaluations are a bit squishy for me and i can not grasp what is actually achieved. Especially in the part of task transfer (starting in line 286) it is not clear to me what exactly was done. For example, I do not know what exactly is supposed to be transferred and also the whole setup did not appear to me to actively solve tasks but to rather explore and learn models. Thus, I also do not understand what is presented in Table 1. The accuracy is computed on what?  In Figure 4, (c) and (d) have only 200 steps or is the addition (in thousands) missing? It is also not clear on how many trials every model was evaluated and thus, for (a) and (b) I'm not sure whether I am just looking at a smoothed curve plotted over the original values? For (c) and (d), it looks more definite like an error area, but it is not clear to me whether I see the variance, twice the standard deviation or whatever of how many evaluations? The same holds for Figure 6.  A problem of the presented action space representation may be that it depends on the number of objects available. This may become quite hard for messy and cluttered environments. Also, the highest amount of objects evaluated was N = 2, so it would be interesting to see how well the approach scales for fuller rooms and how the behavior develops.   Additionally, for me, having almost the same amount of supplementary as submitted pages is a bit too much and unfair towards other authors who are limiting themselves. Thus, an alternative option could be to publish the work as a journal article.


Designing by Training: Acceleration Neural Network for Fast High-Dimensional Convolution
The high-dimensional convolution is widely used in various disciplines but has a serious performance problem due to its high computational complexity. Over the decades, people took a handmade approach to design fast algorithms for the Gaussian convolution. Recently, requirements for various non-Gaussian convolutions have emerged and are continuously getting higher. However, the handmade acceleration approach is no longer feasible for so many different convolutions since it is a time-consuming and painstaking job. Instead, we propose an Acceleration Network (AccNet) which turns the work of designing new fast algorithms to training the AccNet. This is done by: 1, interpreting splatting, blurring, slicing operations as convolutions; 2, turning these convolutions to gCP layers to build AccNet. After training, the activation function g together with AccNet weights automatically define the new splatting, blurring and slicing operations. Experiments demonstrate AccNet is able to design acceleration algorithms for a ton of convolutions including Gaussian/non-Gaussian convolutions and produce state-of-the-art results.

Post Authors' Response  Thanks a lot the authors for their response. I am happy with the responses they have provided to my initial concerns which have improved the manuscript. I would encourage authors to add an appendix should they believe they can convey a more complete message without having to wait for drafting another another longer manuscript.   ------------------------------------------------------------ Initial Review  The author/authors propose the first neural network producing fast high-dimensional convolution algorithms. They utilise the new Acceleration Network (AccNet)  to express the approximation function of splatting/blurring/slicing (SBS) and generalise SBS by changing the architecture of AccNet. Upon finishing the training process, the proposed fast convolution algorithm can be  derived from  the weights and activation functions of each layer. They conducted various experiments that prove the  effectiveness of the proposed algorithm.  The papers is very well written and of good quality. The main part of the paper, i.e. 3rd to 6th page, would benefit from making clearer the exact contribution of the authors. Reason is that it sometimes provides too much information about the fundamentals and the exact implementation and adjustments are likely to be missed. Section 3.2.4 that describes the proposed algorithm could serve for this purpose, i.e. what is being added or proposed to further enhance the SBS. Figure 3 shows the fast filtering approaches based on gCP and gHT decompositions. Is this based on section 3.2.3 that has been proposed by Hackbusch and Kuehn or includes other tensor decomposition methods? I reckon the latter, but it needs to have a better connection to what is being proposed further on especially when describing the reuse of convolution kernels by multiple times.  -Having said all the above, I think that the results are very interesting and demonstrate a solid computational improvement.  Is Figure 5 accurate? Have you really achieved such a reduced time in the detail enhancement, i.e. 66.1s vs 0.65s. That would be a significant improvement. By the way the font size is quite small to be honest.  -The weak point in my humble opinion is the few examples that are presented to demonstrate the computational efficiency, such as figures 4 and 5. I think further evaluation in larger data set is needed so that the bias can be reduced and ascertain the generalisation of this approach, as one could now claim that the examples presented have been selected to suit the proposed algorithm.  -Please expand slightly the conclusion if possible, as it reads as if you have been running out of space. Highlighting the main findings and contributions will enhance the paper.  Minor issues: a) Line 162, page 5 correct "approximates" to "approximate" b) Figure 3 caption second to last sentence "outputs" c) Line 273, page 8, please remove the redundant "the"


Supervising Unsupervised Learning
We introduce a framework to transfer knowledge acquired from a repository of (heterogeneous) supervised datasets to new unsupervised datasets. Our perspective avoids the subjectivity inherent in unsupervised learning by reducing it to supervised learning, and provides a principled way to evaluate unsupervised algorithms. We demonstrate the versatility of our framework via rigorous agnostic bounds on a variety of unsupervised problems. In the context of clustering, our approach helps choose the number of clusters and the clustering algorithm, remove the outliers, and provably circumvent Kleinberg's impossibility result. Experiments across hundreds of problems demonstrate improvements in performance on unsupervised data with simple algorithms despite the fact our problems come from heterogeneous domains. Additionally, our framework lets us leverage deep networks to learn common features across many small datasets, and perform zero shot learning.

By considering a probability distribution over a family of supervised datasets, the authors propose to select a clustering algorithm from a finite family of algorithms or to choose the number of clusters among other tasks by solving a supervised learning problem that matches some features of the input dataset to the output dataset. For instance, in the case of selecting the number of clusters, they regress this number from a family of datasets learning a function that gives a "correct" number of clusters.  The submission seems technically sound; the authors support the claim of the possibility of agnostic learning in two specific settings with a theoretical analysis: choosing an algorithm from a finite family of algorithms and choosing an algorithm from a family of single-linkage algorithms. Their framework also allows proposing an alternative to the desirable property of Scale-Invariance introduced by Kleinberg (2003) by letting the training datasets to establish a scale; this is translated into the Meta-Scale-Invariance desirable property. The authors then show that, with this version of the Scale-Invariance property, it is possible to learn a clustering algorithm that is also Consistent and Rich (as defined by Kleinberg (2003)).  The work is clearly written, although it would be better to explain clearly the fact that now training samples are actually training datasets (lines 87 to 94). Also, I believe there is a better way to describe the concepts related to clustering (lines 94 to 114). Making a "Conclusion" section to summarize the results for people that want to understand the impact of the work quickly would be useful.  To my knowledge, the idea of using meta-learning to propose an alternative to the Scale-Invariance desirable property to show that it is possible to achieve a good clustering is new and interesting. Moreover, the experimental results seem to back-up correctly the proposed claims which are already backed-up by a theoretical analysis which gives this work a considerable significance.


Deep Predictive Coding Network with Local Recurrent Processing for Object Recognition
Inspired by "predictive coding" -a theory in neuroscience, we develop a bidirectional and dynamic neural network with local recurrent processing, namely predictive coding network (PCN). Unlike feedforward-only convolutional neural networks, PCN includes both feedback connections, which carry top-down predictions, and feedforward connections, which carry bottom-up errors of prediction. Feedback and feedforward connections enable adjacent layers to interact locally and recurrently to refine representations towards minimization of layer-wise prediction errors. When unfolded over time, the recurrent processing gives rise to an increasingly deeper hierarchy of non-linear transformation, allowing a shallow network to dynamically extend itself into an arbitrarily deep network. We train and test PCN for image classification with SVHN, CIFAR and ImageNet datasets. Despite notably fewer layers and parameters, PCN achieves competitive performance compared to classical and state-of-the-art models. Further analysis shows that the internal representations in PCN converge over time and yield increasingly better accuracy in object recognition. Errors of top-down prediction also reveal visual saliency or bottom-up attention.

Summary: This paper proposes a method for object recognition based on the predictive coding principle from neuroscience. They use a recurrent neural network (RNN) in each layer computation of a convolutional neural network (CNN). The RNNs performs bottom-up and top-down inference of the features at each layer to minimize layer-wise prediction errors. In experiments, they authors show competitive performance against state-of-the-art recognition methods, and also outperform a baseline based on global predictive coding rather than the local predictive coding approach used here. Additionally, experiments showing analysis of the recurrent cycles within each layer in relation to classification performance.  Pros: Novel local predictive coding based architecture/formulation that improves previous predictive coding based methods Competitive performance in relation to the state-of-the-art recognition models with less parameters. Interesting analysis of the recurrent cycles within the network  Comments: Fewer number of layers argument: Throughout the paper the number of layers is used as argument to highlight the advantage of the proposed method. However, doesn’t the fact that there are RNNs within each layer implicitly add up to the number of layer within the total computation graph? When RNNs are being unrolled through time, we are technically unrolling multi-layer network with inputs at each layer and special gates to move into the next layer of computation. Can the authors give their insight on this?  Similar number of parameters comparison with baselines: In the comparison tables with the state-of-the-art, it would be nice to have comparisons with the same number of parameters. Either reduce the baselines number of parameters or (if possible) increase the number of parameters of the proposed method.  Clarity: In figure 1b, it would be nice if the authors can make it clear that the arrows between the light-gray boxes means the feature is being sent to the next computation block (if I am not mistaken) (e.g., r_l^{T-1} -> r_l^{T-1}).  Overall, I like this paper, however, it would be nice if the authors can address the issues brought up by the reviewer. The use of predictive coding for layers to converge to optimal features to improve the recognition task is attractive.


Gather-Excite: Exploiting Feature Context in Convolutional Neural Networks
While the use of bottom-up local operators in convolutional neural networks (CNNs) matches well some of the statistics of natural images, it may also prevent such models from capturing contextual long-range feature interactions. In this work, we propose a simple, lightweight approach for better context exploitation in CNNs. We do so by introducing a pair of operators: gather, which efficiently aggregates feature responses from a large spatial extent, and excite, which redistributes the pooled information to local features. The operators are cheap, both in terms of number of added parameters and computational complexity, and can be integrated directly in existing architectures to improve their performance. Experiments on several datasets show that gather-excite can bring benefits comparable to increasing the depth of a CNN at a fraction of the cost. For example, we find ResNet-50 with gather-excite operators is able to outperform its 101-layer counterpart on ImageNet with no additional learnable parameters. We also propose a parametric gather-excite operator pair which yields further performance gains, relate it to the recently-introduced Squeeze-and-Excitation Networks, and analyse the effects of these changes to the CNN feature activation statistics.

This paper proposes to augment standard deep residual networks for image classification with explicit gather-scatter operations, in order to improve propagation of useful contextual information.  It implements a per-channel gather-scatter operation.  The gather phase performs either global average pooling or max pooling (termed "picking") per channel, over an extended spatial domain.  The scatter phase upsamples the gathered feature maps back to the original size, and combines them with the input feature maps.  Experiments show integrating gather-scatter operations into residual networks yields a performance boost on the standard ImageNet classification task without any, or with minimal, parameter increase.  The paper justifiably discusses connections with Squeeze-and-Excitation networks [10].  However, it is missing citation, discussion, and comparison to two other highly related recent publications:  (1) Multigrid Neural Architectures     Tsung-Wei Ke, Michael Maire, Stella X. Yu     CVPR 2017  (2) Non-local Neural Networks     Xiaolong Wang, Ross Girshick, Abhinav Gupta, Kaiming He     CVPR 2018  The first of these works, [Ke et al, CVPR 2017] extends residual networks with a scale-space information propagation mechanism: at each layer, information flows both up and down a scale-space pyramid.  This is implemented by combining convolution operations with max-pooling, upsampling, and concatenation at every layer in the network.  These ingredients sound quite similar to the proposed gather-scatter scheme and, in fact, could be viewed as a generalization of the proposed gather-scatter approach to one which operates at all spatial scales.  Like the current paper, [Ke et al] specifically mention rapid integration of contextual cues and demonstrate improved accuracy and parameter savings on ImageNet classification using their multigrid residual network.  The second of these related works, [Wang et al, CVPR 2018], suggests inserting a non-local information gathering operation into neural networks, similar in formulation to the gather operation proposed here.  They discuss multiple functional forms and also present results of experiments modifying residual networks to include their proposed operations.  The existence of both of these published prior works decreases the relative novelty of the approach proposed here.  The fact that neither are cited is a serious problem.  Ideally, both would be cited, discussed, and compared against in experiments.  If the particular implementation proposed by this paper has advantages, a thorough experimental comparison to prior work is needed to justify them.  In light of this prior work, I do not believe the idea or current results alone are sufficiently novel or impressive to carry the paper.  ---  The rebuttal addresses my concerns about discussion of related work and addition of experimental comparison to that work. I have raised my overall score.


Partially-Supervised Image Captioning
Image captioning models are becoming increasingly successful at describing the content of images in restricted domains. However, if these models are to function in the wild -for example, as assistants for people with impaired vision -a much larger number and variety of visual concepts must be understood. To address this problem, we teach image captioning models new visual concepts from labeled images and object detection datasets. Since image labels and object classes can be interpreted as partial captions, we formulate this problem as learning from partiallyspecified sequence data. We then propose a novel algorithm for training sequence models, such as recurrent neural networks, on partially-specified sequences which we represent using finite state automata. In the context of image captioning, our method lifts the restriction that previously required image captioning models to be trained on paired image-sentence corpora only, or otherwise required specialized model architectures to take advantage of alternative data modalities. Applying our approach to an existing neural captioning model, we achieve state of the art results on the novel object captioning task using the COCO dataset. We further show that we can train a captioning model to describe new visual concepts from the Open Images dataset while maintaining competitive COCO evaluation scores.

The paper works on the problem of captioning novel object in images without paired image-caption training data, assuming object labels but no captions are available for the objects.  The paper proposes to use constrained beam search [19] as a subroutine at training time to generate a caption with includes the object labels for an image, relying on a finite state machine to constrain the beam search with multiple labels. This allows to generate new training data, and an EM algorithm is used to generate training data and learn a better captioning model  The approach is more general in the sense that it could not just handle object labels but other forms of incomplete captions, e.g. phrases, however, this is not evaluated in the paper.    Strength: - Novel and interesting direction to approach novel image object captioning - Evaluation which includes ablations and comparison to state-of-the art, including very recent work from CVPR 2018 [23]. - Outperforms all prior work (one small exception is F1 Measure, where [23]+CBS is higher)  Weaknesses: 1. While authors discuss [19], it is not obvious in section 3.1. (and line 57) that [19] uses same Fine state automaton (or is there any difference?). 2. It would have been interesting to compare to the same out of coco test as in [20,21] with F1 score for better comparability. 3. It would be interesting to see (qualitatively) what kind of captions the CBS + fine state machine gives. I guess this is quantitatively shown in row 6, Table 1? Why is the F1 score in this case not significantly higher although gt labels are used? 4. Figure 1(b): It would be great if the authors would clarify the use of the example in Figure 1(b), in general and specifically if this is also used for captioning. As well as which sequences this automaton would allow, anything which starts with “the” but does not contain “score”? a. More state-machine examples would have been interesting in supplemental, also jointly with image + gt labels examples + generated caption. 5. Minor: line 40: “This reflects the observation that, in general, object detection datasets are easier to scale”. I don’t agree that there is much evidence for this. I think it is rather an artifact on what the community focuses on. The authors might want to consider reducing the strength of this statement. However, I do strongly agree that image captioning should be able to exploit detection data as done in this work.  Conclusion: The paper presents an interesting approach for novel object image captioning with strong experimental validation which I advocate to accept.   ----  After going through the author response and reviews: + Overall the authors addressed mine and other reviewers questions and promised to add further comparisons in the final. - unfortunately, the authors did not provide the additional results already in the author response. Both for mine and the concern of not comparable networks for LRCN+CBS, brought up by R2, which I agree to.  In conclusion I recommend acceptance, in expectation that the authors will add the promised results to the final version. I also suggest to include comparable results for LRCN+CBS or a fair ablation of the author's model with the same image network.


Minimax statistical learning with Wasserstein distances
As opposed to standard empirical risk minimization (ERM), distributionally robust optimization aims to minimize the worst-case risk over a larger ambiguity set containing the original empirical distribution of the training data. In this work, we describe a minimax framework for statistical learning with ambiguity sets given by balls in Wasserstein space. In particular, we prove generalization bounds that involve the covering number properties of the original ERM problem. As an illustrative example, we provide generalization guarantees for transport-based domain adaptation problems where the Wasserstein distance between the source and target domain distributions can be reliably estimated from unlabeled samples.

This paper studies the problem of minimizing the worst-case risk over a Wasserstein ball around the underlying distribution. General risk upper bounds are provided for the procedure which minimizes the error over the Wasserstein ball around empirical distribution, which directly leads to excess risk results for Lipschitz classes. The paper also applies this result for specific model classes such as simple neural network class, RKHS, etc. And the results are also applied to domain adaptation.  The key idea in the proof of main theorems (Thm 1~3) is based on the strong duality for distributional robust stochastic optimization. Once we write it in dual form, standard Dudley chaining bounds lead to the desired result. This part is very simple but there're something smart happening here: we only need to consider the distribution class within a Wasserstein ball around empirical distribution, and to compare with best local minimax risk within this class, we don't need to pay for the constant depending on $\rho$ which does not go to zero with n.  However, since the proof of this abstract result is pretty standard, and the key ingredient (duality) is directly from previous works, I doubt if the contribution is strong enough for NIPS. The author also provides examples, but they are simply plugging existing covering number results into the bound. It would be much better if the authors can illustrate through some examples about the surprising properties of distributional robust setup. The domain adaptation part seems even less satisfactory. An important point in previous sections is that, the risk does not depend on Wasserstein distance between empirical measure and the underlying measure, which has exponential dependence on dimension. But the domain adaptation part gets back this dependence, losing the merits of previous bounds.  Detailed comments: In Theorem 2, the excess risk blows up when $\rho$ goes to zero, for $p>1$. This contradicts the intuition that less adversarial noise makes the problem easier. And the choice of $\lambda$ and the resulting risk cannot be always optimal.


Hunting for Discriminatory Proxies in Linear Regression Models
A machine learning model may exhibit discrimination when used to make decisions involving people. One potential cause for such outcomes is that the model uses a statistical proxy for a protected demographic attribute. In this paper we formulate a definition of proxy use for the setting of linear regression and present algorithms for detecting proxies. Our definition follows recent work on proxies in classification models, and characterizes a model's constituent behavior that: 1) correlates closely with a protected random variable, and 2) is causally influential in the overall behavior of the model. We show that proxies in linear regression models can be efficiently identified by solving a second-order cone program, and further extend this result to account for situations where the use of a certain input variable is justified as a "business necessity". Finally, we present empirical results on two law enforcement datasets that exhibit varying degrees of racial disparity in prediction outcomes, demonstrating that proxies shed useful light on the causes of discriminatory behavior in models.

This paper describes an optimization based approach to identify variables that result in "discrimination by proxy" in linear regression models. The authors present a definition of proxy variables, proposes an optimization problem that can be solved to identify proxies, and present a surrogate problem that can be solved efficiently. They extend the results of this approach to settings where a "proxy" variable is exempt, and illustrate how an "exempt" variable can be strengthened. The paper includes experiments on two real-world datasets, which show that the ``defensible" proxies are generates.  PROS  - Well-motivated problem / novel approach. - Well-founded definition of a proxy for linear regression models.  CONS  - Reliability of detection is not considered - Experiments on real-world data do not validate the method (as there is no ground truth) - No guidance in how to set a key parameter for proxy detection (i.e., the association threshold \epsilon)?   Major Comments: -------------  This paper considers a well-motivated problem and proposes an interesting solution. However, I am concerned about several shortcomings in the current manuscript:   1. There is no discussion about how to control false positive / false negatives in proxy detection? If I were to implement their approach, and use it to detect proxies, then the first thing that I would like to have is some degree of certainty regarding how often I am detecting / not detecting proxies. I suspect that this is far more relevant and should be prioritized.  2. There is no empirical validation of the proposed method. The authors include experiments on real-world dataset, but these do not validate the method because we do not know the ground truth in these problems (i.e., the results are not falsifiable and just "make sense")  Both 1 and 2 affect the significance of the work. Both issues could be addressed by adding experiments on a toy problem where we know the ground truth. I'd advise designing a simple simulated regression problem where they know the ground truth. The proposed method should be able to correctly detect a proxy when it exists and avoid detecting a proxy when it does not exist. This would be a good way to discuss the potential to misreport proxy use.   Post Rebuttal -------------  I would like to thank the authors for the work that they put into their paper and the rebuttal. Many of the main concerns in my original review were addressed within the rebuttal. As a result, I am raising my score to a 7.  I expect that the authors will incorporate the FPR/FNR discussion into the text (as it is important for practitioners). The experiments should also be included as they provide some more convincing evidence.  In what follows, I provide some additional comments that can be quickly addressed to improve the paper. There are not intended as criticisms of the work, but simply intended to strengthen the paper in the long run:  1. Provide guidance in how to choose a key hyperarameter for proxy detection (i.e., the association threshold \epsilon)? Some key questions to answer include:  - Should practitioners pick a single value of epsilon? Or run the analysis for different values of epsilon? In the latter case, will there be conflicting results? How should be choose?   - Can epsilon be bounded after looking at a particular dataset? I think that bounds could be derived from Problem 1. If epsilon is too large, then \beta = 0 is the only feasible solution.   - What should a practitioner expect as they increase the value of \epsilon?   2. The paper is missing a discussion on how your work could be used. Say that we have identified a proxy, now what?  3. The related work is too broad. The entire discussion on fair classification methods (e.g., Dwork/Zafar et al) could be replaced with a statement that says: "Our approach aims to explain why a linear regression model may discriminate by proxy. Other works aim to produce classifiers that do not discriminate as defined by a specific definition of fairness [cite fair classification work]." Instead you may want to cite other papers that are focused on proxy detection.   4. The writing could be improved in later parts of the paper. Some tips.  - Text describing notation could be far shorter. For example, l. 103 - 106 could be replaced with "We consider a linear regression model \hat{Y} = \sum_j \alpha_j X_j" where "\alpha_j" represents the coefficient for variable j.   - There is some colloquial language that can be corrected. For example:  - l. 181 catch


Variational Bayesian Monte Carlo
Many probabilistic models of interest in scientific computing and machine learning have expensive, black-box likelihoods that prevent the application of standard techniques for Bayesian inference, such as MCMC, which would require access to the gradient or a large number of likelihood evaluations. We introduce here a novel sample-efficient inference framework, Variational Bayesian Monte Carlo (VBMC). VBMC combines variational inference with Gaussian-process based, active-sampling Bayesian quadrature, using the latter to efficiently approximate the intractable integral in the variational objective. Our method produces both a nonparametric approximation of the posterior distribution and an approximate lower bound of the model evidence, useful for model selection. We demonstrate VBMC both on several synthetic likelihoods and on a neuronal model with data from real neurons. Across all tested problems and dimensions (up to D = 10), VBMC performs consistently well in reconstructing the posterior and the model evidence with a limited budget of likelihood evaluations, unlike other methods that work only in very low dimensions. Our framework shows great promise as a novel tool for posterior and model inference with expensive, black-box likelihoods.

Summary:  The paper considers variational inference in the case where likelihood functions themselves are expensive to evaluate. It suggests approximating the ELBO using probabilistic numerics. A Gaussian process prior is placed on the log joint of the model. A novel acquisition function is proposed along with an approximation of the ELBO for a variational mixture distribution based on the GP posterior and simple Monte Carlo for the mixture entropy. Empirical comparison is performed against a variety of relevant baselines. In meaningful synthetic experiments the proposed method outperforms other methods in higher dimensional cases. In a real example of a neuronal model the proposed models and its variants are the only ones that work.  Technical soundness:  Overall the paper is technically sound. I would have liked one more compelling real world example. I do also have some questions for the authors which would likely effect my score.  Acquisition function: I have some anxiety around the fact that the best performing acquistion function does not depend on the variational parameters (line 159) and that a method that ignores the variational aspect but keeps this search method can often give a better posterior approximation. This seems to run counter to the intuition in line 147 about a sequence of integrals under the corresponding variational approximations. Clearly there are useful things in the algorithm as a whole and the variational marginal likelihood approximation still dominates but should this have changed the emphasis of the paper?  Perhaps the (non-variational) search method in equation 8) is the real star? This also relates to some issues around the way the paper is written in the next section.  Variational mixture distributions: I have also experienced the difficulties the authors describe (footnote 2) with the mixture bound of Gershman et al 2012. Indeed my general impression was that mixture approximations are hard to get working even with MC approximations. Perhaps this is due to the fact (see Jaakkola and Jordan 1998 which the authors shouldd cite) that the improvement in ELBO is at best logarithmic in the number of mixture components and can therefore be lost beneath the Monte Carlo noise. Did the authors find that adding mixture components beyond two or three gave much improvement? Did they rule out that this was an optimization effect? Were the results sensitive to initialization?  Annealed importance sampling: I am surprised by how poorly AIS performed. My understanding from looking at the Gunter et al paper is that the proposal distribution used was a Metropolis-Hastings one. Did the authors look at multivariate slice sampling? This would likely be less sensitive to step sizes and indeed the authors proposed method calls slice sampling as a subroutine.  Clarity:  Apart from the questions discussed above the paper is well written. It references key literature. It is measured and resists hyperbole, carefully drawing the readers attention to potential limitations.  I do think that an expert would have a good chance of reproducing the key results given what is written.  I would have liked more discussion of equation 8. This seems to be a key point in the paper and has not received sufficient emphasis.  I would have really liked a visualisation of the algorithm in action in a case that was simple enough to plot. This would have helped answer my questions about the variatonal approximation above.  It is great that the authors are willing and able to release code after the review process. Why not release it in an anonymized repository at this stage next time? This would help build reviewer confidence in the submission.  I suggest that the authors include larger versions of Figures 1 and 2 in the supplement.  The references need tidying up. For instance in many cases the publication venue has been ommitted.  Originality and significance:  There is an interesting new idea here which I think could be important going forward. Specifically, I like the idea that probabilistic numerics and variational inference can be usefully combined. Given my concerns around the non-variational acquisition function I'm not sure it has been fully exploited yet. The solid experimental contribution should also be given credit.  References:  Gunter, Tom and Osborne, Michael A and Garnett, Roman and Hennig, Philipp and Roberts, Stephen J (2014), Sampling for Inference in Probabilistic Models with Fast Bayesian Quadrature, Advances in Neural Information Processing Systems 27 Gershman, S., Hoffman, M., & Blei, D. (2012) Nonparametric variational inference. International Conference on Machine Learning. Tommi S. Jaakkola , Michael I. Jordan  (1998) Improving the Mean Field Approximation via the Use of Mixture Distributions. Learning in Graphical Models.  Post author feedback comments:  I have read the author response. I am satisfied with their reply and have increased my score.


Non-Adversarial Mapping with VAEs
The study of cross-domain mapping without supervision has recently attracted much attention. Much of the recent progress was enabled by the use of adversarial training as well as cycle constraints. The practical difficulty of adversarial training motivates research into non-adversarial methods. In a recent paper, it was shown that cross-domain mapping is possible without the use of cycles or GANs. Although promising, this approach suffers from several drawbacks including costly inference and an optimization variable for every training example preventing the method from using large training sets. We present an alternative approach which is able to achieve non-adversarial mapping using a novel form of Variational Auto-Encoder. Our method is much faster at inference time, is able to leverage large datasets and has a simple interpretation.

The manuscript considers the problem of unsupervised mapping across domains by use of variational auto-encoders (VAE) instead of the current adversarial training (GAN) approach.  The approach extends a recently proposed non-adversarial mapping (NAM) framework extracting a mapping from a pretrained generative model to a target domain with the present addition of a probabilistic encoder which maps the target image y to the latent domain z_y, i.e. based on the optimization given in equation 2 of the manuscript.  Quality: The issue of mapping between domains is interesting and the framework presented seems new and useful. It however also appears to be a straightforward extension to NAM but providing computational merits and improved performance in terms of the quality of mapping which could warrant publication. The experimentation is somewhat limited and the presentation of the paper could be improved by proofreading the manuscript.  Clarity: The basic idea of VAE-NAM appears intuitive, i.e. from a generative model of X formed by G(z) create an encoder that encodes Y in this latent space z_y=E(y) such that a transformation T of the corresponding X resembles Y, i.e. minimize ||T(G(E(y))),y|| as defined in equation 2 treating the encoder with uncertainty using the framework of VAE.  However, the manuscript could be clearer in its presentation. First of all the encoder is called C throughout the manuscript and in equation 2 called E which causes confusion. Furthermore, second section of the VAE-NAM description is unclear. Should x be y in the first two lines, i.e. line 167 and 168? In section 4.1 line 234 is referenced to table 1 I believe this should be Figure 1. There is further several spelling issues, i.e., is key component -> is a key component analogies the can -> analogies that can  this was found at preserving -> this was found to preserve aforemntioned -> aforementioned direct to code -> directly to code very recently began to generate -> very recently begun to generate  Originality: The manuscript appears to be a straightforward extension of NAM to include a variational autoencoder and the work appears somewhat incremental.  Significance The results are compelling and the procedure seem to have merits both computationally and in quality of results that may warrant publication. Avoiding the adversarial training and the challenges imposed there also makes the current framework attractive with a potential impact.   I thank the authors for their response and find the paper more compelling given their response to the issues raised. I have accordingly increased my evaluation of the paper from just below to marginally above the acceptance threshold.