Differential Privacy for Growing Databases
The large majority of differentially private algorithms focus on the static setting, where queries are made on an unchanging database. This is unsuitable for the myriad applications involving databases that grow over time. To address this gap in the literature, we consider the dynamic setting, in which new data arrive over time. Previous results in this setting have been limited to answering a single nonadaptive query repeatedly as the database grows [DNPR10, CSS11]. In contrast, we provide tools for richer and more adaptive analysis of growing databases. Our first contribution is a novel modification of the private multiplicative weights algorithm of [HR10], which provides accurate analysis of exponentially many adaptive linear queries (an expressive query class including all counting queries) for a static database. Our modification maintains the accuracy guarantee of the static setting even as the database grows without bound. Our second contribution is a set of general results which show that many other private and accurate algorithms can be immediately extended to the dynamic setting by rerunning them at appropriate points of data growth with minimal loss of accuracy, even when data growth is unbounded.

This paper studies how to answer queries on a dynamically growing database while satisfying differential privacy.  It first offers a concrete algorithm, along with formal privacy and utility guarantees, for the linear query case. This algorithm, PMWG, relies heavily upon PMW, which maintains a public histogram to answer a number of linear queries exponential in the database size. PMWG then tweaks PMW to essentially renormalize (independent of the new data) this histogram upon the addition of new data as if that data is uniform -- thereby getting around the histogram's "overfitting" to old data -- and loses only constant factors in accuracy.  Second, the paper provides a general black-box transformation from any sufficiently private and accurate static algorithm to a private and accurate dynamic one, presenting ERM as an application. The paper actually provides two transformations with slightly different accuracy guarantee, but they share a similar structure: use a sequence of epsilons {eps_i} that decays as the database size increases, so that the epsilons sum to a fixed epsilon while the database growth allows a constant (or increasing) level of accuracy even with ever-tighter privacy requirements.  Along the way, the paper introduces and uses a number of dynamic versions of canonical static differentially private algorithms, including private multiplicative weights and sparse vector.  I did not check the proofs in the Supplement, but I did look at all Lemma and Theorem statements.  Positive: The dynamic database setting is a natural and practical one for differential privacy, and I think this paper is a good first cut on the topic. The actual theoretical results are not especially surprising -- I think most experts in the area already have some hazy idea that algorithms like PMWG, BBSchedule, and BBImprove exist. Still, it's good to know that the vague intuitive approach actually works out formally, and it's not like the required analysis is trivial. I think the fact that this paper actually works it out formally for a few dynamic versions of static workhorse differentially private algorithms makes it a good contribution to the field. In my opinion, PMWG and the dynamic versions of sparse-* algorithms are enough to be a fine NIPS paper.  Negative (in order of decreasing importance): 1. Section 4, on BBScheduler and BBImprove, are hard for me to follow. Some of this is perhaps intrinsic to the problems, but for example I still don't know what that g really means, and the difference between BBScheduler and BBImprove is kind of muddy -- honestly, I think it would make more sense to forget BBScheduler altogether, though I don't expect this to happen.  2. Why do the log(n) factors in the dynamic accuracy bounds for PMWG (Theorem 5) not appear in Theorem 1, whose surrounding text repeatedly describes this error as "tight" wrt PMW? I don't think this factor is a big deal, but I'm confused as to why it apparently doesn't count.  3. The paper's exposition has room for improvement. As mentioned above, most of these algorithms have reasonably intuitive interpretations, but these weren't very clear on the first read, e.g. just reframing "updated as if the new data came from a uniform distribution" as a version of normalization or saying "epsilon decay is tied to database growth so that the two balance out/database growth eventually dominates" would have helped me. These are the punchlines, no? It's nice to have memorable takeaways, as is it's easy to miss them and get lost in all of the parameters floating around.  4. The intro is redundant with the abstract and at times is almost a copy. To some extent this is an issue in every paper, but given the (IMO) better uses of space mentioned above, perhaps some reorganization would help. Section 4.2.1, which quickly discusses private ERM for growing databases, feels pretty extraneous -- everybody reading this paper knows what ERM is, no? Seems like this can be a single sentence ("We include details for an example application of BBImprove for private ERM on growing databases in the Supplement.")  5. The organization of the supplement reverses the order of discussion from the main body -- the main body covers PMWG first and then the black-box transformation, and the Supplement does the opposite. Minor, but odd.  6. There are several broken references, e.g. on lines 108, 227, 545. The main body also appears to be too long (???) stretching onto a 9th page. Typos "We give the parameters the" on line 292, "Accurary" on line 588.  Overall: Though I've provided more negative points than positive ones, the positives outweigh the negatives (which are mostly presentation-focused) for me. This paper provides a nice (if not very surprising) analysis of a useful problem, and develops the (scant) existing theory on "dynamic database" differential privacy along several fronts.  I therefore vote to accept.  ---Update After Response--- I read the authors' response and found the clarifications useful. I'm keeping my score as is, and am willing to argue some for acceptance.


Exploration in Structured Reinforcement Learning
We address reinforcement learning problems with finite state and action spaces where the underlying MDP has some known structure that could be potentially exploited to minimize the exploration rates of suboptimal (state, action) pairs. For any arbitrary structure, we derive problem-specific regret lower bounds satisfied by any learning algorithm. These lower bounds are made explicit for unstructured MDPs and for those whose transition probabilities and average reward functions are Lipschitz continuous w.r.t. the state and action. For Lipschitz MDPs, the bounds are shown not to scale with the sizes S and A of the state and action spaces, i.e., they are smaller than c log T where T is the time horizon and the constant c only depends on the Lipschitz structure, the span of the bias function, and the minimal action sub-optimality gap. This contrasts with unstructured MDPs where the regret lower bound typically scales as SA log T . We devise DEL (Directed Exploration Learning), an algorithm that matches our regret lower bounds. We further simplify the algorithm for Lipschitz MDPs, and show that the simplified version is still able to efficiently exploit the structure.

*Summary* This paper deals with undiscounted reinforcement learning. It provides problem-related (asymptotic) lower and upper bounds on the regret, the latter for an algorithm presented in the paper that builds on Burnetas and Katehakis (1997) and a recent bandit paper by Combes et al (NIPS 2017). The setting assumes that an "MDP structure" \Phi (i.e. a set of possible MDP models) is given. The regret bounds (after T steps) are shown to be of the form K_Phi*log T, where the parameter K_\Phi is the solution to a particular optimization problem. It is shown that if \Phi is the set of all MDPs ("the unstructured case") then K_\Phi is bounded by HSA/\delta, where H is the bias span and \delta the minimal action sub-optimality gap. The second particular class that is considered is the Lipschitz structure that considers embeddings of finite MDPs in Euclidian space such that transition probabilities and rewards are Lipschitz. In this case, the regret bounds are shown to not to depend on the size of state and action space anymore.   *Evaluation* In my opinion this is an interesting theoretical contribution to RL. Although the analysis is somewhat restricted (ergodic MDPs, asymptotic analysis) and the paper considers only Lipschitz structures, I think the presented results are an important first step in a promising research direction. I did not have time to check the appendix, and the main part does not present any proof sketches either, so I cannot judge correctness. The paper is well-written, if a bit technical. A bit more intuition on how the algorithm works would have been welcome. Overall, in my opinion this is a clear accept.      *Comments* - In footnote 4, I think the uniqueness is about the optimal policy, not the bias span. - Algorithm: The role of the quantities s_t(x) and the parameters \epsilon, \gamma should be explained. Also, I wondered whether the minimization in the Exploit step is necessary, or whether an arbitrary element of \mathcal{O} could be chosen as well. An explanation of the difference of the Estimate and Explore steps would be helpful as well.   - In l.272 it is claimed that any algorithm for undiscounted RL needs to recompute a new policy at each step. This is not true. Actually, e.g. UCRL and similar algorithms wouldn't work if policies were recomputed at each step. - In the last line "unknown probabilistic rewards" are mentioned as future work, which I did not understand as the current paper (as far as I understood) already deals with such rewards.  *Typos and Minor Stuff* - l.112: Footnote 2 is placed unfortunate.  - l.142: "state"->"states" - l.154: "algorithms"->"algorithm" - eq.(2b): Delete dot in formula, replace comma by dot. - l.186: "are"->"is" - l.196: "Ee"->"We" - l.203: "makes"->"make" - l.205: Delete "would". - l.225: D might be confused with the diameter. - l.239: "model"->"models" - l.250: "a MDP"->"an MDP" - l.253: "on optimal"->"of the optimal" - l.256: "apparent best"->"apparently best" - l.260: "phase"->"phase)" - l.265: "model"->"model structure" - l.278: "an linear programming"->"a linear programming problem" - l.280: Delete "a". - l.282: "algorithm"->"algorithms" - l.284: "Works"->"Work" - l.300: "An other"->"Another" - l.319: Capitalize "kullback-leibler".  PS: Thanks for the detailed comments in the feedback.


Disconnected Manifold Learning for Generative Adversarial Networks
Natural images may lie on a union of disjoint manifolds rather than one globally connected manifold, and this can cause several difficulties for the training of common Generative Adversarial Networks (GANs). In this work, we first show that single generator GANs are unable to correctly model a distribution supported on a disconnected manifold, and investigate how sample quality, mode dropping and local convergence are affected by this. Next, we show how using a collection of generators can address this problem, providing new insights into the success of such multi-generator GANs. Finally, we explain the serious issues caused by considering a fixed prior over the collection of generators and propose a novel approach for learning the prior and inferring the necessary number of generators without any supervision. Our proposed modifications can be applied on top of any other GAN model to enable learning of distributions supported on disconnected manifolds. We conduct several experiments to illustrate the aforementioned shortcoming of GANs, its consequences in practice, and the effectiveness of our proposed modifications in alleviating these issues.

After the rebuttal and discussions with the other reviewers I increased my overall rating from 4 to 5. I agree with the authors and the other reviewers that the observation that a disconnected data manifold can negatively affect GAN performance is important. However, I think the authors should have conducted more experiments to investigate if this problem really arises in practice (e.g. using lower dimensional embeddings in some feature space). I also disagree with the authors that the MSD measure in Euclidean space is a good measure, as Euclidean distance is known to be rather meaningless for images. Moreover, the experimental validation should *always* include the simple baseline where the generators are all trained jointly without the InfoGAN-loss. In the rebuttal, the authors show that this baseline performs (slightly) worse than the proposed method for the toy example, but I think it's vital to include this baseline in all experiments (especially in the celebA + LSUN experiment). I also think it is important to disentangle the effect of the increased model complexity of the proposed method from the effect of the regularization.  ==================================================================  # Summary This paper discusses the potential limitation of current state-of-the-art generative models in modeling probability distributions whose support is concentrated on a union of disjoint submanifolds. To circumvent this problem, the authors propose to use a simple variant of InfoGAN [1] with discrete variables.  # Significance The observation that real-world data distribution might be concentrated on a union of disjoint submanifolds  is important and hints at a possible limitation of current state-of-the-art models. Unfortunately, the authors do not directly verify this hypothesis but instead use it to motivate their method which appears to be identical to InfoGAN [1] with discrete latent variables (+ a learnable prior). It is also not clear why the authors need the InfoGAN architecture to get a good generative model and why the same effect could not be achieved by simply including discrete latent variables into the model (the explanation in ll. 113-114 is not really satisfying as it is not verified experimentally).  # Clarity The paper is well-written and easy to follow. Including an example in ll. 17-18 is a nice start for the paper. However, this particular example is not really a good one, as it is certainly possible to transform a tree into a bird, e.g. by simply "zooming" into the tree where a bird sits.  # Experiments The experiments are interesting, but the evaluation is a bit unclear. Was the MSD computed in Euclidean space or Feature space (ll. 176-185)? Euclidean space is probably not a good choice for an evaluation metric here, as the Euclidean distance between images is often meaningless. The discussion in ll. 192-202 why overfitting cannot occur is also not convincing, as it is not experimentally verified. A good evaluation metric should certainly be able to detect overfitting. The hypothesis that real world data distributions often contain multiple disconnected components should also be verified experimentally, e.g. by using lower dimensional embeddings of the distribution. The experiment on the union of celebA and LSUN bedrooms is interesting, but also a bit artificial, as here it is very clear that this particular probability distribution has multiple disjoint components. How do similar experiments on celebA and/or LSUN alone look like?   # Other comments / questions * There is an extra "." in l. 11 * "no definite reason" in l. 35 should probably be replaced with "no satisfying explanation"  * The discussion in ll. 99-104 is interesting, but when the generator cannot model the support of the true data distribution the realizability assumption of [2,3] also breaks down, which would probably be an even more compelling argument * The explanation that we need to "encourage these generators to each focus on a different submanifold of the real data" (ll. 113-114) is not really satisfying, as this could simply be learned by the generators. There are also no experiments in this paper that verify this claim. * The relationship between "partially covered real submanifolds" and "mode collapse" (l. 114) is completely unclear. * "corss entropy" should read "cross entropy" (l. 160)  [1] Chen, Xi, et al. "Infogan: Interpretable representation learning by information maximizing generative adversarial nets." Advances in neural information processing systems. 2016. [2] Nagarajan, Vaishnavh, and J. Zico Kolter. "Gradient descent GAN optimization is locally stable." Advances in Neural Information Processing Systems. 2017. [3] Mescheder, Lars, Andreas Geiger, and Sebastian Nowozin. "Which Training Methods for GANs do actually Converge?." International Conference on Machine Learning. 2018.


Expanding Holographic Embeddings for Knowledge Completion
Neural models operating over structured spaces such as knowledge graphs require a continuous embedding of the discrete elements of this space (such as entities) as well as the relationships between them. Relational embeddings with high expressivity, however, have high model complexity, making them computationally difficult to train. We propose a new family of embeddings for knowledge graphs that interpolate between a method with high model complexity and one, namely Holographic embeddings (HOLE), with low dimensionality and high training efficiency. This interpolation, termed HOLEX, is achieved by concatenating several linearly perturbed copies of original HOLE. We formally characterize the number of perturbed copies needed to provably recover the full entity-entity or entity-relation interaction matrix, leveraging ideas from Haar wavelets and compressed sensing. In practice, using just a handful of Haar-based or random perturbation vectors results in a much stronger knowledge completion system. On the Freebase FB15K dataset, HOLEX outperforms originally reported HOLE by 14.7% on the HITS@10 metric, and the current path-based state-of-the-art method, PTransE, by 4% (absolute).
must balance computational complexity against model complexity. Not surprisingly, embedding models capable of capturing rich correlations in relational data often have limited computational scalability. On the other hand, models that can be trained efficiently are often less expressive.
We focus on two compositional operators. The first is the full tensor product [18], which captures correlations between every pair of dimensions of two embedding vectors in R d , by considering their outer product. The resulting quadratic (d 2 ) parameter space makes this impractical even for modestly sized KGs. The second is the circular correlation underlying holographic embedding or HOLE [19], which is inspired by holographic models of associated memory. Notably, HOLE keeps the parameter space linear in d by capturing only the sum along each diagonal of the full tensor product matrix.
Our main contribution is a new compositional operator that combines the strengths of these two models, resulting in much stronger knowledge completion system. Specifically, we propose expanded holographic embeddings or HOLEX, which is a collection of models that interpolates between holographic embeddings and the full tensor product.
The idea is to concatenate l ≥ 1 copies of the HOLE model, each perturbed by a linear vector, allowing various copies to focus on different subspaces of the embedding. HOLEX forms a complete spectrum connecting HOLE with the full tensor product model: it falls back to HOLE when l = 1 and all entries in the perturbation vector are non-zero, and is equivalent to the full tensor product model when l = d, the embedding dimension, and all perturbation vectors are linearly independent.
We consider two families of perturbation vectors, low frequency Haar wavelets [6,10] and random 0/1 vectors. We show that using the former corresponds to considering sums of multiple subsequences of each diagonal line of the full product matrix, in contrast to the original holographic embedding, which sums up the entire diagonal. We find that even just a few low frequency vectors in the Haar matrix are quite effective in practice for HOLEX. When using the complete Haar matrix, the length of the subsequences becomes one, thereby recovering the full tensor product case. Our second family of perturbation vectors, namely random 0/1 vectors, corresponds to randomly sub-selecting half the rows of the tensor product matrix in each copy. This is valuable when the full product matrix is sparse. Specifically, using techniques from compressed sensing, if each diagonal line is dominated by a few large entries (in terms of absolute values), we show that a logarithmic number of random vectors suffice to recover information from these large entries.
To assess its efficacy, we implement HOLEX using the framework of ProjE [23], a recent neural method developed for the Freebase FB15K knowledge completion dataset [3,5], where the 95% confidence interval for statistical significance is 0.3%. In terms of the standard HITS@10 metric, HOLEX using 16 random 0/1 vectors outperforms the original HOLE by 14.7% (absolute), ProjE by 5.7%, and a path-based state-of-the-art method by 4%.

In this paper, the authors introduce a method to interpolate between RESCAL and HolE in the setting of knowledge base completion. Whereas RESCAL models the score of a triple (o, r, s) as   (with x the tensor product), HolE models the score of a triple (o, r, s) as  where conv is a circular convolution.  The authors show that by adding a hadamard product with e_o and a vector c_j, with k independent c_j, we obtain a rank-k linear transform of the tensor product e_o x x_s. Thus, by increasing the rank, the authors interpolate in expressivity between RESCAL and HolE. The next part of the paper introduces 2 specific bases, based on haar wavelets and bernoulli vectors. Finally the authors test their algorithms on FB15K.  I like the main idea to interpolate between HolE and RESCAL, but didn't understand section 3.1 and 3.2. It seems to me that the only criteria for expressivity is the rank of C_d. In this case, why not use a truncated Identity matrix for C_d? Why is there a difference in performance between the Haar basis and the bernoulli basis ?  Regarding the Haar basis, why are we happy to get partial sums of the diagonals ?   As currently explained, I didn't understand the value of theorem 2 and 3. Why would we want to "pin-point" (and what does it mean) non-zero values? is it realistic to have a sparse tensor product matrix ? (embeddings are not asked to be sparse, I don't see where the sparsity in this matrix would come from)  I think this part should be re-written, maybe using figures, and theorem 2/3 should be written in a more formal way. The same comment on formality applies to the proofs of theorem 1 and 2, which are elementary but hard to follow because objects are not properly defined before they are used.   It seems natural to introduce what would happen with a truncated identity matrix. This would amount to only using the k first values in the left-hand side vector, which would lead to a RESCAL with a matrix of size (k,d) pre-multiplied by a projector on the first k vectors. This begs the question, is this method very different from a low-rank RESCAL ? I think this comparison would be much more informative than the current section 3.  Regarding experiments, I find the state of the art badly represented. Notably "Knowledge Base Completion: Baselines Strike Back" by Rudolf Kadlec et al. obtains a hit@10 of 89.3 with DistMult, which is higher than the results reported here for HolEx. Thus, the claim in the abstract about beating the state of the art seems wrong.   ---------------------------- AFTER REBUTTAL ----------------------------  Thank you for your feedback, and for discussing the truncated identity matrix solution. Since this doesn't work as well as the Haar or Bernoulli "basis", I would think the real justification for what this work does lies in fourier space, rather than in a low-rank approximation of the tensor product.  After discussion, I change my decision to a 6. I believe that the theory is not currently satisfying and / or supported by experiments.  - low rank approximation of tensor product doesn't seem to be what makes this work. - no experimental evidence that the tensor product estimated are sparse. (which kills justification based on sensing) (regarding the author's feedback, maybe the type "person" won't be encoded on one dimension, but on a linear combination of dimensions. Since everything is initialized at random, I find it hard to believe that the canonical basis will have any meaning in the final embeddings) - no experimental evidence of a full "interpolation" between HolE and RESCAL (i.e., an experiment that shows HolE and RESCAL performances and how this method's performance get close to either HolE or RESCAL depending on the size of the basis used) With these, this paper would easily go from a 6 to a 9.


Regret Bounds for Robust Adaptive Control of the Linear Quadratic Regulator
We consider adaptive control of the Linear Quadratic Regulator (LQR), where an unknown linear system is controlled subject to quadratic costs. Leveraging recent developments in the estimation of linear systems and in robust controller synthesis, we present the first provably polynomial time algorithm that provides high probability guarantees of sub-linear regret on this problem. We further study the interplay between regret minimization and parameter estimation by proving a lower bound on the expected regret in terms of the exploration schedule used by any algorithm. Finally, we conduct a numerical study comparing our robust adaptive algorithm to other methods from the adaptive LQR literature, and demonstrate the flexibility of our proposed method by extending it to a demand forecasting problem subject to state constraints.
Optimism in the Face of Uncertainty (OFU): Abbasi-Yadkori and Szepesvári [1], Faradonbeh et al. [8], and Ibrahimi et al. [11] employ the Optimism in the Face of Uncertainty (OFU) principle [5], which optimistically selects model parameters from a confidence set by choosing those that lead to the best closed-loop (infinite horizon) control performance, and then plays the corresponding optimal controller, repeating this process online as the confidence set shrinks. While OFU in the LQR setting has been shown to achieve optimal regret O( √ T ), its implementation requires solving a non-convex optimization problem to precision O(T −1/2 ), for which no provably efficient implementation exists.
Thompson Sampling (TS): To circumvent the computational roadblock of OFU, recent works replace the intractable OFU subroutine with a random draw from the model uncertainty set, resulting in Thompson Sampling (TS) based policies [2,4,16]. Abeille and Lazaric [4] show that such a method achieves O(T 2/3 ) regret with high-probability for scalar systems. However, their proof does not extend to the non-scalar setting. Abbasi-Yadkori and Szepesvári [2] and Ouyang et al. [16] consider expected regret in a Bayesian setting, and provide TS methods which achieve O( √ T ) regret. Although not directly comparable to our result, we remark on the computational challenges of these algorithms. Whereas the proof of Abbasi-Yadkori and Szepesvári [2] was shown to be incorrect [15], Ouyang et al. [16] make the restrictive assumption that there exists a (known) initial compact set Θ describing the uncertainty in the system parameters, such that for any system θ 1 ∈ Θ, the optimal controller K(θ 1 ) is stabilizing when applied to any other system θ 2 ∈ Θ. No means of constructing such a set are provided, and there is no known tractable algorithm to verify if a given set satisfies this property. Also, it is implicitly assumed that projecting onto this set can be done efficiently.

The authors propose a polynomial time algorithm for robust adaptive control of unknown linear systems (with quadratic objective functions). The algorithm provides a sublinear regret of O(T^{2/3}) with high probability. Moreover, a lower bound on the expected regret is provided by studying the relation between system parameter estimation and regret minimization problems.   The paper is well written, the problem is relevant, and the authors succeed in effectively placing their work relative to the existing literature. I have to admit that I haven't been able to follow details of many of the technicals developments provided in the supplementary material. Furthermore, several issues should be addressed in the revised version.   High-level comments:  - It would be useful if the authors could please explain why regret provides a proper metric for quantifying performance of an algorithm with uncertain parameters. I recognize that several other references also use regret but I'm not sure why this is a proper metric for evaluating performance of controllers for uncertain systems.   - The paper provides probabilistic guarantees. Since there are many sources of stochastic uncertainty, it would be useful if the authors could please clarify what feature of the problem leads to the results that hold with high probability. I would also like to understand how the choice of parameter \delta in the problem setup affects the algorithm.   - The system-level synthesis appears to be a key technical enabler for the controls part of the paper. This approach leads to an infinite-dimensional synthesis problem and the authors go around this issue by solving a finite-dimensional approximation based on FIR. In contrast to standard LQR, the resulting controller appears to be dynamic and it is not clear how much conservatism FIR restriction introduces and how the FIR truncation length  influences the performance. It would be useful if the authors could provide some additional comments about this.    - Technical comments:  - According to the Appendix G.3, the 8th line of the algorithm which is an infinite-dimensional problem can be brought into a finite-dimensional SDP using an approximation based on FIR. Since solving large SDPs is in general challenging, the computational complexity should be discussed. Please explain how such problems can be handled for large-scale systems.   - While the OFU-based method can achieve an optimal regret in LQR problems, the authors state that there is no efficient algorithm for solving non-convex optimization problems in OFU. Along the same lines as in the previous comment, the authors should provide computational complexity analysis of their algorithm and compare it to OFU for a large-scale system. This is important mainly because of recent developments in solving the non-convex optimization subroutine encountered in OFU. On a related note, the paper would benefit from a larger numerical example in Section 4 to better compare the efficiency of various methods against one another.  - Line 6 of the algorithm uses a least-square approach to find the estimates of the matrices A and B. However, this method does not necessarily provide a correct estimate of A (e.g., the estimate \hat{A} may turnout to be stable for an unstable system which may have a detrimental influence on the performance of the designed controller). Moreover, Theorem A.2 depends on the existence of (I + \hat{\Delta})^{-1}, where \hat{\Delta} is unknown. Since the authors utilize non-standard techniques based on recently introduced system-level synthesis, some additional discussion would be useful on how the tools from robust control theory lead to such an estimate. Please also comment on how this affects the practical aspect of the proposed approach and how one should determine the constants of Algorithm 1 (e.g. C_x, C_u, and \rho) when the dynamics are unknown.   - Showing the performance of various algorithms against the number of iterations can be misleading, especially when the algorithms involve subroutines that require different computational complexity. In addition to Figure 1, please also provide performance curves as a function of computation time.  - Please discuss sensitivity of the performance of the proposed algorithm to the choices of C_\eta, \gamma, and the FIR truncation length F.  - It is stated that the outer minimization in Step 8 of the algorithm can be solved via bisection. For larger values of gamma, the inner minimization problem is relaxed meaning that the optimal objective of the inner minimization becomes smaller. However, the coefficient 1/(1-gamma) in the outer minimization increases. Hence, it is unclear why a bisection can find the optimal gamma.  Minor issues:  - How does the assumption of Gaussian noise influence the results (if at all)?  - Line 226: please provide a reference for the heuristic projected gradient descent implementation of OFU.  - Line 43: please define ’T’.  - It would be useful to explain the choice of parameter \sigma_{\eta,i}^2 inside the algorithm as well.  - Line 34: please explain what is meant by ‘PAC-learnable’.  - Line 103: “thereore” should be “therefore”.  - Line 181: "degredation"  - Line 183: “,” should be “.”.  - Line 383: "problemmmm"


Lifelong Inverse Reinforcement Learning
Methods for learning from demonstration (LfD) have shown success in acquiring behavior policies by imitating a user. However, even for a single task, LfD may require numerous demonstrations. For versatile agents that must learn many tasks via demonstration, this process would substantially burden the user if each task were learned in isolation. To address this challenge, we introduce the novel problem of lifelong learning from demonstration, which allows the agent to continually build upon knowledge learned from previously demonstrated tasks to accelerate the learning of new tasks, reducing the amount of demonstrations required. As one solution to this problem, we propose the first lifelong learning approach to inverse reinforcement learning, which learns consecutive tasks via demonstration, continually transferring knowledge between tasks to improve performance.

Summary:  This paper considers the problem of lifelong inverse reinforcement learning, where the goal is to learn a set of reward functions (from demonstrations) that can be applied to a series of tasks. The authors propose to do this by learning and continuously updating a shared latent space of reward components, which are combined with task specific coefficients to reconstruct the reward for a particular task.   The derivation of the algorithm basically mirrors the Efficient Lifelong Learning Algorithm (ELLA) (citation [33]). Although ELLA was formulated for supervised learning, variants such as PG-ELLA (not cited in this paper, by Ammar et al. “Online Multi-task Learning for Policy Gradient Methods”) have applied the same derivation procedure to extend the original ELLA algorithm to the reinforcement learning setting. This paper is another extension of ELLA, to the inverse reinforcement learning setting, where instead of sharing policies via a latent space, they are sharing reward functions.    Other Comments:  How would this approach compare to a method that used imitation learning to reconstruct policies, and then used PG-ELLA for lifelong learning?  To clarify, ELIRL can work in different state/action spaces, and therefore also different transition models. However, there is an assumption that the transition model is known. Specifically it would be necessary to have the transition model to solve for pi^{alpha^(t)} when computing the hessian in Equation (5).   I liked the evaluation procedure in the experiments, where we can see both the difference between reward functions and the difference between returns from the learned policies. Though I’m not sure what additional information the Highway domain gives compared to the Objectworld.   On line 271, k was chosen using domain knowledge. Are there any guidelines for choosing k in arbitrary domains? What was the value of k for these experiments?  Please cite which paper GPIRL comes from on line 276. I don’t remember seeing this acronym in the paper (even on line 48).    Minor/Typos:  In Algorithm 1, I believe the $i$ should be a $j$ (or vice versa)


Fighting Boredom in Recommender Systems with Linear Reinforcement Learning
A common assumption in recommender systems (RS) is the existence of a best fixed recommendation strategy. Such strategy may be simple and work at the item level (e.g., in multi-armed bandit it is assumed one best fixed arm/item exists) or implement more sophisticated RS (e.g., the objective of A/B testing is to find the best fixed RS and execute it thereafter). We argue that this assumption is rarely verified in practice, as the recommendation process itself may impact the user's preferences. For instance, a user may get bored by a strategy, while she may gain interest again, if enough time passed since the last time that strategy was used. In this case, a better approach consists in alternating different solutions at the right frequency to fully exploit their potential. In this paper, we first cast the problem as a Markov decision process, where the rewards are a linear function of the recent history of actions, and we show that a policy considering the long-term influence of the recommendations may outperform both fixed-action and contextual greedy policies. We then introduce an extension of the UCRL algorithm (LINUCRL) to effectively balance exploration and exploitation in an unknown environment, and we derive a regret bound that is independent of the number of states. Finally, we empirically validate the model assumptions and the algorithm in a number of realistic scenarios.

The paper changes the model assumptions for recommender systems (RS) to capture phenomena of real world data that has been largely ignored up to now. Instead of assuming a fixed preference over time, the utility of a recommendation is based on the frequency of previous occurrences in a time window w. This captures the fact that humans get bored of repetition. The authors do a great job in showing that this effect occurs in real data. However they still make quite restrictive model assumptions, EDIT{misunderstood this part in the paper, remove comment: i.e. that the utility of an action is only based on the frequency it occurred over the last w times, without taking the positioning into account. The question is whether their model is a good trade-off between computational feasibility and how well it captures the data. To complete the picture, it would be nice to see how much gain in accuracy there is from considering more complex functions that depend on when exactly a recommendation was made. } The main contribution, beside defining a novel model, is an adaptation of UCRL to this problem. The algorithm exploits that the setting is reduced to a deterministic MDP with linear reward function. They use standard techniques to proof an upper bound for this problem, that has favourable dependencies on the problem parameters. The math seems sound, but I haven't checked every detail in the appendix. The paper is well written and easy to understand. The performance of the algorithm is verified in several experiments, which are semi-synthetic. The experimental setup is reasonable and shows convincingly the advantages of the new model.  The biggest downside of this algorithm is, that it only works for small a window w. The authors correctly note that the regret scales fine in w. But they should highlight that the run-time is actually exponential in w. Thereby making this algorithm computationally infeasible even for moderate window sizes.


Learn What Not to Learn: Action Elimination with Deep Reinforcement Learning
Learning how to act when there are many available actions in each state is a challenging task for Reinforcement Learning (RL) agents, especially when many of the actions are redundant or irrelevant. In such cases, it is sometimes easier to learn which actions not to take. In this work, we propose the Action-Elimination Deep Q-Network (AE-DQN) architecture that combines a Deep RL algorithm with an Action Elimination Network (AEN) that eliminates sub-optimal actions. The AEN is trained to predict invalid actions, supervised by an external elimination signal provided by the environment. Simulations demonstrate a considerable speedup and added robustness over vanilla DQN in text-based games with over a thousand discrete actions.

This paper addresses the issue of having a very large action space when solving a Reinforcement Learning (RL) problem. The authors propose the Action-Elimination Deep Q-Network (AE-DQN) which is a model that eliminates sub-optimal actions. The training process requires an external supervised signal to learn which actions should be eliminated given the current state. The authors validate their approach by conducting experiments on text-based games where the number of discrete actions is over a thousand. They show a considerable speedup and added robustness over vanilla DQN.  More precisely, in the model being proposed, the elimination signal is not used to simply shape the environment reward. Instead, it is decoupled from the MDP by using contextual multi-armed bandits. An Action Elimination Network learns a mapping from states to elimination signal, and given a state provides the subset of actions that should be used by the DQN (trained using standard Q-Learning). In addition, the authors show that their Action Elimination Q-Learning converges to find an optimal policy and a minimal valid action space in the contextual bandits setting.  Main contributions: 1) Action-Elimination Deep Q-Network (AE-DQN) which is a model that eliminates sub-optimal actions in very large actions space using an external action elimination signal. 2) Convergence proof of the proposed Action Elimination Q-learning. 3) State of the art results on the text-based game Zork1.  Pros: - A technique that deals with large action space is of great interest to the community; - Text-based games offer interesting challenges for RL algorithms (stochastic dynamics, delayed reward, long-term memory, language understanding) and have been little explored. - Overall, the paper is well written, and the authors communicate clearly the motivation and main ideas of the proposed method.  Cons: - Extracting an elimination signal from most environments is not trivial. To be fair, the authors do address this concern in their conclusion. - It is unclear how the elimination signal is obtained for Zork1. Usually, in text-based games, that signal comes in the form of text which would need to be understood first (maybe some sort of sentiment analysis model?). - It is unclear to me what is the target of the NLP CNN classification. How does one determine the actions' relevance in each state? Is it by using the elimination signal?  Minor issues: - I was not able to open the 'run_cpu' file in the anonymized code repo. From footnote #2, it seems the implementation of the elimination signal for Zork1 can be found there. I would still like to see it detailed in the paper (or at least in the Appendix). - The definition of the three action spaces is hard to follow. Maybe a table would be clearer if space allows it. - Figure 5(b)'s caption mentions 209 actions whereas in the text (Section 5.2) it says 215 actions. - It is unclear if the step penalty was used for the Open Zork domain? I would assume it isn't to make the results comparable to previous work. I would briefly mention it.  EDIT: In light of the authors' answers, I'm raising my rating.


Glow: Generative Flow with Invertible 1×1 Convolutions
Flow-based generative models (Dinh et al., 2014) are conceptually attractive due to tractability of the exact log-likelihood, tractability of exact latent-variable inference, and parallelizability of both training and synthesis. In this paper we propose Glow, a simple type of generative flow using an invertible 1 × 1 convolution. Using our method we demonstrate a significant improvement in log-likelihood on standard benchmarks. Perhaps most strikingly, we demonstrate that a flow-based generative model optimized towards the plain log-likelihood objective is capable of efficient realistic-looking synthesis and manipulation of large images. The code for our model is available at https://github.com/openai/glow.

This paper proposes some modifications to Real NVP (Dinh et al.) and claim improved generative modeling performance. Quantitative evaluations (likelihood scores) are used to compare with Dinh et al. on CelebA and Imagenet 32x32 and Imagenet 64x64.  The proposed improvements are: 1. Using a learnt invertible channel transformation instead of the fixed shuffling used by Dinh et al. 2. Don't use BatchNorm. Replace it with an affine layer whose parameters are initialized to maintain zero-mean unit-variance. This is called ActNorm in the paper. 3. Generalize Dinh's additive coupling layer into an affine coupling layer. 4. The images are compressed into a 5-bit color space instead of the 8-bit color space typically used.  The paper attempts to solve a significant problem (non-autoregressive likelihood-based deep generative models). However, the paper has a number of problems which makes it hard to accept as a NIPS paper.  In short, the quantitative and qualitative evaluation fall short of convincing an informed reader about the usefulness of the idea in the paper. The proposed changes might be useful, but as it stands now, this paper doesn't give any evidence. Let me explain in more detail:  ## Samples   It's not clear how to judge the qualitative samples. The paper only present one set of samples for the same model (the proposed GLOW model) and the same dataset (CelebA 5-bit). To my knowledge, Dinh et al. don't use 5-bit color space and this paper does not provide any samples of any baseline models on the 5-bit color space. So it's simply impossible to properly judge the samples. The paper also doesn't present any nearest-neighbour analysis (in pixel-space or feature-space) to judge overfitting.   ## Compute  It's not clear whether the contributions are due to the problem-level choices made (such as 5-bit color), solution-level choices (1x1 invertible conv, etc.) or simply more compute used for more iterations. The paper doesn't mention any details about how much compute, gpus or iterations were used in comparison to the Dinh et al. baseline. The authors mention multi-GPU experiments which hints that a relatively large amount of compute was used, and thus doesn't give me confidence that the improvement is due to the solution-level changes.  ## Missing Results  The paper presents quantitative results for Imagenet datasets, and 5-bit CelebA, but does not provide samples for the same. This indicates the model did not fare well in terms of sample quality in these other harder datasets, which in itself is okay, but skipping out in providing the results is an attempt at misleading readers. Since sample quality is an important claim of the paper, and failing to provide the same for the other datasets (that were trained and evaluated quantitatively) is conspicuous.  ## Missing Ablations  This is a minor point. The paper makes no attempt at ablating the individual changes proposed. This could have been done on MNIST if compute was an issue.  ## Missing comparison to autoregressive models  This is a minor point. The paper fails to include likelihood scores for autoregressive models in the table of results. While this doesn't matter for informed readers, it misleads readers who are not tracking this literature into thinking that Glow is a state-of-the-art generative model.  # Suggestions for Improvement  1. Compare against the main baseline models (Real NVP) using the same amount of compute. 2. If 1 is not possible, then add a note in the table of results comparing the amount of compute resources used by the two models. 3. Include and discuss samples of the model which were trained on datasets other than CelebA 5-bit. 4. Include nearest-neighbour analysis to judge overfitting.  # Summary  The paper presents some intuitive modifications to Dinh et al's real NVP model. However, the evaluations fall short of supporting any of the claims made in the paper.  ---- # After Author Reponse  I had missed figure 3 in my previous read. Figure 3 does provide evidence that the 1x1 convolution improves NLL. I assume this is a controlled setting, i.e. all other variables are the same. Perhaps it would have been useful to mention this in the paper. Since the baseline result is same as the one in the Real NVP paper, it's easy to conclude that the baseline results are picked from the Real NVP paper, and not reproduced in a controlled setting. I will update my review, since this part of the review was based on an oversight.  The other parts of my review however still stand. It will be more informative both to reviewers and to readers if the authors also include samples from the controlled baselines (i.e. the reverse and shuffle baselines). I'm sure the authors are well-aware that NLL is not the best metric to compare across different models. Fairly trivial changes in modeling assumptions can hurt or improve NLL scores with pretty much no change in subjective quality of samples. Since we are in the business of evaluating samples subjectively, let's do it somewhat more objectively, by including samples from the baseline. One of the central points in my review was that the authors to change the problem itself by using a 5-bit color space. All the samples shown in the paper are from the 5-bit model. So, while I can judge that the samples "subjectively realistic-looking", it's not clear whether this is due to the choice to shift from 8-bit to 5-bit or the improved modeling. This point remains unaddressed and un-acknowledged by the authors.  In summary, I acknowledge that I had missed figure 3's significance in my previous read, and that does improve my judgement of the paper, but the paper still lacks sufficient rigor in evaluation to be accepted as a NIPS paper.


Mirrored Langevin Dynamics
We consider the problem of sampling from constrained distributions, which has posed significant challenges to both non-asymptotic analysis and algorithmic design. We propose a unified framework, which is inspired by the classical mirror descent, to derive novel first-order sampling schemes. We prove that, for a general target distribution with strongly convex potential, our framework implies the existence of a first-order algorithm achieving O( −2 d) convergence, suggesting that the state-of-the-artÕ( −6 d 5 ) can be vastly improved. With the important Latent Dirichlet Allocation (LDA) application in mind, we specialize our algorithm to sample from Dirichlet posteriors, and derive the first non-asymptoticÕ( −2 d 2 ) rate for first-order sampling. We further extend our framework to the mini-batch setting and prove convergence rates when only stochastic gradients are available. Finally, we report promising experimental results for LDA on real datasets.

This is a very well-written paper and excellently presented with some interesting supporting theoretical results. The paper introduces a method (mirror map) from the optimization literature, mirrored descent, to perform scalable Monte Carlo sampling in a constrained state space. The mirror map works by transforming the sampling problem onto an unconstrained space, where stochastic gradient Markov chain Monte Carlo (MCMC) algorithms, in particular, stochastic gradient Langevin dynamics, can be readily applied. The Fenchal dual of the transformation function is used to transform the samples from the unconstrained space back onto the constrained space.   In the paper, the authors state that a "good" mirror map is required. It isn't quite clear what this means. It would seem from the main paper that the mirror map function, h, should be twice continuously differentiable. However, this seems to conflict with the results in the appendix where h \in C^5 and not h \in C^2. Could the authors please clarify?  In Section 3.2.2. (simplex section), a mirror map is given for the simplex example, however, it isn't clear how this is derived, or in fact, how one would create their own mirror in a different setting, e.g. sampling on the positive real line. Is there a general recipe that can be applied? How do you know that the chosen mirror map is optimal, assuming that there exists a wider class of mirror maps?  My main criticism of the paper is related to the simulation study, which is quite limited. It's a little strange that only two Dirichlet examples are given even though the authors could apply their methodology to other constrained spaces. Is there a reason why only the Dirichlet was considered?   Figure 1 only plots the results for the first dimension. Can we assume that the results are the same for all dimensions? This is an important point as it isn't completely clear whether or not the proposed approach works better than Patterson & Teh method (SGRLD) when sampling near to the boundary. It also appears from the simulation study that the improvement offered by the mirror map is only minor compared to SGRLD, could the authors comment on this further? Perhaps explain whether or not there are scenarios where SGRLD would be better.  Overall, this is a nice contribution to the stochastic gradient MCMC literature with strong supporting theoretical results. A particularly nice outcome of this paper is related to the simplex example, where the mirror map approach transforms the sampling problem from being non-log-concave to strictly log-concave.


Hardware Conditioned Policies for Multi-Robot Transfer Learning
Deep reinforcement learning could be used to learn dexterous robotic policies but it is challenging to transfer them to new robots with vastly different hardware properties. It is also prohibitively expensive to learn a new policy from scratch for each robot hardware due to the high sample complexity of modern state-ofthe-art algorithms. We propose a novel approach called Hardware Conditioned Policies where we train a universal policy conditioned on a vector representation of robot hardware. We considered robots in simulation with varied dynamics, kinematic structure, kinematic lengths and degrees-of-freedom. First, we use the kinematic structure directly as the hardware encoding and show great zero-shot transfer to completely novel robots not seen during training. For robots with lower zero-shot success rate, we also demonstrate that fine-tuning the policy network is significantly more sample-efficient than training a model from scratch. In tasks where knowing the agent dynamics is important for success, we learn an embedding for robot hardware and show that policies conditioned on the encoding of hardware tend to generalize and transfer well. Videos of experiments are available at: https://sites.google.com/view/robot-transfer-hcp.

Disclaimer: my background is in control theory and only recently I have invested most of time in reading and doing research in the area of machine learning and reinforcement learning with specific focus on robotics and control. I went through the submitted paper carefully, including the supplementary material. Therefore I am quite confident with my assessment, especially since the problem that the addressed problem is well inside my core expertise (adaptive control). As I previously said, I am very confident with the problem, less confident with the theoretical framework (reinforcement learning) used to solve it. The math presented in the paper is relatively shallow and carefully checked. I am quite confident with the used reinforcement learning tools (PPO, DDPG, DDPG+HER).   = MAJOR COMMENTS =  1. The problem addressed in the paper is interesting. The very interesting part of the paper is the implicit encoding which proposes a solution to condition the control policy on quantities that are not directly measurable (e.g. dynamic properties). In the area of robotics, a similar problem has been solved with adaptive control (Slotine 1988, “Adaptive manipulator control: A case study”) where dynamic parameters (link masses, inertias and centers of mass, joint damping and friction) are estimated on-line using their linearity. The mathematical details of the proposed implicit encoding are confined in a relatively short subsection (Section 4.2 "Implicit encoding"). This section should be expanded and presented in depth. Details on how the cost function $L$ is constructed should be added so that the reader can appreciate what is novel with respect to (Slotine 1988).  - The paper would be very difficult to replicate due to the lack of details. Here is a list of things that should be better described/detailed in order to improve the paper repeatability. Page 3. Section 4 (first paragraph). What is the structure of the policy network? What is the state $s_t$ (e.g. joint positions and velocities)?  Page 4. Section 4.2. What is the cost function $L$? What is the vector of parameters $\theta$? This is fundamental to understand how $v_h$ is estimated from the available measurements ($\theta$?).   = MINOR COMMENTS = Appendix B, Table 2. How is the length of a link measured (e.g. how are these parameters related to the Denavit-Hartenberg parameters)? What is the meaning of the armature of a joint?   Page 2, Section 2 "Related work". A section on adaptive control for robotics should be added. A source of references can be found in (Slotine 1988) and in the book "Applied nonlinear control" by Jean-Jacques Slotine 1991.  Page 3, Section 4.1 "Explicit encoding". The adopted explicit encoding is quite subjective and information-dense. It is subjective in the sense that the position of the local coordinate systems is totally arbitrary and the proposed results might depend on the specific choice of these coordinate systems. It is information-dense because it includes all the coordinate systems across the kinematic chain. It would be good to test how the proposed approach behaves when reducing this information, possibly reducing it to the end-effector pose only.


Learning Disentangled Joint Continuous and Discrete Representations
We present a framework for learning disentangled and interpretable jointly continuous and discrete representations in an unsupervised manner. By augmenting the continuous latent distribution of variational autoencoders with a relaxed discrete distribution and controlling the amount of information encoded in each latent unit, we show how continuous and categorical factors of variation can be discovered automatically from data. Experiments show that the framework disentangles continuous and discrete generative factors on various datasets and outperforms current disentangling methods when a discrete generative factor is prominent.

This paper presents a new framework (JointVAE) for learning disentangled representations.The particular problem that this paper focuses on is the  separating discrete and continuous factors of variation in a fully unsupervised manner. Most existing approaches for learning disentangled representations a prior in the form of a Gaussian with diagonal covariance, which is not suitable for capturing categorical features. In the proposed approach a Gaussian distribution represents the continuous features (such as slant and thickness in MNIST) and a Gumbel-Softmax  represents categorical features such (as digit-type). Inducing disentanglement is done by penalizing the KL term between the inference distribution and the prior in a controlled capacity manner as proposed by Burgess et al. (2017) which the upper bound on the mutual information between ‘x’ and ‘z’ gradually increases during training. The results shows successful disentanglement on various datasets including MNIST, Fashion-MNIST, and CelebA.   This is a well-written paper. The authors do a good job of analyzing both the strengths and weaknesses of their proposed method for disentangling discrete features in different datasets. I think that the main of the paper lies in the relatively thorough experimentation. I thought the results in Figure 6 were particularly interesting in that they suggest that there is an ordering in features in terms of mutual information between data and latent variable (for which the KL is an upper bound), where higher mutual information features appear first as the capacity is increased. I also appreciate the explicit discussion of the robust of the degree of disentanglement across restarts, as well as the sensitivity to hyperparameters. Given the difficulties observed in Figure 4 in distinguishing between similar digits (such as 5s and 8s), it would be interesting to see results for this method on a dataset like dSprites, where the shapes are very similar in pixel space. The inferred chair rotations in Figure 7 are also a nice illustration of the ability of the method to generalize to the test set.   The main thing that this paper lacks is a more quantitative evaluation. A number of recent papers have proposed metrics for evaluating disentangled representations. In addition the metrics proposed by Kim & Mnih (2018) and Chen et al. (2018), the work by Eastwood & Williams (2017) [1] is relevant in this context. All of these metrics presume that we have access to labels for true latent factors, which is not the case for any of the datasets considered in the experimentation. However, it would probably be worth evaluating one or more of these metrics on  a dataset such as dSprites.   A minor criticism is that details the training procedure and network architectures are somewhat scarce in the main text. It would be helpful to briefly describe the architectures and training setup in a bit more detail, and explicitly call out the relevant sections of the supplementary material. In particular, it would be good to list key parameters such as γ and the schedule for the capacities Cz and Cc, e.g., the figure captions. In Figure 6a, please mark the 25k iterations (e.g. with a vertical dashed line) to indicate that this is where the capacity is no longer increased further.   Questions  - How robust is the ordering on features  Figure 6, given the noted variability across restarts in Section 4.3? I would hypothesize that the discrete variable always emerges first (given that this variable is in some sense given a “greater” capacity than individual dimensions in the continuous variables). Is the ordering on the continuous variables always the same? What happens when you keep increasing the capacity beyond 25k iterations. Does the network eventually use all of the dimensions of the latent variables?   - I would also appreciate some discussion of how the hyperparameters in the objective were chosen. In particular, one could imagine that the relative magnitude of Cc and Cz would matter, as well as γ. This means that there are more parameter to tune than in, e.g., a vanilla β-VAE. Can the authors comment on how they chose the reported values, and perhaps discuss the sensitivity to these particular hyperparameters in more detail?  - In Figure 2, what is the range of values over which traversal is performed?  Related Work  In addition to the work by Eastwood & Williams, there are a couple of related references that the authors should probably cite:   - Kumar et. al [2] also proposed the total correlation term along with Kim & Mnih (2018) and Chen et al. (2018).   - A recent paper by Esmaeli et al. [3] employs an objective based on the Total Correlation, related to the one in Kim & Mnih (2018) and Chen et. al (2018) to induce disentangled representations that can incorporate both discrete and continuous variables.   Minor Comments  - As the authors write in the introduction, one of the purported advantages of VAEs over GANs is stability of  training. However, as mentioned by the author, including multiple variables of different types also makes the representation unstable. Given this observation, maybe it is worth qualifying these statements in the introduction.   - I would say that section 3.2 can be eliminated - I think that at this point readers can be presumed to know about the Gumbel-Softmax/Concrete distribution.  - Figure 1 could be optimized to use less whitespace.  - I would recommend to replace instances of (\citet{key}) with \citep{key}.  References   [1] Eastwood, C. & Williams, C. K. I. A Framework for the Quantitative Evaluation of Disentangled Representations. (2018).  [2] Kumar, A., Sattigeri, P. & Balakrishnan, A. Variational inference of disentangled latent concepts from unlabeled observations. arXiv preprint arXiv:1711.00848 (2017).  [3] Esmaeili, B. et al. Structured Disentangled Representations. arXiv:1804.02086 [cs, stat] (2018).


Multi-Layered Gradient Boosting Decision Trees
Multi-layered distributed representation is believed to be the key ingredient of deep neural networks especially in cognitive tasks like computer vision. While non-differentiable models such as gradient boosting decision trees (GBDTs) are still the dominant methods for modeling discrete or tabular data, they are hard to incorporate with such representation learning ability. In this work, we propose the multi-layered GBDT forest (mGBDTs), with an explicit emphasis on exploring the ability to learn hierarchical distributed representations by stacking several layers of regression GBDTs as its building block. The model can be jointly trained by a variant of target propagation across layers, without the need to derive backpropagation nor differentiability. Experiments confirmed the effectiveness of the model in terms of performance and representation learning ability.

Short overview: Authors propose to build a neural network using gradient boosted trees as components in the layers. To train such a structure, since the gbdts are not able to propagate the gradient, they propose to use a method inspired by the target propagation: each gradient boosted trees is built to approximate a gradient of loss of prediction function and a pseudo target, with respect to the prediction function. Pseudo targets are updated at each iteration using the reverse mapping of the built tree representation and the pseudo label of the next layer. The reverse mapping can be found using the reconstruction loss. At each iteration, each layer's ensemble grows by one boosting tree Authors hint at potential applications of blocking adversarial attacks, that rely on estimating the gradients of the final loss with respect to input, which would not work for layers that can't propagate the gradients, however this direction is not explored in this paper  Detailed comments: Overall, an interesting idea of co-training gbdts with nns. The idea is not new, but discarding backprop and using target propagation, that allows gbdts to do what they are best for - e.g. build trees to simulate gradients - is neat and witty.   Overall, well written and well structured, but I do have some points of confusion after reading it:  a) Pseudo labels are all computed from top to bottom - i am not sure where you think is the top and where the bottom is. is it to say that the first pseudo label is from the last (output layer) and then you go backward to the input layer. If yes, this top to bottom needs to be rephrased. Some people draw neural nets horizontally actually... (you included, figure 1!), but even if you draw vertically, the output is the bottom, and you go from the bottom to the top  b) What is the structure of your pseudo-inverse function G_i^t? Is it essentially a matrix of dimension R^d_i x R^d_(i-1)?  In your algorithm, I am not sure why you fit the regression tree and update the inverse mapping interchangeably for a number of iterations? Why not to first minimize the loss (using the tree from the previous iterations) to obtain the inverse mapping (so for a number of iterations, do a gradient descent update using the reconstruction loss) and then when it is settled, build a tree using the inverse loss and found inverse mapping, which can be used to get pseudo label at this point. (i am referring to the for loop k=1..K). Instead to update the G you make a pseudo gradient descent step using the built tree, which does not seem very intuitive to me. Plus it also seems that you discard all the trees built at this step (you use them just to update G)  c )As far as i remember, target propagation works slower than backprop, I am not sure if you observed the same   My main concern is the experiments and their reproducibility: the way they are described, it is not clear what the actual setup was used for their method, and if the comparison is fair (were all the competing models tuned?). I am having hard time believing that well tuned 2-hidden layer nn would underperform so badly. What do you mean for a setup of 5-3 for instance? AT first i thought those were the number of neurons in nn, but on the second thought, i think you mean 3 is the dimension of the first tree layer, 5 of the second. But how many trees are in each layer? This really needs to be clarified.  Why accuracy and reconstruction loss of is not reported for the synthetic datasets. As well as metrics of at least one competing baselines, like just a nn? How are those 128-128 setups are chosen for income prediction? i think the fact that nns are trained with exactly the same structure (128-128 for a hidden layer) means you have a huge overparameterization, that's why they don't do well. The structure of the net should be just tuned to provide a fair point of comparison. The same as hyperparameters for xgboost (eg learning rate of 0.3 is default and is large, income is a simple dataset, and i expect that you just overfit at this point)   Additionally, I not sure what the exact target application is? The adversarial learning is interesting, but it was not explored, and apart form this, i am struggling to imagine where an ensemble per layer should be more efficient than a tuned hidden layer.  Suggestions: to facilitate the reading, it would be nice to add a short section on target propagation, since it is a backbone of your method, to facilitate the reading. You seem to have enough space for that  Nits: line 185: the experiments ... is->are line 187: where did CNNs came from? i think even ur synthetic experiments dont use the convolutional layer (but i am not sure). Income prediction surely does not Table 1: are the diffs significant on 16k test dataset for income? line 270: interesting to integrate line 274: does not make  I read authors  rebuttal and I they addressed the majority of questions I had


The Limits of Post-Selection Generalization
While statistics and machine learning offers numerous methods for ensuring generalization, these methods often fail in the presence of post selection-the common practice in which the choice of analysis depends on previous interactions with the same dataset. A recent line of work has introduced powerful, general purpose algorithms that ensure a property called post hoc generalization (Cummings et al., COLT'16), which says that no person when given the output of the algorithm should be able to find any statistic for which the data differs significantly from the population it came from. In this work we show several limitations on the power of algorithms satisfying post hoc generalization. First, we show a tight lower bound on the error of any algorithm that satisfies post hoc generalization and answers adaptively chosen statistical queries, showing a strong barrier to progress in post selection data analysis. Second, we show that post hoc generalization is not closed under composition, despite many examples of such algorithms exhibiting strong composition properties.

This paper demonstrates some of the limits of mechanisms satisfying post hoc generalization, which guarantees accuracy in the adaptive data analysis setting.  They show both an information-theoretic and computational lower bounds for sample complexity, and then show that mechanisms satisfying post hoc generalization do not in the worst case compose non-trivially.  These results tie up some loose ends in adaptive data analysis, e.g. closing the previously-open gap in sample complexity for statistical queries, and gives a more complete picture of what is possible in this area.  On the other hand, I would not call this the most clearly written paper.  In particular, there is very little discussion or overview of the proofs.  I recognize how difficult this is to do given the page limit constraints, but since a fair amount of the proofs are put into the appendix anyway, I consider it very important to communicate the ideas contained within the proofs rather than highlighting the technical details, as is done here.  Nor does this paper highlight the difference between their strategy and what is contained in the previous literature.  The basic strategy, after all, remains the same as in the lower bounds established in previous literature:  Algorithm 2 attempts to uncover the sample given to M, then asks a query that M will overfit on, plus using fingerprinting, etc.  What exactly is the new idea here?  Also, to be blunt, why is the result that post hoc generalization does not compose interesting?  After all, we know that we can achieve O(\sqrt{k}/\epsilon^2) sample complexity using differential privacy, and to achieve a better sample complexity, this paper shows that we can't guarantee post hoc generalization at all.  It seems that when it comes to statistical queries, post hoc generalization is a bit besides the point, no?  A few very small things: - Would be helpful to be much more explicit about sample complexity for uniform convergence, that way the reader can compare this to your results. (line 48) - Algorithm 2:  A_1 should start empty: no one has been accused yet. - Algorithm 2:  \tilde{q}_i^j presumably needs to lose the tilde.  Edit (post author response):  On the new idea in your results:  I appreciate your attempt to stress the idea of an approximate attack rather than an exact attack.  This seems like you're pointing out that a (very similar) attack, even if it doesn't work as well, still works well enough, which may or may not be of sufficient interest/novelty for your readers. On the interesting-ness of composition:  AFAIK, there's no good way to directly show that an algorithm satisfies post hoc generalization without also satisfying DP, information bounds, or the like.  This means it seems rather unlikely that you would actually need composition without already having stronger assumptions on the algorithms that you would like to compose, even if that assumption isn't DP.  Which in turn means I don't know where this result would likely be relevant.


Efficient online algorithms for fast-rate regret bounds under sparsity
We consider the problem of online convex optimization in two different settings: arbitrary and i.i.d. sequence of convex loss functions. In both settings, we provide efficient algorithms whose cumulative excess risks are controlled with fast-rate sparse bounds. First, the excess risks bounds depend on the sparsity of the objective rather than on the dimension of the parameters space. Second, their rates are faster than the slow-rate 1 / p T under additional convexity assumptions on the loss functions. In the adversarial setting, we develop an algorithm BOA+ whose cumulative excess risks is controlled by several bounds with different trade-offs between sparsity and rate for strongly convex loss functions. In the i.i.d. setting under the Łojasiewicz's assumption, we establish new risk bounds that are sparse with a rate adaptive to the convexity of the risk (ranging from a rate 1 / p T for general convex risk to 1 /T for strongly convex risk). These results generalize previous works on sparse online learning under weak assumptions on the risk.

# Update after reading the author response  Thank you for the explanations. I have improved my score. I hope the following points help the authors see why I am still not enthusiastic about acceptance.  1- I still think the comparison to previous work is inadequate. For example, the SAEW algorithm of [12] uses several similar techniques (like BOA plus sequential restarting), and achieves a bound (in a less general setting) that scales with $d_0$ rather than $d_0^2$. (it was only after discussion with other reviewers that I realized your sentence in the Conclusion - "These rates are deteriorated compared with the optimal one that require restrictive assumption." - is perhaps referring to this difference). How do your techniques differ? How do your running times differ? I had suggested a table summarizing the previous work and where your work stands w.r.t. them in terms of assumptions, rates, and complexity, but from the author response, it feels like the authors are mainly looking into only moving the contributions earlier in the intro.  2- I still believe it is hard for the reader to see the general view of your paper. All the relevant points are there, but the context is missing or is lost between seemingly smaller improvements here and there (e.g. relaxing sc to exp-concave or Lojasiewic). It would have been much nicer if the intro started by saying why we need quantile bounds (instead of getting to it only in L138), what others had done (e.g. [18,31,22]), and what specifically was missing in previous work (like the efficient adversarial alg). The reader of course knows that you want to "obtain lower bounds than any existing online algorithm in sparse setting", but they first need to see an overview of what existed and what was missing to be able to understand in which direction you will be extending the previous work. A good example of such an introduction is maybe that of [18], among many others. Once this is established, then the contribution section could highlight your most important ideas (more like expanding on the list you have at the end of your introduction section, as opposed to attending to the details of the Lojasiewic's assumption). In this way, it is also easier for the reader to understand (at a high level) your improvements over previous work, because you have already talked about their limitations.   3- I think the 8-page limit should not result in the paper not being self-contained. Relevant results and extra comments could be recalled / put in the appendix, and short explanations could be given to connect things.  I hope that the authors consider these points in the final version if the paper is accepted, or in the subsequent submissions if it is not.  # Summary  The paper seems to provide new projection-free algorithms for online learning over the l1-ball with guarantees that depend on the sparsity of the optimal parameter / competitor. This is what *seems* to be the main contributions of the paper: 1- An algorithm (Alg-1) for the expert-advice setting that enjoys (per Thm 2.1) a quantile bound, which scales with O(1/T) when the losses are exp-concave, improving / extending previous work [18, 31, 22]. 2- A projection-free algorithm (BOA+) for online strongly-convex optimization over the l1 ball, by restarting Alg-1 on grids that include new leaders (in the Follow-The-Leader sense) from the previous rounds. 3- Another algorithm (SABOA) for stochastic strongly-convex optimization over the l1 ball, similar to BOA+ but with a different grid construction. 4- O(1/T) rates for BOA+ and SABOA that depend on the number of non-zero elements of the competitor / optimal parameter.  However, the paper is generally hard to read, in the sense that it is not self-contained (at all), is not coherently written, comparison to previous work is scattered throughout the paper, and and the overall message is hidden by the details. In particular, the high-level view of the paper, the extent of technical novelty, the exact relation to previous work, and if / why the bounds are optimal in each of the problem parameters (i.e., sparsity of the competitor, Lipschitz and exp-concavity constants, etc) is not entirely clear.  As a result of the inaccessibility of the paper, I do not recommend publishing at NIPS.  ## Questions for the authors Please consider answering Q1-Q5 below in the author response. (Please consider answers that shed light on the general view and significance of your work).  # Quality and Significance  From a technical point of view, the contributions seem to be interesting. However, the quality of the presentation suffers, to the point of making the results inaccessible, except to the reader with direct experience working with the underlying algorithms / ideas.  ## Algorithm 1 Alg-1 is the Squint algorithm of [18] with a geometric prior on learning rates. The result seems to improve over [31, Section 4] by requiring only exp-concavity of the losses (in expectation) as opposed to strong convexity. In addition, the KL-div is not simplified to log(K) - as [31] does in their Thm 4.3 for example, so quantile bounds are obtained. It extends [18] (and the relevant prior work that obtain O(1/sqrt(T)) quantile bounds, not cited here) by providing the O(1/T) rate for exp-concave losses. It seems to improve over [22] by being a completely online algorithm and removing a gap term in the bound.  **Q1**: Where similar priors used for Squint before? Compared to [31], are there any other technical novelties I have not mentioned above? What is the challenge of replacing strong convexity with weak exp-concavity (wouldn’t very similar techniques as in [31, Sec 4] directly apply here?)  **Q2**: Why is the weakening of exp-concavity important? Can you provide concrete examples of useful losses where the weak exp-concavity holds, but exp-concavity doesn’t (i.e., what are the “regularity conditions”?)  ## BOA+ and SABOA BOA+ and SABOA extend Alg-1 to the l1 ball (Alg-1 is for a finite set of experts) by restarting it, for exponentially increasing number of time steps, on descritized grids that gradually include improved approximations of the optimal parameter. The algorithms are interesting as they are projection-free, yet obtain the O(1/T) rates of convergence under strongly-convex (or Lojasiewicz) losses.  Again, the lack of coherence and proper comparison with previous work prevents complete appreciation of the results. It seems that the main benefit over previous work are that the algorithms are projection-free, and the Lojasiewicz condition is used, generalizing strong-convexity to a class of non-convex functions. Also, the same bounds apparently apply to general convex functions as well (reducing to O(1/sqrt(T)) in that case).  **Q3**: How do your results (Sec 3) benefit from the exp-concavity condition, as you still need to assume strong-convexity, or it’s non-convex counter-part (Lojasiewicz’ condition)? Would they also benefit from the weakening of exp-concavity in A2?  **Q4**: Regarding [26] and [12], isn’t competing with the minimizer over R^d harder than with the minimizer of B_1, as the former always has a lower loss than the latter, and thus the bounds will be stronger?  **Q5**: How do the computational complexity of BOA+ / SABOA compare with previous work? Are [13] and [32] the only relevant work here? How do the bounds compare? Is the dependence on each of the problem parameters (|| Theta ||_0, G, alpha, beta, mu, etc) optimal? What / where are the relevant lower bounds?  # Clarity  I found the paper hard to read, even for someone with experience in online learning, the expert advice framework, bandits, etc. The reader needs deep familiarity with specific previous work (Squint, BOA, etc). While the relevant sections are there (contributions, previous work, etc.), one has to read the paper multiple times, as well as the related work, to find out what the actual contributions are, why they are important, how they compare with previous work, etc. A few examples are provided below:  1- I recommend consulting a professional editing service to improve general readability. There are several sentences that are not informative, or are confusing. It is not clear what some pronouns refer to. For example: “These results generalize previous work [..]. **They** are obtained under [..] Lojasiewicz condition.” Which results? All of them? But this seemed to relate to convex / exp-concave functions. One has to read until the end to find out which results is given under what condition.  2- The introduction should discuss what is the goal of the paper, what are the high-level results, and why they matter and were not obtained before. Instead, it gets to details such as using Dirac masses to get the regret, and goes to “Previous work”. What do you want to do? Why?  3- The previous work section seems incomplete, uninformative, and vague. What is “a subroutine centered around the current estimate”? Why should we know this? We still don’t know what the paper is trying to do… Shouldn’t Lines 41,42 actually just go to the “contributions” section?  4- Similar vague sentences and paragraphs appear throughout the paper. The paper is not self-contained (e.g., “extended to a unique approximately sparse Theta* [ ..] see [1;3]”). Hand-wavy explanations and phrases are used where appropriate mathematical expressions should appear (e.g., “set of hard-truncated and dilated soft-thresholded versions of [..] as in (43)”). On the other hand, explanation is absent where it would be useful. For example, why should one consider SABOA instead of BOA+ in the stochastic setting? Why can we use previous iterates instead of the leaders here?  # Originality  The comparison with related work is unclear. The reader needs to go to the related work and find the results and compare them with the current results (even, e.g., the respective Theorem numbers are not given).  I recommend providing a table comparing previous work (e.g., as in [22]), showing whether each result concerned regret or cumulative risk [as in 22], strongly-convex, exp-concave, or weak exp-concave losses, was it an h.p. or expectation bound, etc.


From Stochastic Planning to Marginal MAP
It is well known that the problems of stochastic planning and probabilistic inference are closely related. This paper makes two contributions in this context. The first is to provide an analysis of the recently developed SOGBOFA heuristic planning algorithm that was shown to be effective for problems with large factored state and action spaces. It is shown that SOGBOFA can be seen as a specialized inference algorithm that computes its solutions through a combination of a symbolic variant of belief propagation and gradient ascent. The second contribution is a new solver for Marginal MAP (MMAP) inference. We introduce a new reduction from MMAP to maximum expected utility problems which are suitable for the symbolic computation in SOGBOFA. This yields a novel algebraic gradient-based solver (AGS) for MMAP. An experimental evaluation illustrates the potential of AGS in solving difficult MMAP problems.

Main ideas ========== The paper develops the relation between solving an MDP and performing inference in a Bayesian network. The direction, however, is novel as far as I can tell: using MDP algorithms to solve an inference problem.  The first part shows that an existing MDP algorithm (ARollout)  is in fact performing a BP iteration over the DBN that represents the MDP.  In the second part, a different MDP algorithm (SOGBOFA) is used to solve a particular inference problem of choosing a subset of values with the maximal marginals (MMAP). The resulting SOGBOFA-based solver often loses to the state-of-the-art, but for harder cases it can outperform the state of the art.    The idea is interesting and the results are useful, mainly in expanding the understanding of the relation between planning and inference.   Regarding quality, I was not able to verify all details of the proofs and algorithms, but as far as I could verify the paper is technically sound.   Strength ======== Originality: I am not aware of work on using planning to solve inference problem, and on mapping an MDP solver to BP.   Significance: I like the mapping proposed by the authors, and I wonder if it can be extended to other planning algorithms, i.e., using them to solve MMAP as well.    Weaknesses ========== Clarity: the paper was very hard for me to follow. I mention some of the reasons below, but in general, more background is needed. However, I admit that this work is somewhat outside my area of expertise, and so perhaps this background is not needed for someone that is more in the field.   Examples of clarity issues: Line 91: "... to estimate values ..." - which values? Line 97: "c_{ij}" what is the j representing? Line 130: "... concrete values, ... use the expressions ..." - which values? which expressions? Line 140: "... an evaluation of the graph ..." - what do you by this? In general the paragraph starting with line 140 should be elaborated.  Also, the term "automatic differentiation" is used several times but is not explained (and a reference for it is also not provided).  Line 197: "For the purpose of this paper it suffices to consider IDs where decision nodes do not have parents and there is a single value node V." - please better explain: why is it sufficient to consider this? I think that you relax this requirement later in the paper, but if that's the case then say so explicitly, e.g., "We begin by assuming X and then relax it."    Rebuttal question: ================= I find the reduction from MMAP to SOGBOFA confusing, since usually reduction is done between problems. This raises the following question: can one reduce an MMAP problem to an MDP problem? I belive the answer is yes, in which case why should one use specifically SOGBOFA to solve this problem?    Minors: ====== Line 96: you probably want to have "if" not in math mode, to get the spacing correct.  Line 97: "... where p_i is p(x=1) ..."  - do you mean "p_1 is p(x=1)"?


Lipschitz regularity of deep neural networks: analysis and efficient estimation
Deep neural networks are notorious for being sensitive to small well-chosen perturbations, and estimating the regularity of such architectures is of utmost importance for safe and robust practical applications. In this paper, we investigate one of the key characteristics to assess the regularity of such methods: the Lipschitz constant of deep learning architectures. First, we show that, even for two layer neural networks, the exact computation of this quantity is NP-hard and state-ofart methods may significantly overestimate it. Then, we both extend and improve previous estimation methods by providing AutoLip, the first generic algorithm for upper bounding the Lipschitz constant of any automatically differentiable function. We provide a power method algorithm working with automatic differentiation, allowing efficient computations even on large convolutions. Second, for sequential neural networks, we propose an improved algorithm named SeqLip that takes advantage of the linear computation graph to split the computation per pair of consecutive layers. Third we propose heuristics on SeqLip in order to tackle very large networks. Our experiments show that SeqLip can significantly improve on the existing upper bounds. Finally, we provide an implementation of AutoLip in the PyTorch environment that may be used to better estimate the robustness of a given neural network to small perturbations or regularize it using more precise Lipschitz estimations.

The paper studies the problem of estimating the Lipschitz constant of a mapping realized by a trained neural network.  Several distinct contributions are made:   (i) It is shown that exactly computing the Lipschitz constant of a two layer MLP (Multi Layer Preceptron) with ReLU activation is NP-hard (worst-case result).   (ii) A straightforward algorithm for bounding the Lipschitz constant of a computational auto-differentiable graph is given.  The algorithm, named AutoLip, is a generalization of multiplying spectral norms of Jacobians for a sequence of mappings, and hence usually provides pessimistic bounds.   (iii) A technique for efficiently applying power method to a high-dimensional affine transformation is given.  This technique assumes efficient implementation of the mapping in an auto-differentiable graph.  It allows estimating the leading singular value of the transformation, which is equal to the Lipschitz constant.   (iv) For the case of MLP, an algorithm named SeqLip is presented, which is less naive, and tighter than AutoLip.  The degree to which it is tighter is theoretically analyzed under simplifying assumptions. The paper concludes with experiments comparing different estimations of the Lipschitz constant for several trained networks.  In my opinion, the problem studied in this paper is important, not only for robustness to input perturbations (motivation given in the text), but also for improving recent generalization bounds that depend on Lipschitz constant.  I advise the authors to mention this latter point in the text.    In terms of quality, the paper is well written, and its contributions are interesting.  I did not verify the proofs in supplementary material, but the results seem very reasonable and I believe they are correct.  One point I find troubling is the connection, or lack thereof, to prior work -- absolutely nothing is said about existing literature.  I kindly ask the authors to include such reference in their rebuttal, and subsequently add it to the manuscript.  To my knowledge contributions (i)-(iv) above are novel.  Assuming this is the case, I recommend accepting the paper.  Points which I believe can improve this work:   * The text covers multiple ideas, many of which are in my opinion distinct, and vary in their importance.  Of contributions (i)-(iv) above, I think (iv) is the most interesting one, and (iii), which is also interesting, can be presented as part of (iv).   (i) and (ii) are less interesting (the first because it is a worst case analysis; the second because it is trivial).  There is also some text beyond (i)-(iv) which I think adds almost nothing -- the discussion on Lipschitz constants of activation, pooling, dropout and batch norm layers.  I believe the text would benefit from being more focused on the important parts.   * It seems that the improvement of SeqLip over AutoLip is not that significant (one order of magnitude, while AutoLip is likely overpessimistic by much more than that).  I think further refinement of SeqLip would be of great interest, and encourage the authors to pursue this goal, either in this work or in a follow-up.   * The experiments are very minimal, which is a shame because they are insightful.  I would add many more models to the evaluation, placing the results in supplementary material if needed.   * I think there are some typos in the text (for example in Eq 11).  This is not critical, because the reader can easily reproduce the derivations independently.  ---- Edit following discussion period: I have read the authors' response, and in particular their reference to existing literature.  I have also read the other reviews.  While I agree with the criticism of R3 with regards to the experiments, in my opinion, the contribution of this work is theoretical -- it presents a technique (SeqLip) for bounding the Lipschitz constant of a MLP, going a step beyond the trivial routine of multiplying spectral norms.  My stand on the paper remains -- I recommend its acceptance.