Horizon-Independent Minimax Linear Regression
We consider online linear regression: at each round, an adversary reveals a covariate vector, the learner predicts a real value, the adversary reveals a label, and the learner suffers the squared prediction error. The aim is to minimize the difference between the cumulative loss and that of the linear predictor that is best in hindsight. Previous work demonstrated that the minimax optimal strategy is easy to compute recursively from the end of the game; this requires the entire sequence of covariate vectors in advance. We show that, once provided with a measure of the scale of the problem, we can invert the recursion and play the minimax strategy without knowing the future covariates. Further, we show that this forward recursion remains optimal even against adaptively chosen labels and covariates, provided that the adversary adheres to a set of constraints that prevent misrepresentation of the scale of the problem. This strategy is horizon-independent in that the regret and minimax strategies depend on the size of the constraint set and not on the time-horizon, and hence it incurs no more regret than the optimal strategy that knows in advance the number of rounds of the game. We also provide an interpretation of the minimax algorithm as a follow-the-regularized-leader strategy with a data-dependent regularizer and obtain an explicit expression for the minimax regret.

The problem of online linear regression is considered from an individual sequence perspective, where the aim is to control the square loss predictive regret with respect to the best linear predictor $\theta^\top x_t$ simultaneously for every sequence of covariate vectors $x_t \in R^d$ and outcomes $y_t \in R$ in some constraint set. This is naturally formulated as a sequential game between the forecaster and an adversarial environment.  In previous work [1], this problem was addressed in the "fixed-design" case, where the horizon T and the sequence of covariate vectors $x_1^T$ is known in advance. The exact minimax strategy (MMS) was introduced and shown to be minimax optimal under natural constraint sets on the label sequence (such as ellipse-constrained labels). The MMS strategy consists in some form of least squares, but where the inverse cumulative covariance matrix \Pi_t^{-1} is replaced by a shrunk version P_t that takes future instance into account. The precision matrices P_t are determined by a backwards recursion (Eq (1)), starting from P_T = \Pi_T^{-1}.  This work extends these results to the case when the horizon T and the covariate sequence x_t are unknown a priori, although the latter has to follow some constraint on its cumulative covariance. The key point of the paper is that the backwards recursion can be inverted into a forwards recursion. This enables to play MMS strategy with only the knowledge of the initial precision matrix P_0. The resulting strategy guarantees a regret bound that holds, and is exactly minimax optimal, for the class of all sequences of covariates x_1^T (of any horizon T) for which the backwards MMS strategy would give an initial precision matrix P_0 that coincides with the one chosen by the algorithm (with natural additional constraints on the labels), as shown in Theorem 2. In Theorem 3, it is shown that forward MMS is still minimax when relaxing the above equality on precision matrices into an upper bound, with some necessary additional restrictions on the sequences of covariates and outcomes. Finally, the forward MMS strategy is interpreted as a Follow-the-regularized-leader with a data-dependent regularizer, and Theorem 5 gives an explicit upper bound on minimax regret.  The results in this paper constitute a surprising twist on and a significant extension of the results in [1]. The key fact is the ability to invert the backwards recursion, from which the forward "arbitrary horizon/design" algorithm and its minimax optimality follow from results in [1]; this means the results are somewhat incremental, although interesting since they rely on a non-trivial observation. The paper is well-written and clearly motivated, although it has some typos and minor notational inconsistencies which could be fixed.  I would rate this a clear accept, if it weren't for one issue with the presentation of the results. Specifically, it is claimed several times that the algorithm is minimax simultaneously for all time horizons T, which can seem quite surprising. While this is true, this comes from the fact that in this setting the scale parameter (a discounted cumulative covariance of the covariate sequence) replaces the time horizon T as a measure of complexity/length of the sequence. There is in fact a correspondence between the two quantities, since the game ends precisely when this scale condition is violated. Hence, while the algorithm is minimax for all horizons T, it is only minimax for one scale parameter, and the universality over T merely comes from the fact that T is no longer used to parameterize the class. This is not as strong as saying that the strategy is minimax simultaneously for all class parameters (here, time horizons), which is the case for the NML algorithm in the log-loss case under restrictive conditions (which is mentioned in lines 53-57). Again, this is very reasonable and the opposite would indeed have been surprising (since the paper considers exact minimax regret, a price would be expected for adaptation to the scale/horizon), so my concern is not with the results per se, but rather with the wording and presentation which could make the above distinction more explicit.  As a related remark/question, perhaps the correspondence between \Sigma and T in typical scenarios (e.g., iid sequences) could be somewhat quantified: it would seem that \Sigma scales as (1+log T) C, C being the average/expected covariance matrix.   Minor remarks, typos and suggestions:  - Lines 20 and 25: seemingly a problem on the font for \mathbb{R}, and an extra ";" in the formula l25; - Eq (MMS): s_t is undefined at this stage; - L46 "allow", l51 "motivate"; - L104: it seems V should also depend on Pi_t; also, perhaps it could be expressed explicitly in terms of its arguments (s_t, etc.); - L112-114: the notations for the constraint sets (in particular their arguments) seem to vary a lot in the text (shouldn't B depend on x_t ?); see also line 135; - L158: maybe it could be useful to say how the P_s are defined in this sentence (with forward recursion starting with Sigma_0); - L187: the union should be an intersection; - Related to the aforementioned point, maybe the dependence on the initial precision matrix of forward MMS could be made explicit in the notations for clarity; - There is no conclusion at the end of the paper, likely due to space constraints, hopefully with the additional page in camera-ready some perspectives for future work will be included; - Some missing references in lines 453 and 479.  [1] Peter L. Bartlett, Wouter M. Koolen, Alan Malek, Manfred K. Warmuth, and Eiji Takimoto. Minimax fixed-design linear regression. In Proceedings of The 28th Annual Conference on Learning Theory (COLT), pages 226–239, 2015.   EDIT: I read the authors' response, which addressed the issue on the presentation of the results. The updated formulation is more convincing and accurately describes the content of the results. I have updated my score accordingly.  On another note, since the scale parameter $\Sigma_0$ is hard to interpret as is, I still think it could be interesting to relate it to more familiar parameters like the time horizon T and some sort of average covariance matrix, if only in some restricted case. This could give some sense of when the algorithm should be expected to stop, or when its minimax regret guarantee starts to become "tight", or what the cost of "misspecification" (i.e., anticipating a wrong covariance matrix via the choice of $\Sigma_0$) is.


The Lingering of Gradients: How to Reuse Gradients over Time
Classically, the time complexity of a first-order method is estimated by its number of gradient computations. In this paper, we study a more refined complexity by taking into account the "lingering" of gradients: once a gradient is computed at x k , the additional time to compute gradients at x k+1 , x k+2 , . . . may be reduced. We show how this improves the running time of gradient descent and SVRG. For instance, if the "additional time" scales linearly with respect to the traveled distance, then the "convergence rate" of gradient descent can be improved from 1/T to exp(−T

Summary: -------- The paper studies very lazy updates of stochastic gradients where one uses previous computed gradients instead of computing new ones. They justify such approach by introducing the notion of "lingering gradients", that they define properly. A theoretical approach is provided, assuming this new property, the authors propose a careful storage of previous gradients to exploit this property. Then experiments are made on a simpler method based on SVRG that, while loosing the theoretical properties previously proven, show strong improvements of their approach compared to classical ones.  Quality and clarity: -------------------- The paper is very well written. Motivation is clear and outline well carried on. The proofs are complicated but readable and correct (though I put below details that I''d like to be clarified). Such quality of paper offers the possibility to open discussions in conference.  Originality and significance: ----------------------------- The paper gives theoretical tools to tackle a heuristic that was interesting in practice and where they show real improvements. The storage of gradients using appropriate structure is original to me and may be worth disseminating.  Nevertheless, the original definition of lingering gradients seems quite unrealistic. The natural application of this framework is non-smooth optimization, where in some regions, the gradient indeed doesn't change. However characterizing how these regions smoothly change seems impossible. It seems to me that this lingering gradient is an approximation for very smooth functions (Lipschitz constant very small). -> Could the authors relate the approximate lingering property (up to a given epsilon) to the simple smoothness of the gradient in the experiments they do ? In other words can these Lipschitz constant be computed to understand better the approximation ?  Note that Holder smoothness of the function would also be interesting to relate the the beta exponent of their definition. It seems to me that in such cases it would be interesting to relate the proposed approach to approximate gradient methods that exploit the Holder smoothness see e.g. [Y. Nesterov 2013, Universal Gradient Methods for Convex Optimization Problems] but in a stochastic case.  Conclusion: ----------- Though I think the definition of lingering gradients is for the moment unrealistic, the proposed approach is an interesting step towards heuristics that use previous gradients in a lazy way. Overall the quality of the paper enables to open discussions in the conference that could be fruitful. Since the experiments part is particularly convincing, having the code available would definitely improve the paper (and so my overall appreciation).  Proof check: ------------ l 427: Precise that you used B_{k_i}(k_{t-1}-k_i) \subset B_{k_i}(k_t-k_i) because k_t > k_{t-1} l 461: Could you precise the terms hidden in the O ? l 490, Bounds on T_{time}: Detailing the inequalities would ease the reading. In particular I don't get the last inequality: form the assumptions we have log(m_S) > log(T)  Details: -------- l 117: the LP (2.1) is convex, the proposed formulation is therefore not a reduction to convex optimization but another smoothed formulation of the problem. l 490: Shouldn't it be m_S = \Omega(min(...,...)) without the \Theta ? l 490: In the case beta \in (0,1): erase additional parenthesis in the lower bound on m_S, correct m_s in m_S in the bound of T_{time}


Variational Inference with Tail-adaptive f -Divergence
Variational inference with α-divergences has been widely used in modern probabilistic machine learning. Compared to Kullback-Leibler (KL) divergence, a major advantage of using α-divergences (with positive α values) is their mass-covering property. However, estimating and optimizing α-divergences require to use importance sampling, which may have large or infinite variance due to heavy tails of importance weights. In this paper, we propose a new class of tail-adaptive fdivergences that adaptively change the convex function f with the tail distribution of the importance weights, in a way that theoretically guarantees finite moments, while simultaneously achieving mass-covering properties. We test our method on Bayesian neural networks, and apply it to improve a recent soft actor-critic (SAC) algorithm (Haarnoja et al., 2018) in deep reinforcement learning. Our results show that our approach yields significant advantages compared with existing methods based on classical KL and α-divergences.

# Paper ID 2274  Variational Inference with Tail Adapted f-divergence   - The paper proposes an improved variational inference method, f-divergence method that improves over alpha divergence (and KL divergence). The paper is well written and the problem considered is extremely relevant to NIPS. The ideas are theoretically justified and the approximations made are principled. The experiments are well designed and show good performance compared to alpha divergence.  - The method seeks to improve upon alpha divergence. Specifically, the alpha parameter, kept constant during trianing, may experience problems associated with fat-tailed distributions due to variation in the density ratio. Consequently, adjusting alpha is difficult. A constant alpha is the current state-of-the-art which this algorithm improves upon.  - The main contribution is a theoretically justified method to compute the gradient of the f divergence without knowing f itself. Both variants (score function, reparameterization) are described. After making computational approximations, the resulting algorithm is a stochastic approximation to the gradient of the f-divergence. Computationally, the key simplification to the approximation is using the (non-negative) rank of the density ratio to weight each data point in the sample.  - The experimental section is compelling. The results demonstrate the practical utility of the ideas and approximations. In particular, the method outperforms the previous state-of-the-art (alpha divergence) on the hardest instances. This aligns nicely with the theory making for a very nice result. The same trend occurs in all three problem types considered (mixture model, regression task, SAC).  - As far as I can tell, this paper proposes a novel variational inference algorithm with strong theoretical justifications, improving upon the previous best method. The experiments clearly show the superiority of the proposed method. The paper is very well written (with a few minor typos) and seems like an novel contribution to the variational inference literature.


A Convex Duality Framework for GANs
Generative adversarial network (GAN) is a minimax game between a generator mimicking the true model and a discriminator distinguishing the samples produced by the generator from the real training samples. Given an unconstrained discriminator able to approximate any function, this game reduces to finding the generative model minimizing a divergence score, e.g. the Jensen-Shannon (JS) divergence, to the data distribution. However, in practice the discriminator is constrained to be in a smaller class F such as convolutional neural nets. Then, a natural question is how the divergence minimization interpretation will change as we constrain F. In this work, we address this question by developing a convex duality framework for analyzing GAN minimax problems. For a convex set F, this duality framework interprets the original vanilla GAN problem as finding the generative model with the minimum JS-divergence to the distributions penalized to match the moments of the data distribution, with the moments specified by the discriminators in F. We show that this interpretation more generally holds for f-GAN and Wasserstein GAN. We further apply the convex duality framework to explain why regularizing the discriminator's Lipschitz constant, e.g. via spectral normalization or gradient penalty, can greatly improve the training performance in a general f-GAN problem including the vanilla GAN formulation. We prove that Lipschitz regularization can be interpreted as convolving the original divergence score with the first-order Wasserstein distance, which results in a continuously-behaving target divergence measure. We numerically explore the power of Lipschitz regularization for improving the continuity behavior and training performance in GAN problems.

So I generally like the subject and theory of this paper very much. The authors illuminate some of the connections between many different GAN formulations and regularizations under the same convex dual framework. Unfortunately, the overall message I think will be lost on most readers, as the main bullet points of the paper seem fairly disjoint and sometimes are unclear w.r.t. GANs in the literature. For instance, when the distribution Q is introduced, it’s unclear what this is and how it is optimized. It’s finally clarified a bit in equation (21), but sadly there’s no real experiments to demonstrate that this formulation is sensible. When we finally get to equation (20), it’s unclear what this form gets us (other than eqs 21 and 22).   The biggest problem though is the lack of experimentation. I like the theory, but I’m constantly asking myself what is the consequences of all of these lemmas and whether any of this is grounded in experimentation. The authors provide some experiments on MNIST, but we’re to be led to believe that some samples might be better than others based on visualization alone (I certainly cannot tell that some sets are better than others or even good w.r.t. the target dataset). The reason we need more experimentation, particularly in this paper despite it’s contribution being mostly theoretical, is that the family of functions F is a central player in this paper and the claims being made. In neural network optimization, F is not constrained only by the architecture / regularization, but by gradient descent based optimization.  In this form, I would not recommend the paper being accepted without a better balance of theory and experimentation showing that the theory is grounded as well as to provide much needed interpretation for the reader. I would strongly encourage the authors to restructure and condense some of the theory to allow for room for this. For instance, can we explore equations 21 and 22 more with toy data? Are there ways of illuminating for the reader the importance of the 3rd player (as realized through Q)? Are there experiments that can better show the importance of the mixture model w.r.t. the duality framework?  =============== So, as I said I really like this work. The proposed changes I think sound marginally better, but what I'm hoping for is a more thorough analysis w.r.t. the method that goes beyond just inception scores (its unclear after reading the response that authors intend to do any more than this). I think for this work to have broader impact, that some additional analyses are necessary illuminating my the new objective is similar to / different from existing methods (WGAN, Roth et al., Mescheder et. al., Spectral norm, etc).  So please consider including more analysis in the final version if the paper gets accepted.


Visual Memory for Robust Path Following
Humans routinely retrace paths in a novel environment both forwards and backwards despite uncertainty in their motion. This paper presents an approach for doing so. Given a demonstration of a path, a first network generates a path abstraction. Equipped with this abstraction, a second network observes the world and decides how to act to retrace the path under noisy actuation and a changing environment. The two networks are optimized end-to-end at training time. We evaluate the method in two realistic simulators, performing path following and homing under actuation noise and environmental changes. Our experiments show that our approach outperforms classical approaches and other learning based baselines.

This paper considers the problem of retrace the trajectory from noisy visual observations. The proposed approach firstly takes a sequence of images of the path and encode as sequential memories. In the online setting, a neural network is able to attend to the past memory and take actions accordingly. The algorithm has been verified over two interesting 3D indoor scene dataset.    Pros:    * The paper is very well-written. Introduction is clearly motivated, novelty and relationship with previous works have been discussed in a very modest tone, approach is very clear and figures are well illustrated. I really appreciate the author’s efforts on writing and enjoy reading this paper.     * Proposed algorithm is well designed. I especially like how the proposed approach could be related with the mapping, localization and motion planning procedure in traditional robotics.    * The author considered challenging cases where path is not exactly the same and changes present in the world.     * It shows an interesting, promising, yet unsolved direction to NIPS community, which I believe would draw attentions in short future.    Cons:    * The proposed method is limited to work only with very related paths (with limited perturbed noise or reversed trajectory). And it’s difficult to see the potential of generalize to a very different path using the current feature synthesis algorithm. Human indeed has this capability with very limited exploration.    * Failure cases were not discussed in detail. There are half of the cases failed. Could you analyze more of those cases and point out potential future directions to authors?    * Baselines (especially classic SLAMs) are not favoured under current setting.     * Recurrent settings might be restricted to short paths, e.g. 30 steps in paper’s setting. It is also unclear to me what information hidden states take to policy function. Could you report performance with a non-recurrent policy pi(mu_t, Phi(O_t))?    Some detailed comments:   It seems SLAM baseline suffers a lot in this simple scenario, in which world changes are not very significant. This is quite surprising to me given the capacity of COLMAP. Thus I would like to know more about the author’s experiment on the SLAM baseline to ensure a proper comparison:    * Are you using incremental bundle adjustment or full bundle adjustment setting in COLMAP?     * Do you give BA solver your true camera intrinsic matrix?     * How many frames you used per each path? Still 30? And action between each step is also pure rotation with 30 degree or 40cm forward? Note that both settings are not realistic in real-world applications (images could be received at much higher rate) and could make SLAM based approaches unfavoured.     * Have you actually managed to estimate accurate camera pose and reconstruct dense point cloud with good quality using COLMAP under your setting? Do you think localization failures are due to poor mapping or poor SIFT matching?    * In the path following stage, why do only do SIFT matching against the points only, rather than the full incremental BA pipeline (SIFT matching plus non-linear reprojection error minimization) to estimate the new image of the camera pose?   Robotics community does have a lot of online localization algorithm with loop closure detection capability, e.g. Fabmap. These works should be definitely considered to compare as stronger geometric baselines.


Are ResNets Provably Better than Linear Predictors?
A residual network (or ResNet) is a standard deep neural net architecture, with stateof-the-art performance across numerous applications. The main premise of ResNets is that they allow the training of each layer to focus on fitting just the residual of the previous layer's output and the target output. Thus, we should expect that the trained network is no worse than what we can obtain if we remove the residual layers and train a shallower network instead. However, due to the non-convexity of the optimization problem, it is not at all clear that ResNets indeed achieve this behavior, rather than getting stuck at some arbitrarily poor local minimum. In this paper, we rigorously prove that arbitrarily deep, nonlinear residual units indeed exhibit this behavior, in the sense that the optimization landscape contains no local minima with value above what can be obtained with a linear predictor (namely a 1-layer network). Notably, we show this under minimal or no assumptions on the precise network architecture, data distribution, or loss function used. We also provide a quantitative analysis of approximate stationary points for this problem. Finally, we show that with a certain tweak to the architecture, training the network with standard stochastic gradient descent achieves an objective value close or better than any linear predictor.

Summary This paper studies loss surface of residual networks (ResNets). The main contribution of this paper is to prove that any local minimum of risk (empirical or population) of ResNet is better than the risk achieved by the best linear predictor. The authors prove this by upper-bounding the gap between the risk achieved by ResNet and the best linear predictor for “approximate local minima,” which is defined as \epsilon-SOPSP in the paper. For any local minima of ResNet, the upper bound becomes zero, which implies that the local minima is at least as good as (in terms of risk value) the best linear predictor.   After that the authors point out that, the bound on the gap depends on the norm of parameters, so the bound is small only if the parameters are close to zero. Thus, unless one can bound the norm of parameters, no meaningful algorithmic result can be derived from the main theorem. They show by an example in which an approximate local minimum has much worse risk than linear predictor, and also that regularization can introduce spurious local minimum.  The paper concludes with an algorithmic result that, by tweaking the ResNet architecture, training with stochastic gradient descent achieves a risk value close to (or better than) linear predictors.  Comments I like this work a lot. Understanding the loss surface of neural network models is very important, and this theoretical work proves that ANY local minima of ResNet are provably better than linear models. Given that many other existing results require differentiability of local minima, it is very interesting that the main theorem works even for non-differentiable local minima. It is also impressive that they prove this under almost no assumption on data distribution, network architecture, or loss function.  The authors are honest about the weakness of the main theorem; they admit that it is hard to get an algorithmic result. However, they partially remedy this issue by tweaking the architecture with almost the same expressive power and show a positive algorithmic theorem.  The writing of the paper is very clear and the paper reads smoothly.   Overall, it is a well-written paper that contains an interesting result on the energy landscape of ResNets. I recommend acceptance.  Minor points - In the proof sketch of Lemma 1, I think it’d be better to be more explicit about the definition of $d\ell$, for example: $d\ell = \frac{\partial}{\partial p} \ell(p;y) \vert_{p = w^T (x+V f_\theta (x))}$. And maybe replace $d\ell$ with some other single character?  - There is a missing right parenthesis in the beginning of Theorem 2.  ***************Modified on 7/31*************** I’m terribly sorry that it’s already late, but I want to ask the authors (if they have chance to see this) for clarification of whether the ResNet architecture in the paper can be implemented with a stack of standard ResNet units.  In the “pre-activation” structure, usually each resnet unit is defined as $t \mapsto t + g_i(t)$, where $g_i(t)$ can be some network, say $g_i (t) = V_i * ReLU(U_i * t)$. If we stack $L > 1$ such residual units, the network would look like  h_0 = x, h_i = h_{i-1} + g_i( h_{i-1} ) for i = 1, … , L, h_{L+1} = w^T * h_L,  and the output $h_{L+1}$ can be written as  h_{L+1}  = w^T [ x + g_1(x) + g_2(h_1) + … + g_L(h_{L-1}) ] = w^T [ x + V_1 * ReLU(U_1 * x) + V_2 * ReLU(U_2 * h_1) + … + V_L * ReLU(U_L * h_{L-1}) ].  In contrast, in this paper, the authors analyze a ResNet $x \mapsto w^T [ x + V f_\theta (x) ]$. To me, it seems difficult to formulate the above network output $h_{L+1}$ into this particular form.  Does the theorem in the paper hold for this arbitrarily deep stack of residual units, or does it only hold for *only one* residual unit followed by a linear layer? If the former is the case, can you clarify why it holds for $L>1$ units? If the latter is the case, the paper should in some way emphasize this, because readers can get confused between “$f_\theta$ can have arbitrary structure and depth” and “the entire ResNet can be arbitrarily deep.” A possible option could be to refer to the architecture as a “shallow ResNet.”  Again, I’m so sorry that I’m too late for additional questions and comments. If you get to see this modification before the deadline, it would be very helpful if you can provide answers.  ***************Modified after rebuttal*************** I would like to thank the authors for generously providing feedback to my last-minute questions. And yes, it’d be more helpful for readers to emphasize (although the authors already put significant efforts on it!) the difference between a residual unit vs network, because some readers may not notice the difference of the terms (unit vs network) and then misunderstand the point of the paper; actually, this almost happened to me.  Even though the paper only considers ResNets with a single residual unit, my evaluation of the paper stays the same. This is because this paper pushes forward the understanding of loss surface of ResNets, and many existing theoretical works on (conventional) fully connected neural nets are also limited to one-hidden-layer networks.


Local Differential Privacy for Evolving Data
There are now several large scale deployments of differential privacy used to collect statistical information about users. However, these deployments periodically recollect the data and recompute the statistics using algorithms designed for a single use. As a result, these systems do not provide meaningful privacy guarantees over long time scales. Moreover, existing techniques to mitigate this effect do not apply in the "local model" of differential privacy that these systems use. In this paper, we introduce a new technique for local differential privacy that makes it possible to maintain up-to-date statistics over time, with privacy guarantees that degrade only in the number of changes in the underlying distribution rather than the number of collection periods. We use our technique for tracking a changing statistic in the setting where users are partitioned into an unknown collection of groups, and at every time period each user draws a single bit from a common (but changing) group-specific distribution. We also provide an application to frequency and heavy-hitter estimation.

This paper mainly considered streaming mean estimation problem under local differential privacy model. Though there has been lots of work about locally private mean estimation problem, utility guarantees of naïve implementation for previous methods in the streaming model will degrade with time period t at least in order O(\sqrt{t}), because of composition theorem. The main contribution of this paper was proposing an interactive private algorithm that achieved utility only in O(\ln T) order under reasonable settings, and its idea kind of resembled sparse vector technique in DP model. Further, authors also showed a direct application of their technique for streaming heavy hitter problem.      Overall, this paper is well written, and I personally quite like its idea. Given fixed privacy budget, utility guarantees of common private algorithms usually degrade with time t in polynomial order, which is a well-known problem in privacy literature. This paper makes some sound assumptions about the underlying estimation problem, and obtains an O(\ln T) order performance guarantee, which advances existing techniques and results.   However, I feel kind of confused about some parts in the paper, and I put them in the below “Cons” part.   Pros: 1. Design a novel locally private algorithm that can achieve O(ln T) order performance in the streaming estimation problem under reasonable assumptions; 2. Apply above technique to common heavy hitter problem and obtain corresponding guarantees.   Cons: 1. In the proof of lemma 7.6 (line 454), how do we obtain this inequality through the fact “user i sets VoteYes_i^t’ as true”?  What if b^* equals -1, and 2^{log T – b^*} divides t? In this case, VoteYes_i^t’ is also set as yes, but we don’t have above inequality (i.e. line 454); 2. Assumption 4.2 seems to be a little restricted. What if this assumption doesn’t hold? For example, the size of smallest subgroup is in order O(n^{1/4});  3. In line 261, authors assume d >> n for heavy hitter problem. I know it is a common parameter regime, but I wonder whether d >> n is a necessary condition like assumption 4.2. What if d is not so large as specified in line 297?   Other minor comments: 1. In line 159, \mu^r should be \mu^r = 1/n * (|S_1|*\mu_1^r+ … + |S_m|* \mu_m^r); 2. In line 198, in the definition of \tao_b, constant 10 should be 12 according to the appendix; 3. In line 5 of the algorithm 5 (and other similar places), the notation of noise level b is misleading with the subscript of \tao_b; 4. In line 19-21 of the algorithm 5, it seems it doesn’t play any role in the algorithm, and could be deleted; 5. Line 386 is reduplicative with previous sentence; 6. In the bottom inequality of page 10, it should be a^t in the bracket, not (a^t, \tilde{p}^t); 7. In the second equation in line 392, authors omitted a^t in the conditional probability for \tilde{p}^t.   Update: Thanks for authors' detailed responses. For my first concern, I just misunderstood the decide condition. For the 4th comment in "minor comments" part, there is a typo  and algorithm 5 should be algorithm 1. Overall, it is a good paper.


Iterative Value-Aware Model Learning
This paper introduces a model-based reinforcement learning (MBRL) framework that incorporates the underlying decision problem in learning the transition model of the environment. This is in contrast with conventional approaches to MBRL that learn the model of the environment, for example by finding the maximum likelihood estimate, without taking into account the decision problem. Value-Aware Model Learning (VAML) framework argues that this might not be a good idea, especially if the true model of the environment does not belong to the model class from which we are estimating the model. The original VAML framework, however, may result in an optimization problem that is difficult to solve. This paper introduces a new MBRL class of algorithms, called Iterative VAML, that benefits from the structure of how the planning is performed (i.e., through approximate value iteration) to devise a simpler optimization problem. The paper theoretically analyzes Iterative VAML and provides finite sample error upper bound guarantee for it.

The paper proposes a modification of a reinforcement learning (RL) framework, called Value-Aware Model Learning (VAML), that makes the associated optimization problem more tractable. VAML is a model-based approach that takes into account the value function while learning the model. In its original formulation, VAML poses the problem as a “min max” optimization in which one seeks a model considering the worst-case scenario over the space of representable value functions. This paper proposes to replace the problem above with a sequence of optimizations whose objective functions include the actual value-function approximations that arise in value iteration (that is, one replaces the “max” above with a sequence of concrete approximations). The paper presents a theoretical analysis of the proposed method, first providing finite sample guarantees for the model-based approximation, then providing a general error propagation analysis, and finally combining the two.  The main point of the paper, to replace a worst-case minimization with a sequence of easier problems, is insightful, and can potentially lead to practical algorithms. The paper is also didactic, with clear explanations throughout that help the reader understand each step of the derivations. I have a few observations regarding the proposed approach itself and the way it is presented in the current version of the paper.  It seems to me that the fact that the model P computed by IterVAML depends on the current value function approximation Q restricts the use of the data, and potentially the applicability of the method itself. The way Algorithm 1 is presented suggests that it can be used in the online scenario, where the agent alternates between collecting sample transitions and refining its value function approximation Q. When the model is independent of the current approximation Q, like in conventional model-based approaches, the data collected in successive iterations can be slowly incorporated into the model (this is the case of Dyna, for example). However, as far as I can see, the only way to incorporate all the data into a model that depends on the current Q, like in IterVAML, is to store all the sample transitions collected along the way.   Although the point raised above might come across as an algorithmic limitation of lesser importance, it may in fact touch upon one of the core arguments in favor of model-based RL. Model-based approaches similar to IterVAML learn a set of conditional distributions P(.|X,A) that are then used to compute an approximation of Q(X,A). As the former object is potentially much more complex than the latter, when is it reasonable to expect that this indirect approach will be more effective than simply learning Q? One possible case is when P is in fact easier to learn than Q. Since in general there is no reason to expect this to be true, another argument in favor of model-based approaches is that P can use all the data, even if the collection of samples is influenced by the current policy computed by value iteration. This is only possible because there is a single model; one cannot do the same with value functions without storing all the data because approximation Q_{t+1} depends on Q_{t}. Thus, unless one is willing to store all samples collected, P can indeed become a better approximation than Q_t, for a large enough t,  despite the fact that it is a more complex object. However, by tying each of its models to a value function approximation, IterVAML loses the ability to incorporate all the data into the model without storing all the samples. In summary: when IterVAML is used online, it seems less likely that the strategy of computing an approximation P to then compute Q will be as effective as in other model-based approaches (including the original formulation of VAML). Note that this issue does not arise in batch RL, where one has a single, fixed, set of sample transitions.  The current version of the paper also lacks a more intuitive discussion of the potential benefits of taking the value function into account while learning the model. In line 82 the authors argue that the objective function optimized by the original VAML is tighter than that optimized under other criteria, such as maximum likelihood estimation. It would be nice to see a more intuitive explanation of this point, perhaps specialized to IterVAML, especially if such an explanation were accompanied by some simple experiment illustrating the point. It would also be interesting to see simple experiments illustrating how IterVAML compare against other model-based approaches, such as the original VAML or some standard algorithm that minimizes the KL-divergence. Such a comparison would also provide some intuition on the issue raised on the previous paragraph.  Other points:  -- The discussion on the policy evaluation version of IterVAML, starting in line 141, seems a bit lengthy to me. Why not simply say that the evaluation case can be obtained if we define a new MDP with a single action per state corresponding to the action that the policy to be evaluated would pick?  -- Assumption A2 is difficult to understand; having a more intuitive explanation here would help a lot.  -- The presentation seems a bit rushed at times. For example, in line 269 the authors promise an explanation that is never given (probably a result of editing the full version of the paper). The paper also ends quite abruptly; it would be nice to have a summary in the end reminding the reader of all the contributions.  Minor corrections:  Line 88: provide *an* expression 101: MDP -> MDPs 199: It is probably best to use another notation for the radius, because ‘R’ is too much associated with rewards in the context of RL 201: I think mathcal{N} is not defined 229: Closing parenthesis is wrongly superscripted 255: function -> functions 257: *an* integer k \ge 0 258: “denote” shows up twice  Post-rebuttal update: --------------------------  Thank you for the very detailed explanation addressing my concern! I suggest the authors mention this issue in the main paper and perhaps add the discussion in the rebuttal to the appendix. It would also be nice to see simple experiments illustrating the behavior of the proposed approach in practice, as mentioned in some of the reviews and also in the rebuttal.


Learning a latent manifold of odor representations from neural responses in piriform cortex
A major difficulty in studying the neural mechanisms underlying olfactory perception is the lack of obvious structure in the relationship between odorants and the neural activity patterns they elicit. Here we use odor-evoked responses in piriform cortex to identify a latent manifold specifying latent distance relationships between olfactory stimuli. Our approach is based on the Gaussian process latent variable model, and seeks to map odorants to points in a low-dimensional embedding space, where distances between points in the embedding space relate to the similarity of population responses they elicit. The model is specified by an explicit continuous mapping from a latent embedding space to the space of high-dimensional neural population firing rates via nonlinear tuning curves, each parametrized by a Gaussian process. Population responses are then generated by the addition of correlated, odor-dependent Gaussian noise. We fit this model to large-scale calcium fluorescence imaging measurements of population activity in layers 2 and 3 of mouse piriform cortex following the presentation of a diverse set of odorants. The model identifies a low-dimensional embedding of each odor, and a smooth tuning curve over the latent embedding space that accurately captures each neuron's response to different odorants. The model captures both signal and noise correlations across more than 500 neurons. We validate the model using a cross-validation analysis known as co-smoothing to show that the model can accurately predict the responses of a population of held-out neurons to test odorants.

The authors describe the application of a Gaussian Process Latent Variable Model to neural population recordings in the Piriform cortex, obtained from mice, using a fluorescent calcium reporter. The goal of the study is to model the relations between different odorants, which presumably are encoded by the brain to lie in some smooth and continuous latent space, although the Piriform cortex itself has never been shown to have the same obvious topography as e.g. the visual or auditory cortices. The central claim of the paper is that although odorants can be "described by thousands of features in a high-dimensional chemical feature space", the "dimension of olfactory perceptual space" is unknown, and the authors propose a model that identifies a "low-dimensional space governing odor perception" learned from the neural activity caused by presenting an odor to a mouse undergoing brain imaging. Discovering a true or even merely useful latent embedding of monomolecular odors is indeed an open problem, and there is a question of whether there even exists a continuous latent space for odorants. There is a body of prior work on this topic that unfortunately was not cited, such as Koulakov 2011, Mamlouk et al 2003, Haddad et al 2008 and Iurilli et al 2017.   The model of choice here is a GPLVM augmented to account for specific covariance structure among trial-to-trial noise variability per neuron, as well as covariance between neural population responses to odorant stimuli. This is modeled efficiently using a Kronecker-factored full covariance matrix. Although interesting, it is my understanding that the specific model structure and optimization procedure are not methodological innovations. Regardless, if the goal of the work is to discover useful representations of odors, comparison to alternate model formulations for building the embeddings is crucial. For instance, how well does this modeling approach compare to using logistic regression on the Dragon molecular descriptors as input?  Along those lines, I particularly appreciated the study of alternate treatments of the covariance structure in the model on synthetic data, but would have preferred this to be run on the real data. Further, applying this kind of test to the choice of the RBF kernel, which likely has dramatic effects on the structure of the embedding, would provide further evidence on whether the particular modeling choices made here serve the purpose of the work. For instance, if the authors hypothesized that odors are naturally organized in a smooth embedding space, the choice of the RBF kernel artificially inflates the likelihood of finding such an embedding after fitting the model.  The experimental set up was described well, but some details were missing. It would be useful to know if there is a mouse driver line, or if a non-targeted viral transfection was used, how exactly 'neuropil contamination' was removed (this is often a very ad hoc computer vision processing step that can dramatically affect data quality).   Overall, I found the paper clearly written, although dense in places. Unfortunately, I do not believe that this submission contains new knowledge of interest to the NIPS community, and the scientific findings are not tested thoroughly enough in the current state of the paper to be of significance.  Some specific comments:  - The use of the term 'topography' is a bit confusing, as it is usually used in the neuroscience context to refer to a physical correspondence to stimulus space and the layout of a cortical area. E.g., retinotopy refers to the layout of the retina mapping physically onto visual cortex.  - The prior for embedding locations is set to be an uninformative prior, due to the lack of inclusion of chemical features per odorant. However, that information is available, and would be very useful to incorporate.  - No axes or labels on 2B  - An existing hypothesis about how odors are encoded in the piriform cortex is that it is performing random projections through the anatomy of connections from the mitral cells to pyramidal cells in the piriform. Random projections would thus be a natural baseline to compare against.   - Typo: "captures each neuron?s response"    ---------------------------------------------------------------- After author's response and reviewer discussion: I was surprised to see how out-of-range this review was, and after discussion with other reviewers, have revised my score upwards. My main critical thoughts on the paper were: - If this is a methods paper, it relies heavily on a single application, doesn't present clear technical superiority over competitive alternative methods, and doesn't present sufficient technical novelty, with a small exception. The GPLVM model proposed is technically interesting and challenging, but the main features used are not new, which the authors acknowledge. The exception here, as the authors point out, is modeling multi-trial covariance structure, which is indeed a tough problem to tackle in neuroscience data. I weighted the work down because of limited technical novelty, mostly ignoring the fact that this modeling approach has not been applied widely for modeling neural spiking data. Fusing an old technique with a new application is indeed an interesting endeavor, and thoroughly useful -- if we generate new knowledge in the application space.  - If this is an applications paper, we did not learn anything new about the application. The stated goal of the work is to find a smooth olfactory embedding, which practitioners in the field do hope exist and have been searching for for quite some time. Properties of a putative embedding have huge potential impact in olfactory neuroscience, and in understanding how sensory circuits in general process information from the outside world. However, the authors unfairly stack the deck in their favor -- they build a GPLVM model with a smooth priors, and do not provide the reader a way to evaluate if their embedding is superior along any other axis besides smoothness. I weighted the work down because their results in this application were presented without sufficient context and comparison, and because of the circular nature of how they defined success -- smoothness is good, so we fit a model that can likely only produce smooth embeddings.  However, it was brought to my attention by fellow reviewers that non-novel but cutting-edge methods (which this paper uses) cleanly applied to interesting domains (which this paper does), even if no new knowledge is generated, is of interest to the NIPS community.


Towards Text Generation with Adversarially Learned Neural Outlines
Recent progress in deep generative models has been fueled by two paradigms -autoregressive and adversarial models. We propose a combination of both approaches with the goal of learning generative models of text. Our method first produces a high-level sentence outline and then generates words sequentially, conditioning on both the outline and the previous outputs. We generate outlines with an adversarial model trained to approximate the distribution of sentences in a latent space induced by general-purpose sentence encoders. This provides strong, informative conditioning for the autoregressive stage. Our quantitative evaluations suggests that conditioning information from generated outlines is able to guide the autoregressive model to produce realistic samples, comparable to maximum-likelihood trained language models, even at high temperatures with multinomial sampling. Qualitative results also demonstrate that this generative procedure yields natural-looking sentences and interpolations.

Update after author response: It's good to see the efforts of conducting human evaluation, and I encourage the authors to include more details, e.g., how the annotators are selected, what questions are asked, into the next revision. I'm changing the score to 5 now.  ============== This manuscript proposes a generative adversarial approach for text generation. Specifically, sentences are encoded into vectors in a "latent space" by a RNN encoder; a decoder conditions on the vector and generates text auto-regressively. The latent representations are adversarially regularized towards a fixed prior. The method is evaluated on both unconditional and conditional generation settings, and the paper argues for better performance than baselines.  The method is not that surprising, given Hu et al., (2017) (missing in the references), and Kim et al., (2017), among others. Many claims are made, including both theoretical justifications and better empirical performance. However, I do not find either of them is well-supported. I can hardly find any insightful theoretical discussion of the proposed method, in comparison to previous works. It is probably still okay, should the submission put more thoughts into the empirical part by, e.g., showing better performance in carefully designed human evaluation. Unfortunately, this is not the case.  That being said, the third point of the claimed contribution --- the gradient-based interpolation between sentence embeddings appears intriguing to me. I would encourage the authors to discuss how it relates to existing works in more detail, and put it earlier in the narrative.  In sum, this submission would have been much stronger should it come out, e.g., one year earlier. I cannot recommend that it is accepted to NIPS in its current status, but would be looking forward to a revised version.  References  Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P. Xing. 2017. Toward Controlled Generation of Text. In Proc. of ICML.


Topkapi: Parallel and Fast Sketches for Finding Top-K Frequent Elements
Identifying the top-K frequent items in a collection or data stream is one of the most common and important operations in large data processing systems. As a result, several solutions have been proposed to solve this problem approximately. We observe that the existing algorithms, although theoretically sound, are suboptimal from the performance perspective because of their limitations in exploiting parallelism in modern distributed compute settings. In particular, for identifying top-K frequent items, Count-Min Sketch (CMS) has an excellent update time, but lacks the important property of reducibility which is needed for exploiting available massive data parallelism. On the other end, the popular Frequent algorithm (FA) leads to reducible summaries but its update costs are significant. In this paper, we present Topkapi, a fast and parallel algorithm for finding top-K frequent items, which gives the best of both worlds, i.e., it is reducible and has fast update time similar to CMS. Topkapi possesses strong theoretical guarantees and leads to significant performance gains due to increased parallelism, relative to past work.

This paper presents topkapi, an new algorithm that estimates the most frequent elements in a set in a distributed fashion.  This paper presents some positive developments on an important topic for big data and would be a good candidate for publication at this conference. However, this reviewer feels that the paper could be significantly strengthened before publication.  - structure: the current paper goes at length to describe a hybrid between CMS and FA, but it casually turns around in section 3.3 to essentially dismiss the sketch part of the algorithm. The reviewer would suggest to focus earlier on presenting the algorithm as FA with sketch-flavored hashing.  - proofs: this reviewer is of the opinion that formal guarantees are important for such algorithms and should be made explicit to practitioners. Unfortunately, all the theorems are relegated to the appendix. At least presenting the main results in the paper would reinforce the paper, as it currently is presented more as a discussion without obvious formal backing.  - presentation: CMS computes counts of elements, while FA focuses on the identification of the most frequent elements. The paper puts them on equal footing, which led this reviewer to wonder if identifying or computing the frequencies of the top-k elements was the eventual goal of the paper. In particular, presenting CMS as not good for reducibility seems to distract the flow, as pure sketches (without tracking the top elements in a heap) are excellent for distribution and reducibility.  Some other comments: - Both 'Frequent' and 'FA' are used, which is a bit confusing. The authors should pick one notation. - The reviewer wonders how the guarantees would be strengthened if, for each of the CMS bins in the sketch, 2 or 3 FA counts would be kept instead of just one. The update would still be trivial with 2 comparisons, but it could lead to more robust estimates, at least in theory. - In practice, it is useful to also have access to the frequencies of the most frequent elements (which is provided by sketches). How would the authors plan to address this point? - Regarding the experiments, the reviewer would be curious to see how pure CMS fares (with a first sketch building pass followed by a heavy hitter hash-based identification pass). This would be more fair and also simpler than a heap augmentation of CMS.  Overall, the reviewer would welcome this contribution, provided the points above are addressed.  UPDATE: The reviewer thanks the authors for their comments. - regarding the 2-pass CMS + Top-K, this is a 'natural' approach for batch or memory-constrained settings (and of course prohibitive for streams, but the context of the paper is more general than streaming). Given the good performance of topkapi, I would recommend this is included in the final submission, especially as it explicitly now compares against spark, and spark has both built-in CMS and top-k. - regarding bounds on results, the authors rightfully mentioned that CMS and FA provide upper and lower bounds on the exact value. I believe this is worth mentioning as this is important information in practice - the updated version of the paper mentions spark prominently in the abstract as a point of comparison, but the reviewer sees no experiment against it (although the results would be expected to be quite in favor of topkapi). This should be addressed in the final version, either by removing mention of spark or at least mentioning some experiments. These are fairly minor points though, and this reviewer would welcome this contribution provided the points above are addressed in the final revision.


A Unified View of Piecewise Linear Neural Network Verification
The success of Deep Learning and its potential use in many safety-critical applications has motivated research on formal verification of Neural Network (NN) models. Despite the reputation of learned NN models to behave as black boxes and the theoretical hardness of proving their properties, researchers have been successful in verifying some classes of models by exploiting their piecewise linear structure and taking insights from formal methods such as Satisifiability Modulo Theory. These methods are however still far from scaling to realistic neural networks. To facilitate progress on this crucial area, we make two key contributions. First, we present a unified framework that encompasses previous methods. This analysis results in the identification of new methods that combine the strengths of multiple existing approaches, accomplishing a speedup of two orders of magnitude compared to the previous state of the art. Second, we propose a new data set of benchmarks which includes a collection of previously released testcases. We use the benchmark to provide the first experimental comparison of existing algorithms and identify the factors impacting the hardness of verification problems.

The main contribution of this paper is to provide a framework for NN verification based on Branch and Bound optimization which allows for reframing existing state-of-the-art verification methods as special cases of this framework. With this unified framework, the authors were able to identify algorithmic improvements in the verification process that lead to a speed-up of almost two orders of magnitude.   In order to design a unified framework, the authors convert the verification problem into a global optimization problem. The idea is to convert any boolean formula over linear inequalities into a sequence of linear and max-pooling layers.  The verification problem is then reduced to finding out the sign of the global minimum of this network. To find this global minimum, the authors present an approach based on Branch and Bound paradigm.  They also show how verification methods based on Satisfiability Modulo Theory (SMT) fit in this framework. A generic branch and bound based algorithm is presented to find a global minimum for the verification problem. The authors also show how some of the published works for NN verification based on SMT theory (like Reluplex by Katz et.al and Planet by Ehlers et.al) can be converted to branch and bound framework for verification. With this unified framework, the authors were able to come up with many promising suggestions to improve the speed-up of verification algorithms which they include in their proposed algorithm. They evaluate their method against SMT based methods like Reluplex and Planet, MIP solvers based like MIPplanet. They use Collision Detection, Airborne Collision Avoidance System (ACAS) and PCAMNIST as their datasets for evaluation. PCAMNIST is a novel dataset introduced by the authors to understand the factors that influence various verification methods.   The authors observe that on the CollisionDectection dataset most of the solvers succeeded within 10s. In particular, they found SMT based solvers to be extremely fast on this dataset. On ACAS, most of the solvers could not verify within the timelimit of 2 hours. However, their method was able to achieve 98.40% within an hour as compared to the baseline Reluplex which achieved only 79.26% success within the two hour limit.  In addition to the evaluation, the authors do an ablation study on what changes made the most impact in terms of performance on the methods that they proposed. They come up with the following conclusions on their study: a) Smart branching contributes significantly to the performance gap b) Branching over inputs rather than over activations does not contribute much improvement c) Better bounds contribute to the rest of the performance gap In their experiments, they also observe the following: a) Larger the number of inputs, harder is the verification problem b) Wider networks take more time to solve as they have more non-linearities c) Properties with large satisfiability margin are easy to prove Quality:  This paper proposes a novel framework for NN verification. They make a strong case on how this framework can be instrumental in bringing about rapid improvements in the performance of NN verifications. They provide thorough analysis on what makes their proposed algorithm perform better than the other state-of-the-art approaches. The amount of speed-up that their approach achieves is also significant. Though this approach shows promising results on smaller networks, it would be interesting to see how it scales on larger networks.  Overall, the quality of this paper is good.  Clarity: The paper is quite clear in presenting the contributions that it makes in the field of verification methods for neural networks. Their main idea has been illustrated with simple examples that are easy to understand. The results and conclusions have been presented well with graphs. However, in Section 5.1 Figure1(b), the y axis is not visible clearly.    Originality: The field of formal verification of Neural networks has emerged fairly recently during the last few years. Most of the published works so far have used either SMT theory or MIP formulation to solve the verification problem. To the best of my knowledge, there has been no study that proposes a unified framework for this problem. The authors present a unique and interesting approach that enables us to convert any SMT based verification method or MIP based verification method to a branch and bound based paradigm. This framework makes it easier to come up with solutions to improve the verification process as the authors demonstrate through the various experiments.   Significance: Deep neural networks are now increasingly becoming a major component of many safety-critical applications. Thus, formal verification of neural networks has become an important research problem that needs to be solved in order to scale deep neural networks in safety-critical applications. By proposing a unified framework for NN verification, this paper comes up with significant improvements in runtime for verification algorithms. It would be interesting for the research community to analyze this framework which might lead to even more efficient algorithms.       A few detailed comments:  In Section 5.1 Figure1(b), the y-axis is not visible clearly. In Figure 4. (c) The values on x-axis are not clearly visible.  Though this approach shows promising results on smaller networks, it would be good to include larger networks and observe how the algorithm scales. Although the authors in this work focus on piecewise-linear NN’s, adding a few lines as to the feasibility of scaling this framework to general Neural networks might be helpful in motivating other research groups to study this framework further.


Dialog-based Interactive Image Retrieval
Existing methods for interactive image retrieval have demonstrated the merit of integrating user feedback, improving retrieval results. However, most current systems rely on restricted forms of user feedback, such as binary relevance responses, or feedback based on a fixed set of relative attributes, which limits their impact. In this paper, we introduce a new approach to interactive image search that enables users to provide feedback via natural language, allowing for more natural and effective interaction. We formulate the task of dialog-based interactive image retrieval as a reinforcement learning problem, and reward the dialog system for improving the rank of the target image during each dialog turn. To mitigate the cumbersome and costly process of collecting human-machine conversations as the dialog system learns, we train our system with a user simulator, which is itself trained to describe the differences between target and candidate images. The efficacy of our approach is demonstrated in a footwear retrieval application. Experiments on both simulated and real-world data show that 1) our proposed learning framework achieves better accuracy than other supervised and reinforcement learning baselines and 2) user feedback based on natural language rather than pre-specified attributes leads to more effective retrieval results, and a more natural and expressive communication interface.
: In the context of interactive image retrieval, the agent incorporates the user's feedback to iteratively refine retrieval results. Unlike existing work which are based on relevance feedback or relative attribute feedback, our approach allows the user to provide feedback in natural language.
on the specific form of user interaction largely constrain the information that a user can convey to benefit the retrieval process.
In this work, we propose a new approach to interactive visual content retrieval by introducing a novel form of user feedback based on natural language. This enables users to directly express, in natural language, the most prominent conceptual differences between the preferred search object and the already retrieved content, which permits a more natural human-computer interaction. We formulate the task as a reinforcement learning (RL) problem, where the system directly optimizes the rank of the target object, which is a non-differentiable objective.
We apply this RL based interactive retrieval framework to the task of image retrieval, which we call dialog-based interactive image retrieval to emphasize its capability in aggregating history information compared to existing single turn approaches [13,14,15,16]. In particular, a novel end-to-end dialog manager architecture is proposed, which takes natural language responses as user input, and delivers retrieved images as output. To mitigate the cumbersome and costly process of collecting and annotating human-machine dialogs as the system learns, we utilize a model-based RL approach by training a user simulator based on a corpus of human-written relative descriptions. Specifically, to emulate a single dialog turn, where the user provides feedback regarding a candidate image relative to what the user has in mind, the user simulator generates a relative caption describing the differences between the candidate image and the user's desired image. 1 Whereas there is a lot of prior work in image captioning [17,18,19], we explore the problem of relative image captioning, a general approach to more expressive and natural communication of relative preferences to machines, and to use it as part of a user simulator to train a dialog system. The efficacy of our approach is evaluated in a real-world application scenario of interactive footwear retrieval. Experimental results with both real and simulated users show that the proposed reinforcement learning framework achieves better retrieval performance than existing techniques. Particularly, we observe that feedback based on natural language is more effective than feedback based on predefined relative attributes by a large margin. Furthermore, the proposed RL training framework of directly optimizing the rank of the target image shows promising results and outperforms the supervised learning approach which is based on the triplet loss objective. The main contributions of this work are as follows:
• A new vision/NLP task and machine learning problem setting for interactive visual content search, where the dialog agent learns to interact with a human user over the course of several dialog turns, and the user gives feedback in natural language.
• A novel end-to-end deep dialog manager architecture, which addresses the above problem setting in the context of image retrieval. The network is trained based on an efficient policy optimization strategy, employing triplet loss and model-based policy improvement [20].
• The introduction of a computer vision task, relative image captioning, where the generated captions describe the salient visual differences between two images, which is distinct from and complementary to context-aware discriminative image captioning, where the absolute attributes of one image that discriminate it from another are described [21].
• The contribution of a new dataset, which supports further research on the task of relative image captioning. 2

This paper proposes a novel approach to interactive visual content retrieval.  It formulates the task as a reinforcement learning problem, and rewards the dialog system for improving the rank of the target image during each dialog turn.  To alleviate the cost of collecting human-machine conversations as the dialog system learns, the dialog system is trained with a user simulator, which is itself trained to describe the differences between target and retrieved images.  Experiments show the advantage of the proposed RL-based dialog approach over alternative baselines.   [strengths]  1. To my knowledge, this is the first dialog-based interactive image retrieval system where the system and user communicate using natural language.  The proposed framework and its components are appropriate for the problem.  2. The idea of using a user simulator for relative image captioning is interesting.  This helps avoid the costly alternative which would be to collect human-machine dialogs as the system learns.  3. A new dataset for the task of relative image captioning is introduced.  However, the usefulness of the dataset may be limited (see below, weaknesses  point 2).  [weaknesses]  1. The paper's conclusion that "user feedback based on natural language rather than pre-specified attributes leads to more effective retrieval results" is too strong given the weakness of the pre-specified attributes baseline (Sec. 5.2). To me, this is the biggest weakness of the paper. Specifically, if my understanding is correct, that baseline uses a 10-D attribute vector, which is computed using hand-crafted features like GIST and color histograms [13].  Thus, it is not clear whether the improvement is due to natural language feedback (as the paper claims) or if it's due to better features.  This aspect needs to be studied in more detail.   2. For the relative image captioning dataset to be truly useful, a more systematic and automatic way of evaluation should be presented that goes beyond user studies (as presented in Appendix C). This is important so that new methods that try to tackle this problem can be properly compared.    3.  It'd be helpful if the paper could provide some typical failure cases so that the reader can have a better understanding of the system.   [summary]  Overall, due to the novelty of the problem and sound approach, my initial rating is to accept the paper. However, there are also some weaknesses (as mentioned above) that I would like to see addressed in the rebuttal.  [post-rebuttal final comments] Overall, the rebuttal has addressed to some degree the weaknesses stated in my initial review. I am, however, not completely satisfied with the authors' response regarding my point that the paper's claim "user feedback based on natural language rather than pre-specified attributes leads to more effective retrieval results" is too strong. Any comparison with hand-crafted feature based approaches needs to be stated clearly as so, as it is difficult to draw other conclusions out of such a comparison; i.e., if the baseline were to use a deep representation, it may perform just as well as the proposed approach. This needs to be empirically verified. Although this weakness is not enough for me to lower my initial rating, I feel that the paper either needs to (1) perform the appropriate experiment (i.e. the baseline should use a 10-D attribute vector derived from a deep neural network) or (2) clearly state that the baselines' features are hand-crafted and thus much weaker and may be the key cause for its lower performance. The rebuttal states that the revised paper will have something along the lines of (2), but I think it's important to explicitly state that the baseline is using "hand-crafted features".


Learning To Learn Around A Common Mean
The problem of learning-to-learn (LTL) or meta-learning is gaining increasing attention due to recent empirical evidence of its effectiveness in applications. The goal addressed in LTL is to select an algorithm that works well on tasks sampled from a meta-distribution. In this work, we consider the family of algorithms given by a variant of Ridge Regression, in which the regularizer is the square distance to an unknown mean vector. We show that, in this setting, the LTL problem can be reformulated as a Least Squares (LS) problem and we exploit a novel metaalgorithm to efficiently solve it. At each iteration the meta-algorithm processes only one dataset. Specifically, it firstly estimates the stochastic LS objective function, by splitting this dataset into two subsets used to train and test the inner algorithm, respectively. Secondly, it performs a stochastic gradient step with the estimated value. Under specific assumptions, we present a bound for the generalization error of our meta-algorithm, which suggests the right splitting parameter to choose. When the hyper-parameters of the problem are fixed, this bound is consistent as the number of tasks grows, even if the sample size is kept constant. Preliminary experiments confirm our theoretical findings, highlighting the advantage of our approach, with respect to independent task learning.

This paper studies how to define an algorithm that, given an increasing number of tasks sampled from a defined environment,  will train on them and learn a model that will be well suited for any new task sampled from the same environment. The scenario  just described corresponds to the 'learning to lean' problem where a learning agent improves its learning performance with the number of tasks.  Specifically in this work the focus is on the 'ridge regression' family of algorithms and the environment consists in tasks that can be  solved by ridge regression with models around a common mean. In other words, we need a learning algorithm that besides solving regression problems, progressively learns how to approximate the environment model mean.   The transfer risk is a measure of how much the knowledge acquired over certain available tasks allow to improve future learning.  In the proposed setting the transfer risk corresponds to the difference between the estimated model environmental mean and the correct environmental mean. It can be well estimated with theoretical guarantees and can be explicitly minimized to 'learn to learn' the optimal regression model.  The novel contribution of the paper is in the choice of splitting the data of each tasks in two parts, one on which the regression problem is learned and the training error is minimized and a second one useful to estimate and minimize the transfer risk. Finally the proposed strategy is tested onto two synthetic dataset where the results confirm the paper thesis.  Ovearall the paper is well written but I find it dense in some points which makes it a little bit difficult to read. The proposed method is theoretically grounded and I did not find any evident flaw. I just remain with one doubt: what could be real-world cases where the assumptions done for the synthetic settings hold? And what would be the challenges to be tackled in case of extensions to non-convex learning problems?  After rebuttal ----------------- I've noticed that in the rebuttal some of the reviewers' questions are indicated as 'open problems' or deferred to future works, Anyway I do not have strong arguments against this paper. I align with the other reviewers's score by raising my rate from 6 to 7.


Stochastic Spectral and Conjugate Descent Methods
The state-of-the-art methods for solving optimization problems in big dimensions are variants of randomized coordinate descent (RCD). In this paper we introduce a fundamentally new type of acceleration strategy for RCD based on the augmentation of the set of coordinate directions by a few spectral or conjugate directions. As we increase the number of extra directions to be sampled from, the rate of the method improves, and interpolates between the linear rate of RCD and a linear rate independent of the condition number. We develop and analyze also inexact variants of these methods where the spectral and conjugate directions are allowed to be approximate only. We motivate the above development by proving several negative results which highlight the limitations of RCD with importance sampling.

This paper focuses on quadratic minimization and proposes a novel stochastic gradient descent algorithm by interpolating randomized coordinate descent (RCD) and stochastic spectral descent (SSD). Those two algorithms were studied in prior but they have drawbacks to apply for practical usage. Hence, authors combine them by random coin-tossing (i.e., with probability \alpha run RCD and with probability 1-\alpha run SSD) and provide the optimal parameters to achieve the best convergence rate. In addition, they study the rate of their algorithm for some special cases, e.g., a matrix in objective has clustered or exponentially decaying eigenvalues. In experiments, they report practical behavior of their algorithm under synthetic settings.  The proposed algorithm takes advantages of two prior works, i.e., efficient gradient update from RCD and fast convergence from SSD. And it is interesting that the rate of the algorithm is between that of RCD and SSD, but not surprising. Moreover, authors study some extreme cases for RCD that the rate is arbitrary large and it is questionable that the proposed algorithm have the cases. It would be better to mention some extreme cases for SSCD.  And authors benchmark their algorithm under only synthetic dataset. It is desired to test the algorithm under real-world dataset and report its performance. Furthermore, there have been studied many algorithms for quadratic minimization, the proposed algorithm should compete with other works.  Overall, the main contribution of this paper is interesting with an intuitive interpolating approach and some case studies. However, the limited experiments make this work weaken. I consider this works as marginally above the acceptance.   ***** I have read the author feedback. I appreciate authors for providing the details about my questions. *****


Scalar Posterior Sampling with Applications
We propose a practical non-episodic PSRL algorithm that unlike recent state-of-theart PSRL algorithms uses a deterministic, model-independent episode switching schedule. Our algorithm termed deterministic schedule PSRL (DS-PSRL) is efficient in terms of time, sample, and space complexity. We prove a Bayesian regret bound under mild assumptions. Our result is more generally applicable to multiple parameters and continuous state action problems. We compare our algorithm with state-of-the-art PSRL algorithms on standard discrete and continuous problems from the literature. Finally, we show how the assumptions of our algorithm satisfy a sensible parametrization for a large class of problems in sequential recommendations.

This paper scales up the previous posterior sampling RL methods, such as Thompson sampling, by extending to continuous state and action space. A Bayesian-style regret bound is given, and the author shows theoretically that the proposed algorithm has the state-of-the-art efficiency in time and space, and enjoys the applicability to continuous state and action problems. Given the merits, the paper seems to be written in a hasty manner, which makes the reading experience tougher than I expect.   There are some technical concerns 1. Assumption 2 seems to be restrictive to Gaussian noise. If the assumption is violated for non-Gaussian noise, what will happen and how robust is the algorithm?  2. One major concern is that the author claims that the proposed algorithm can scale up to continuous state/action space problems. However, in the experiments, there are only discrete action domains, which makes the comparison study less convincing. 3. The authors state that result is more generally applicable to continuous state-action problems; when dynamics of the system is parameterized by a scalar, their regret scales with the square root of the number of parameters and is independent of the number of states. This is opposite to the general sense. Since regret usually increases with the number of states. This paper assumes the MDP model is parameterized by a scalar, the parameterization of MDP is related with the number states. the regret is proportional with the complexity of the model parameterization. 4. Other methods for comparison study. The paper compares through simulations the performance of DS-PSRL algorithm with the latest PSRL algorithm called Thompson Sampling with dynamic episodes (TSDE). Some other algorithms should be compared with the proposed algorithm.   Some minor suggestions: pp2. The introduction is too long. A ‘Related work’ section should be split to formulate a separate section. line 169, ‘differential’ —> ‘differentiable’ line 192-194 the reference format of many places is incorrect. such as this place: there should be brackets of [Ouyang et al.] line 240-241 the TSDE work by Ouyang et al. should be introduced as early as the ‘Related work’ part. line 286: it should be “The algorithm provides generalization.”


Unsupervised Attention-guided Image-to-Image Translation
Current unsupervised image-to-image translation techniques struggle to focus their attention on individual objects without altering the background or the way multiple objects interact within a scene. Motivated by the important role of attention in human perception, we tackle this limitation by introducing unsupervised attention mechanisms that are jointly adversarially trained with the generators and discriminators. We demonstrate qualitatively and quantitatively that our approach attends to relevant regions in the image without requiring supervision, which creates more realistic mappings when compared to those of recent approaches. Figure 1: By explicitly modeling attention, our algorithm is able to better alter the object of interest in unsupervised image-to-image translation tasks, without changing the background at the same time.

To conduct image-to-image translation, this paper introduces attention mechanisms to CycleGAN [1]. The divergence between only the relevant parts of both source and target domains is minimized. Compared to recent approaches, the proposed method performs better in conserving the background and gets the lowest FID in quantitative analysis.   Related works Related works follows a clear structure. For image-to-image translation, it introduces supervised and unsupervised methods, followed by mentioning the problem which is the background is affected, and then it refers to some related research on attention learning.  Approach The idea of having continuous attention map for generator is good, and it is also good to read the authors’ explanations of why and how it is valuable. The reason why they use segmented attention map for discriminator is also well presented.  Experiemnts It is interesting to see Figure 3 and Figure 5 have translation results when there is no interested class in source domain, showing this approach applies appropriate attention mechanisms. Some experiment details like the choices of hyperparameters are absent, not being presented in the submission paper or supplementary material. But in the end of the paper, the authors say the code will be released, hopefully this will be the case.  Lack of qualitative analysis about UNIT approach[4].  Other comments The readability of this paper is good. The structure and diagram of the proposed attention method are clearly presented. In conclusion, the authors mention one constraint of the proposed method, which is not being robust to shape changes, but this paper doesn’t involve related examples. It would be better to have the mentioned horse-to-bird results in supplementary material. L210: the last row of Figure 5 doesn’t seem to be an albino tiger, but the last third row is, which the last row of the top part.


On Learning Intrinsic Rewards for Policy Gradient Methods
In many sequential decision making tasks, it is challenging to design reward functions that help an RL agent efficiently learn behavior that is considered good by the agent designer. A number of different formulations of the reward-design problem have been proposed in the literature. In this paper we build on the Optimal Rewards Framework of  that defines the optimal intrinsic reward function as one that when used by an RL agent achieves behavior that optimizes the task-specifying or extrinsic reward function. Previous work in this framework has shown how good intrinsic reward functions can be learned for lookahead search based planning agents. Whether it is possible to learn intrinsic reward functions for learning agents remains an open problem. In this paper we derive a novel algorithm for learning intrinsic rewards for policy-gradient based learning agents. We compare the performance of an augmented agent that uses our algorithm to provide additive intrinsic rewards to an A2C-based policy learner (for Atari games) and a PPO-based policy learner (for Mujoco domains) with a baseline agent that uses the same policy learners but with only extrinsic rewards. We also compare our method with using a constant "live bonus" and with using a count-based exploration bonus (i.e., pixel-SimHash). Our results show improved performance on most but not all of the domains.

This work attempts to use learnable intrinsic rewards in addition to conventional extrinsic reward from the environment to boost the agent performance calculated as conventional returns as the cumulative extrinsic rewards. This work can be seen as a variant of previous reward shaping and auxiliary rewards works but cannot be a more general version because though with more general mathematical form, it loses the consideration of domain knowledges, the key insights of previous works. Compared with its closest related works [Sorg et al. 2010, Guo et al. 2016], the method proposed here can be used in bootstrapping learning agents rather than only planning agents (i.e., Monte Carlo sampling of returns).         Strengths:  1. This work implements an intuitively interesting idea that automatically shaping the rewards to boost learning performance. And compared with closely related previous works, it is more general to be used in modern, well performed learning methods (e.g., A2C and PPO). 2. The presentation is clear and easy to follow.  3. The experiment evaluation shows obvious improvement in the performance of the proposed method.         Weakness:   1. The technical design is in lack of theoretical support. Even if we accept the design that the policy parameters are updated to optimize the combinational returns while the intrinsic reward parameters are updated to optimize only the extrinsic returns, which itself needs more justification (though has been taken in Sorg et al. 2010), the update derivation for the intrinsic reward parameters (i.e., Eq.7-10) is hardly convincing.   2. Though the superior performance is shown in the experiment, there are still worse performance cases (e.g., the forth panel in Fig.4), which lacks explanation.  3. As pointed out in Sec.1, the basic assumption to use reward shaping (or adding intrinsic reward) is to "change for better or for worse the sample (and computational) complexity of the RL agent learning from experience in its environment using the transformed reward function". But throughout the work, no theoretical, or experimental analysis can support this assumption based on the proposed design. One possible way to repair it will be to use simple thought/simulation experiments to showcase this, which will be much clearer than the performance shown in complicated benchmarks.


Reducing Network Agnostophobia
Agnostophobia, the fear of the unknown, can be experienced by deep learning engineers while applying their networks to real-world applications. Unfortunately, network behavior is not well defined for inputs far from a networks training set. In an uncontrolled environment, networks face many instances that are not of interest to them and have to be rejected in order to avoid a false positive. This problem has previously been tackled by researchers by either a) thresholding softmax, which by construction cannot return none of the known classes, or b) using an additional background or garbage class. In this paper, we show that both of these approaches help, but are generally insufficient when previously unseen classes are encountered. We also introduce a new evaluation metric that focuses on comparing the performance of multiple approaches in scenarios where such unseen classes or unknowns are encountered. Our major contributions are simple yet effective Entropic Open-Set and Objectosphere losses that train networks using negative samples from some classes. These novel losses are designed to maximize entropy for unknown inputs while increasing separation in deep feature space by modifying magnitudes of known and unknown samples. Experiments on networks trained to classify classes from MNIST and CIFAR-10 show that our novel loss functions are significantly better at dealing with unknown inputs from datasets such as Devanagari, NotMNIST, CIFAR-100, and SVHN.

# 5501 # Update based on the Authors Response Thank you for the clarification. Based on the response I am willing to vote for an accept.  # Summary This paper proposes a new approach to improve the model’s robustness against unknown classes. The authors propose a new objective function that, unlike common detection algorithms, does not add the background class to softmax. Instead, for a given background example, the objective function maximizes the entropy of the softmax layer, which leads to a low magnitude feature representation for the background sample.   In addition, the authors propose the use of objectosphere loss to distinguish the known/unknown classes further by penalizing the feature representation of the background sample so that the network does not respond to unknown objects.   Theoretical and empirical proof demonstrate the potential capability of the proposed approach.  # Quality The authors define the problem well and provide a thorough literature review. The theoretical background is also well explained in the method section. Experimentation also shows the capability of the proposed approach.  I would want to see one more set of experiment, in which the proposed approach is used within a real detection pipeline such as, as described in the paper RPN of F-RCNN.   # Clarity This is a well-written paper. It flows well, and easy to navigate.  It is not explicitly stated in the paper whether the author will release their implementation for the reproducibility of the paper.  # Significance The problem defined in the paper is significant especially within the real-world applications of object detection algorithms due to excessive exposure to unknown classes in real-world scenarios. The paper’s solution is significant to tackle these issues.


DropBlock: A regularization method for convolutional networks
Deep neural networks often work well when they are over-parameterized and trained with a massive amount of noise and regularization, such as weight decay and dropout. Although dropout is widely used as a regularization technique for fully connected layers, it is often less effective for convolutional layers. This lack of success of dropout for convolutional layers is perhaps due to the fact that activation units in convolutional layers are spatially correlated so information can still flow through convolutional networks despite dropout. Thus a structured form of dropout is needed to regularize convolutional networks. In this paper, we introduce DropBlock, a form of structured dropout, where units in a contiguous region of a feature map are dropped together. We found that applying DropbBlock in skip connections in addition to the convolution layers increases the accuracy. Also, gradually increasing number of dropped units during training leads to better accuracy and more robust to hyperparameter choices. Extensive experiments show that DropBlock works better than dropout in regularizing convolutional networks. On ImageNet classification, ResNet-50 architecture with DropBlock achieves 78.13% accuracy, which is more than 1.6% improvement on the baseline. On COCO detection, DropBlock improves Average Precision of RetinaNet from 36.8% to 38.4%.
(a) (b) (c) Figure 1: (a) input image to a convolutional neural network. The green regions in (b) and (c) include the activation units which contain semantic information in the input image. Dropping out activations at random is not effective in removing semantic information because nearby activations contain closely related information. Instead, dropping continuous regions can remove certain semantic information (e.g., head or feet) and consequently enforcing remaining units to learn features for classifying input image.

The authors present a version of dropout that zeros out square blocks of neurons rather than individual neurons. They experience robust improvements relative to dropout.  This is a good paper, with potentially very wide-spread applicability.   The authors only present validation error in all their studies, but not test error. They claim that this is "common practice". I quickly checked the ResNet / DenseNet / Stochastic Depth / ShakeShake and ShakeDrop papers. Only one paper (DenseNet) seems to use validation error consistently and one (ResNet) partially. So I wouldn't call reporting the validation error common practice. In any case, this is a terrible practice for obvious reasons so I would urge any author (including the authors of this paper) to break the trend by reporting test error. If the authors feel that reporting validation error is necessary for comparion, they can report both.    Did you use the validation set for hyperparameter tuning? If so, in what way? Since you write "we first estimate the block_size_factor to use (typically 0.5)", it seems you are using it for at least some tuning. How is reporting validation error on the final model then still unbiased?  I am confused by your treatment of the DropBlock hyperparameters. On page 3, you start by saying that you don't set gamma and block_size directly, but block_size_factor and keep_prob. But then in line 156 you say "gamma = 0.4". I am assuming you mean "keep_prob = 0.4"? Similarly, in table 2 / line 210, you use gamma, but I think you mean keep_prob.  How do you compensate for DropBlock? E.g. in dropout, if a feature is dropped with probability gamma, all other outputs are multiplied with 1/(1-gamma). How do you handle this in DropBlock?  Have you tried block_size_factor=1, i.e. dropping entire channels? What happens then?  ### Additional comments post-rebuttal ###  I apologize for the stupid comment regarding the imagenet test set. I didn't know that it is "secret".   "We are happy to create a holdout partition of training data and re-do the experiments for the final submission of 33 the paper." I am envious of your seemingly unlimited GPU access :) But seriously, I wouldn't want to burden you with this is everyone else if using the validation set for final numbers. I will leave it up to you whether you think having a test partition would improve the paper.  The block_size_factor=1 results are interesting. Maybe you can find space to include them in the final paper. Also, you write "we first estimate the block_size_factor to use (typically 0.5)". I'd like to see more details on how this estimation happened. Also, I can't find drop_block_size for figure 4 (it's probably 0.5 ...).   In figure 4, right side, "after sum" is missing an 'm'.