Faster Online Learning of Optimal Threshold for Consistent F-measure Optimization
In this paper, we consider online F-measure optimization (OFO). Unlike traditional performance metrics (e.g., classification error rate), F-measure is nondecomposable over training examples and is a non-convex function of model parameters, making it much more difficult to be optimized in an online fashion. Most existing results of OFO usually suffer from high memory/computational costs and/or lack statistical consistency guarantee for optimizing F-measure at the population level. To advance OFO, we propose an efficient online algorithm based on simultaneously learning a posterior probability of class and learning an optimal threshold by minimizing a stochastic strongly convex function with unknown strong convexity parameter. A key component of the proposed method is a novel stochastic algorithm with low memory and computational costs, which can enjoy a convergence rate of O(1/ √ n) for learning the optimal threshold under a mild condition on the convergence of the posterior probability, where n is the number of processed examples. It is provably faster than its predecessor based on a heuristic for updating the threshold. The experiments verify the efficiency of the proposed algorithm in comparison with state-of-the-art OFO algorithms.

The paper proposes a stochastic/online algorithm for optimizing the F-measure. Previous stochastic approaches for F-measure optimization either come with guarantees of rates of convergence to the optimal value of a 'surrogate' objective, or come with asymptotic consistency guarantees for the 'original' F-measure. In this paper, the authors build on ideas from Busa-Fekete et al. (2015) and develop an algorithm which has better empirical performance than previous methods, while enjoying (non-asymptotic) rates of convergence to the optimal F-measure value.  The proposed algorithm is based on the observation that the optimal classifier for the F-measure can be obtained by thresholding the posterior class probability at a suitable point, and performs simultaneous stochastic updates on the posterior class probability model and on the threshold. While Busa-Fekete et al. pose the threshold selection problem as a root finding problem, the present paper formulates this problem as a strongly convex optimization problem. This allows the authors to show that under very specific assumptions on the posterior class probability model and on the algorithm for learning the model, the proposed approach has a O(1/\sqrt{n}) convergence rate to the optimal F-measure value.   Pros: - The idea of posing the threshold selection problem as a strongly convex optimization problem is indeed interesting. - The proposed algorithm seems to empirically out-perform the state-of-the-art algorithms for F-measure, achieving higher or comparable F-measure values at a faster rate.  Cons: However, the convergence/consistency guarantee that the authors provide seems to be under strong assumptions on the posterior class probability distribution and on the algorithm used to learn the posterior probability model. In particular, the authors assume/state:  - The posterior class probability distribution can be parametrized as a generalized linear model  - The expected logistic loss computed on the posterior probability model is (globally) *strongly convex* - There exists an online algorithm that enjoys *high probability convergence rates* to the optimal parameters w^* of the posterior prob. model, i.e. can provides iterates w_1,\ldots, w_T such that for each ’t’, with high probability, ||w_t - w^*|| \leq C’ / t, for a problem-independent constant C’  (of these the second statement is not an assumption, but stated in ll. 217)  I find the above to be problematic:  Firstly, the assumption that the posterior class probability distribution takes a very specific parametrized form seems restrictive, especially because the analysis gives no room for model misspecification. I understand the analysis might also extend to slightly more general posterior probability distributions that can be written as a  sigmoid of a score function, but even this is a special case, and excludes a whole range of real-world distributions.  Secondly, the authors’ statement that the expected logistic loss is (globally) strongly convex is not true in general. The authors cite Agarwal et al. (2012) [1] , but even there it is only shown that the expected logistic loss is locally strongly convex (p.6) where the strong convexity parameter depends on the marginal distribution over ‘x’ (in particular on the min. Eigen value of the covariance matrix of ‘x’).  Finally, in the absence of (global) strong convexity, I do not think (as the authors claim) the algorithm of Agarwal et al. (2012) [15] can be applied to provide high probability convergence guarantees on individual model vectors. Indeed there have been some works (e.g. Bach et al., "Adaptivity of Averaged Stochastic Gradient Descent to Local Strong Convexity for Logistic Regression”, JMLR’14) that analyze SGD specifically for the logistic loss, but seem to provide weaker “expected” guarantees (and where C’ would be distribution-dependent)  (also please see pp. 2, end of paragraph 2 in Bach (2014) for a discussion on strong convexity of logistic loss)  It would be good if the authors are able to provide a reference to a specific algorithm that can satisfy the assumptions they make.  I do understand that the previous work of Busa-Fekete et al. also makes assumptions on the class probability learner, but their assumptions was on the class probability estimates \eta(x) and in particular on the expected absolute difference between the current class probability estimates and the true class probabilities (i.e. on E[|\eta(x) - \hat{\eta}_t(x)|]). The present paper makes stronger assumptions, namely a specific parametric form for the underlying class probability model, and high probability rates of convergence on the model parameters.  Having said this, I would also like to mention that I am sympathetic to this line of work and can see why it is challenging to provide non-asymptotic guarantees for F-measure optimization (i.e. on the original F-measure and not a surrogate). The strongly convex re-formulation that the authors propose for threshold search is definitely interesting, and their empirical results indicate that this formulation can be beneficial in practice. However, the theoretical guarantees provided seem to be under restrictive assumptions.   Other Comments: - Experiments: -- Please report F-measure performance as a function of running time rather than *iteration count*. Since an iteration may constitute different computations for different methods, it is only fair that you compare the methods against their running time. Indeed you do include run-time plots in the supplementary material, but please do have them in the main text. -- I see a large gap between FOFO and OFO for some of the data sets. Do you find that when you run for enough iterations eventually, both FOFO and OFO converge to the same threshold? How does the learned threshold compare with an offline/brute-force threshold tuning procedure? -- I see that the STAMP method you compare against has an epoch size parameter. How do you tune this parameter? Is there a specific reason you chose the number of classifiers in OMCSL to 10?  - Remark on line 232: You mention that Theorem 2 (convergence of inner SFO) can be used to derive a O(1/\sqrt{n}) convergence result for FIFO. I don’t see immediately how you would get this result (are you referring to having only one run of SFO with T = n?). In particular, Theorem 2 holds under the assumption that |\theta_1 - \theta^*| \leq R. This is true for the first invocation of SFO, but for subsequent invocations (k > 2), where the radius R is multiplicatively reduced, how do you make sure that this condition continues to hold?  - Same sample used for learning probability model and threshold: It seems to me that in Algorithm 2, the same sample is used by for both updating the posterior model parameters w_t (line 15) and updating the threshold \theta_t (line 13). I wonder if this would introduce an additional probabilistic dependence in your analysis. My guess is that a dependence is avoided by first updating the threshold and then the posterior model.  Minor comments: - Definition 1: Might be good to explicitly mention that the probability is over random draw of n samples - Lemma 2: It seem from the proof in the supplementary material that c = 2/\pi for all \theta. May want to just mention this in the lemma. - Theorem 3: SFO -> FOFO, also should ‘m’ be a superscript on \theta (looking at Algorithm 1)  ------- POST REBUTTAL ------- ------------------------------------ I thank the authors for the rebuttal. Having read their response and heard the views of the other reviewers, I'm increasing my score to 7.  I would however recommend that the authors:  (1) restate assumption 1 in terms of a condition on the estimated class probabilities \hat{\eta}_t (rather than making a specific parameterized assumption on the posterior distribution) (2) give examples where this assumption holds (one of which is when the expected loss is strongly convex) (3) clarify up-front in the paper that their result holds under a ‘point-wise’ convergence assumption and that it differs from previous works (currently the authors just mention that they need a ‘mild’ condition)  If one were to just assume that the estimated probabilities converge in a weaker expected sense (e.g. the assumption in [3[), would your algorithm at least have an asymptotic convergence guarantee like [3]? If so, I would appreciate it if you could add a discussion on this.


Weakly Supervised Dense Event Captioning in Videos
Dense event captioning aims to detect and describe all events of interest contained in a video. Despite the advanced development in this area, existing methods tackle this task by making use of dense temporal annotations, which is dramatically source-consuming. This paper formulates a new problem: weakly supervised dense event captioning, which does not require temporal segment annotations for model training. Our solution is based on the one-to-one correspondence assumption, each caption describes one temporal segment, and each temporal segment has one caption, which holds in current benchmark datasets and most real-world cases. We decompose the problem into a pair of dual problems: event captioning and sentence localization and present a cycle system to train our model. Extensive experimental results are provided to demonstrate the ability of our model on both dense event captioning and sentence localization in videos.

This paper addresses the problem of weakly-supervised dense event captioning (WS-DEC), where there are only video captioning labels without temporal segmentation labels. Utilizing the one-to-one correspondence assumption, it proposes a new iterative formulation for WS-DEC that alternates between caption generation and sentence localization. The proposed algorithm is evaluated on ActivityNet Captions benchmark.    + This paper tackles the weakly-supervised dense event captioning (WS-DEC), which extends the existing DEC problem by excluding the requirement of temporal segmentation labels. This WS-DEC task has not been discussed in previous research.   + The problem formulation seems well-defined and effective (section 3.1). The proposed approach is designed by alternating two subproblems, sentence localization and event captioning, both of which are formulated to be mathematically clear.   + The paper reads very well.    1. I am not sure the term WS-DEC is correct for the problem of this paper.  - While the DEC formulation requires two types of labels for sentences and temporal localization, the proposed WS-DEC exploits only sentence labels. Obviously, sentence labels are much more informative than localization labels (only two numbers ranged in [0,1]).   2. The proposition 1 (Fixed-point-iteration) is dubiously and invalidly applied to the WS-DEC formulation. - First of all, citation is required for the proposition 1. I don’t think the convergence proof of fixed-point algorithm with Lipschitz continuity is on authors’ contribution.  I cannot any authors’ novelty here. It is a simple re-utterance of the existing proposition with only new notations for the functions in Eq.(1)-(2). - The functions that this paper uses are not Lipschitz continuous (Line 137); thus, this proposition is not applicable to the problem formulation of this paper.  - Moreover, this paper argues that only a single-round iteration is enough to get a reasonable solution empirically (Line 140), which may hint that the fixed point-iteration is neither necessary nor sufficient for the problem. - In summary of my opinion, this proposition could be mentioned as a background rationale for the proposed iterative algorithm, but strictly speaking it does not seem to logically or mathematically connected to the formulation of WS-DEC.  3. Some analyses on experimental evaluation seems hasty and defective. - As shown in Table 1 and 2, the proposed unsupervised algorithm works better than some initial supervised models (e.g. [10, 28]) but underperforms the best-performing methods (e.g. [11.29]). This tendency should be explicitly mentioned in the paper.   - I do not think the statement in Line 276-277 is correct. The supervised implementation works worsen than ABLR with nontrivial margins.   4. Minor comments - How can we set the N_r (Line 130), which seems critical for the final output? - What is the rationale behind choosing perplexity loss for text reconstruction? (Line 185) It is for performance or implementation easiness?


Multiple Instance Learning for Efficient Sequential Data Classification on Resource-constrained Devices
We study the problem of fast and efficient classification of sequential data (such as time-series) on tiny devices, which is critical for various IoT related applications like audio keyword detection or gesture detection. Such tasks are cast as a standard classification task by sliding windows over the data stream to construct data points. Deploying such classification modules on tiny devices is challenging as predictions over sliding windows of data need to be invoked continuously at a high frequency. Each such predictor instance in itself is expensive as it evaluates large models over long windows of data. In this paper, we address this challenge by exploiting the following two observations about classification tasks arising in typical IoT related applications: (a) the "signature" of a particular class (e.g. an audio keyword) typically occupies a small fraction of the overall data, and (b) class signatures tend to be discernible early on in the data. We propose a method, EMI-RNN, that exploits these observations by using a multiple instance learning formulation along with an early prediction technique to learn a model that achieves better accuracy compared to baseline models, while simultaneously reducing computation by a large fraction. For instance, on a gesture detection benchmark [26], EMI-RNN requires 72x less computation than standard LSTM while improving accuracy by 1%. This enables us to deploy such models for continuous real-time prediction on devices as small as a Raspberry Pi0 and Arduino variants, a task that the baseline LSTM could not achieve. Finally, we also provide an analysis of our multiple instance learning algorithm in a simple setting and show that the proposed algorithm converges to the global optima at a linear rate, one of the first such result in this domain. The code for EMI-RNN is available at [14].

The authors propose a method for fast and efficient classification of sequential data, motivated by the deployment of such methods on embedded or resource-constrained devices. The method has two key components: (1) it is formulated as a multiple instance learning problem, where each time series is broken down into shorter ones and labels are generated automatically for them, and (2) an early classification formulation that allows the network to emit outputs before processing the whole input sequence. The model is evaluated on five datasets; when compared to the standard LSTM, the proposed solution achieves similar accuracy (sometimes better, sometimes worse) while providing important computational savings.   PROS - Efficient model for sequence classification is specific tasks, like wake word detection in voice-driven assistans (e.g. Google Home or Alexa). Speed and low-consumption ML models are key to the development of such systems. - Results on the evaluated datasets are good, as the model can provide computational savings while matching or even improving the accuracy of the baseline LSTM. - The authors show the advantages of the method in resource-constrained devices, namely Raspberry Pi0 and 3.  CONS - The model looks somewhat niche, as it seems very tailored for the tasks in the paper. Nevertheless, these tasks are important and interesting to the community. - Although the text is generally well written and motivated, some parts are hard to follow and lack some detail. The manuscript needs a final discussion/conclusion section as well. Please see more extensive comments below.   TL;DR: The paper addresses an interesting task and provides motivation for most choices. Some parts of the experimental setup need to be clarified so that readers can understand the contributions more easily and reproduce the results.    COMMENTS  - The model introduces some new hyperparameters. How sensitive is it to the particular choice of these parameters? Please comment on this, as it is very helpful for practitioners trying to replicate/extend this work. - Equation 3.1.1 is very hard to follow, as most variables have not been defined in the text (e.g. s_i or n). Please clarify this part of the manuscript, it is one of the main contributions of the paper. - What is “train-LSTM”? One SGD step, or a full training? If it is a full training, how many times is this process repeated before convergence?  - The description of the evaluation protocol could be more thorough. Why can MI-RNN provide savings when it needs to evaluate some tokens in the time series more than once when the windows overlap, as shown in Figure 1?  - There is no description about how the method was implemented in Raspberry Pi. Did the authors use some particular framework? - Please cite the following related paper: Ma et al., “Learning Activity Progression in LSTMs for Activity Detection and Early Detection”, CVPR 2016 - The format of the references section is not consistent. Please select between full conference name or acronym (e.g. [2] or [3]), follow a common format for arxiv pre-prints (e.g. [10] and [31]). Some pre-prints have already been published (e.g. [7]). - Do not attach the main paper with the supplementary material. - There are some typos in the text, please proof-read:    11: tend to (be) discernible    33: predict (the) label    58: We [we]    69: it’s -> its    79: missing comma before “etc”    88: please do not use “etc” and list all the tasks    118: missing whitespace    124-125: use consistent format (with or without comma) for x_{i,T}    144: “models process”    199: are there the same number of positives and negatives (n) in Z, or is this a typo?    213: “noise the bags”?    Figure 4: this is a table and should be labeled accordingly   ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------  The results in the paper are good, but I had concerns regarding clarity and reproducibility. The author feedback helped with this, and I believe that the paper will be better once the authors introduce these clarifications in the manuscript. They will also release code so that other can easily reproduce their experiments. After reading the author feedback, I would lean towards acceptance. I increased the overall and reproducibility scores accordingly.


Heterogeneous Bitwidth Binarization in Convolutional Neural Networks
Recent work has shown that fast, compact low-bitwidth neural networks can be surprisingly accurate. These networks use homogeneous binarization: all parameters in each layer or (more commonly) the whole model have the same low bitwidth (e.g., 2 bits). However, modern hardware allows efficient designs where each arithmetic instruction can have a custom bitwidth, motivating heterogeneous binarization, where every parameter in the network may have a different bitwidth. In this paper, we show that it is feasible and useful to select bitwidths at the parameter granularity during training. For instance a heterogeneously quantized version of modern networks such as AlexNet and MobileNet, with the right mix of 1-, 2-and 3-bit parameters that average to just 1.4 bits can equal the accuracy of homogeneous 2-bit versions of these networks. Further, we provide analyses to show that the heterogeneously binarized systems yield FPGA-and ASIC-based implementations that are correspondingly more efficient in both circuit area and energy efficiency than their homogeneous counterparts.

The paper makes an observation that in general 2-bit models are more accurate than 1-bit model. The goal in this paper is to make the performance of lower-bit models closer to than of without (significantly) compromising on their efficiency. The paper achieves this by binarizing to a mixture of bitwidths as opposed to the traditional approach of a uniform bitwidth to all values. In particular, it extends the approach presented in Tang et al. (2017) that homogeneously binarizes to n bits. The paper proposes to examine individual values and decide whether to binarize them (using the Mask Tensor M). The paper proposes an algorithm to achieve an average bidwidth of B (non-integer) by taking as input a targeted distribution over individual bitwidths, P. In order to achieve this target bitwidth distribution, the algorithm greedily assigns low bitwidths to values that are closer to average values and so are likely to be accurate without employing additional bits and vice versa. The experiments are conducted using AlexNet and MobileNet using ImageNet data.  The paper is well-written and easy to follow. The method is somewhat of a simple extension of the idea presented in previous work but the end-results are promising.   I understand that the algorithm not only takes the desired average number of bits (B) as input but also the distribution over individual bitwidths (P). First of all, this is not clear in Algorithm 1, and should be explained. From the algorithm's description it seems that the only input is B (apart from other things).   How sensitive is the algorithm to the choice of P? Ideally, the authors should have shown the variation in performance for different desired distributions (all yielding the same value of average bitwidth B). However, I am willing to listen to authors' opinion on this. Also, are there any recommendations/heuristics to choose a good P?  Minor comment: the sentence in line 26 is difficult to read.  I have read the author response. Thanks for your clarifications. Please include them in your next version.


Unsupervised Learning of Object Landmarks through Conditional Image Generation
We propose a method for learning landmark detectors for visual objects (such as the eyes and the nose in a face) without any manual supervision. We cast this as the problem of generating images that combine the appearance of the object as seen in a first example image with the geometry of the object as seen in a second example image, where the two examples differ by a viewpoint change and/or an object deformation. In order to factorize appearance and geometry, we introduce a tight bottleneck in the geometry-extraction process that selects and distils geometryrelated features. Compared to standard image generation problems, which often use generative adversarial networks, our generation task is conditioned on both appearance and geometry and thus is significantly less ambiguous, to the point that adopting a simple perceptual loss formulation is sufficient. We demonstrate that our approach can learn object landmarks from synthetic image deformations or videos, all without manual supervision, while outperforming state-of-the-art unsupervised landmark detectors. We further show that our method is applicable to a large variety of datasets -faces, people, 3D objects, and digits -without any modifications.

Summary: This paper proposes a method for conditional image generation by jointly learning "structure" points such as face and body landmarks. The authors propose to use a convolutional neural network with a modified loss to capture the image transformation and landmarks. They evaluate their approach on a set of datasets including CelebA, VoxCeleb, and Human 3.6M.  Positive:  -The problem addressed is an important problem and the authors attempt to solve it using a well engineered approach. -The results seem to be competitive to some state of the art approaches.  Negatives: -The pre-processing using heat maps, normalizing them into probabilities, then using a gaussian kernel to produce the features is a bit heuristic.  -The claim in P4 L132-135 saying this is usually done using GANs, it would have been good to show some results and comparison against GAN based methods. Comparison against [37] for image generation using landmark/control/structure points would have helped. -The use of two networks, landmark network and image generation network, rather than training a joint network seems inefficient.  Landmarks and image generation are two complimentary that could be modeled in a joint network.


Experimental Design for Cost-Aware Learning of Causal Graphs
We consider the minimum cost intervention design problem: Given the essential graph of a causal graph and a cost to intervene on a variable, identify the set of interventions with minimum total cost that can learn any causal graph with the given essential graph. We first show that this problem is NP-hard. We then prove that we can achieve a constant factor approximation to this problem with a greedy algorithm. We then constrain the sparsity of each intervention. We develop an algorithm that returns an intervention design that is nearly optimal in terms of size for sparse graphs with sparse interventions and we discuss how to use it when there are costs on the vertices.

Summary: The authors study the problem of designing experiments for learning causal structure under the assumption that each variable has a cost to intervene on, and the objective is to minimize the total experiment cost. It is first shown that the problem is in general NP hard, and then a greedy algorithm is proposed which is proven to be an approximate algorithm. Furthermore, the case that in each intervention, a limited number of variables can be intervened on is considered. An approximate algorithm is proposed for this case as well. Finally, the proposed approach is evaluated on synthetic data. The work for most of the part, is written clearly and is easy to understand.  Comments: - In line 69, the vice versa part does not seem to be necessarily true.  - In line 102, when we force a variable to just take a certain value, it may not necessarily change the distribution of its children. Also, what does the authors mean by "new skeleton" in line 105?  - In Definition 2, the minimum cost intervention design problem is defined as finding a graph separating system of size at most m. What if there is no such separating system? I recommend changing it to minimum possible size, or m>m^*, where m^* is the minimum possible size.  - In Definition 4, it seems that in reality we always need to have |I_i|=k. If so, can we say that Definition 5 is a special case of Definition 6?  - How should we choose the cost values in practice? Do the authors recommend that these values should be defined manually by an expert? If so, this restricts the application of the proposed approach.  - A major concern as usual regarding this work and similar work is its practicality aspects. It is not clear if the proposed method can be applied in practice, where there are latent confounders, statistical errors, undefined cost values, etc.  - It seems that in reality the values of the costs may not follow a distribution similar to the design of experiments. I expect much larger costs following a heavy tailed distribution, or even more realistically, many variables should have cost equal to infinity, representing that intervention on those variables are not possible. Especially the latter situation is not considered in this work.  - The cost quantization step seems unjustifiable in practice and seems to be only useful for theoretical guarantees to work. Clearly, this step can possibly change the significance of costs and make the distribution of costs close to uniform.  - One important missing experiment in the last section is evaluating the approach on graphs with different sparsities.   - Also, it would have been more useful if the authors performed simulations with the initial observational test step, so that we can see the effect of that step on the total performance as well. This will include error in estimating the skeleton, not having chordal graph, etc.  - The proof of proposition 10 in the supplementary materials is not clear. Especially, it is not clear why S should be an independent set. This is important as Theorem 11 is also using this proposition.  - One question regarding the presentation of the work: the case of restricting the number of intervened variables in each intervention is the general case, and the whole section 5 could be considered as a special case of that. The same is true for definition. I was wondering if there was a special reason that the authors did not merge these results and definitions?


Temporal Regularization in Markov Decision Process
Several applications of Reinforcement Learning suffer from instability due to high variance. This is especially prevalent in high dimensional domains. Regularization is a commonly used technique in machine learning to reduce variance, at the cost of introducing some bias. Most existing regularization techniques focus on spatial (perceptual) regularization. Yet in reinforcement learning, due to the nature of the Bellman equation, there is an opportunity to also exploit temporal regularization based on smoothness in value estimates over trajectories. This paper explores a class of methods for temporal regularization. We formally characterize the bias induced by this technique using Markov chain concepts. We illustrate the various characteristics of temporal regularization via a sequence of simple discrete and continuous MDPs, and show that the technique provides improvement even in high-dimensional Atari games.

This paper is very interesting. One previous assumption in TD learning is that reward are close with states in proximity of the state space, which has been pointed out by many papers is not realistic and have problems for spatial value function regularization. Instead, this paper make the assumption that rewards are close for states.  Overall this paper has a very good motivation, and the literature review shows that the author is knowledgable of this field. This paper could open a novel area of temporal regularization that received inadequate attention before. Here are my major concerns: 1. The author tries to squeeze too much content to show the effectiveness of the proposed method. For example, the experiments covers both value function approximation and control. To me, the effect of control by the improved value function approximation is indirect and less clear than showing results of policy evaluation. I would suggest the author to remove (or put it to the appendix) the control part (such as line 137-140, and section 5.6) and focus more on the value approximation part. 2. The comparison study of different temporal regularization methods is incomplete. As the author pointed out in line 128, different temporal regularization methods could be used. Then the comparison study of different methods are completely missing in the paper, which makes the paper incomplete from this perspective. 3 connections with previous work: This temporal regularization will serve a very similar purpose as Residual gradient (RG) vs. TD on backward bootstrapping, where the value is affected by both the previous state and the next state. The benefit is stability and the disadvantage of deteriorated prediction. This has been investigated in several previous papers, such as  Sutton, R. S., Maei, H. R., Precup, D., Bhatnagar, S., Silver, D., Szepesvári, C., & Wiewiora, E. (2009, June). Fast gradient-descent methods for temporal-difference learning with linear function approximation. In Proceedings of the 26th Annual International Conference on Machine Learning (pp. 993-1000). ACM. (detailed discussed backward bootstrapping)  Li, L. (2008, July). A worst-case comparison between temporal difference and residual gradient with linear function approximation. In Proceedings of the 25th international conference on machine learning (pp. 560-567). ACM. (compare the prediction and stability of TD and RG, though not explicitly bridging the connections with backward bootstrapping)   Some minor suggestions: line 56-67: it is much better to give reasons and toy examples why reversal Markov chain property (Def 2) is less violated in practice than Detailed balance (Def. 1). line 110& 122. It is better to change ‘Property 1’ —> ‘Proposition 1’. line 124 Does this imply the average reward \rho is more robust than the value function V? what else does it imply here? line 135 reference for ’trend-ajusted model’ is missing. === I have read the response, and have updated my reviews.


Efficient inference for time-varying behavior during learning
The process of learning new behaviors over time is a problem of great interest in both neuroscience and artificial intelligence. However, most standard analyses of animal training data either treat behavior as fixed or track only coarse performance statistics (e.g., accuracy, bias), providing limited insight into the evolution of the policies governing behavior. To overcome these limitations, we propose a dynamic psychophysical model that efficiently tracks trial-to-trial changes in behavior over the course of training. Our model consists of a dynamic logistic regression model, parametrized by a set of time-varying weights that express dependence on sensory stimuli as well as task-irrelevant covariates, such as stimulus, choice, and answer history. Our implementation scales to large behavioral datasets, allowing us to infer 500K parameters (e.g., 10 weights over 50K trials) in minutes on a desktop computer. We optimize hyperparameters governing how rapidly each weight evolves over time using the decoupled Laplace approximation, an efficient method for maximizing marginal likelihood in non-conjugate models. To illustrate performance, we apply our method to psychophysical data from both rats and human subjects learning a delayed sensory discrimination task. The model successfully tracks the psychophysical weights of rats over the course of training, capturing day-to-day and trial-to-trial fluctuations that underlie changes in performance, choice bias, and dependencies on task history. Finally, we investigate why rats frequently make mistakes on easy trials, and suggest that apparent lapses can be explained by sub-optimal weighting of known task covariates.

The paper presents a new estimator for a dynamic logistic regression model used to characterize human or animal behavior in the context of binary decisions tasks. Despite the very large number of parameters such a model can be estimated robustly and with tolerable computational cost by exploiting the structure of the posterior covariance. The estimator is then applied to a delayed auditory  discrimination task in humans and animals. The results show signatures of learning and history interference in the rodent compared to human. Additionally the model fit manages to predict behavior in test data. The model is simple and interpretable which will likely make the use of the tools popular within the community.  Comments: - the choice of prior for w is a bit odd - since the variance of the marginal for w_t grows linearly with t; Also the fact that the noise is independent seems somewhat limiting - how about things like fluctuation in attention those would likely affect the processing of both stimuli and potentially even some of the history... - it would be nice if you would directly compare the dynamics of w with the dynamics of learning, as measured behaviorally; the prediction estimates in fig 3 goes into that direction but it would be interesting to see a direct visualization of the two.  - similarly do the fluctuations in ws predict fluctuations in performance in humans? - it wasn't clear if the code associated will be released with the paper  Minor: - unspecified w_0: I am assuming w_0 = 0  L should be defined somewhere explicitly in eq 3 - unclear what 'behavioral realization' is supposed to mean in relation to figure 2: are individual lines MAP estimates for w, where the true w is kept fixed but the observation noise is redrawn independently? And are the hyperparameters estimated in panel b the same way or are the ws redrawn for each experiment? -at some point the text mentions sparse priors for w which could be misleading - it's just the prior covariance matrix is sparse.  Post rebuttal: Thanks for the clarifications. I still the prior should be justified by more than mathematical convenience but it's ok as long as the text justifies it accordingly.


Fairness Behind a Veil of Ignorance: A Welfare Analysis for Automated Decision Making
We draw attention to an important, yet largely overlooked aspect of evaluating fairness for automated decision making systems-namely risk and welfare considerations. Our proposed family of measures corresponds to the long-established formulations of cardinal social welfare in economics, and is justified by the Rawlsian conception of fairness behind a veil of ignorance. The convex formulation of our welfare-based measures of fairness allows us to integrate them as a constraint into any convex loss minimization pipeline. Our empirical analysis reveals interesting trade-offs between our proposal and (a) prediction accuracy, (b) group discrimination, and (c) Dwork et al.'s notion of individual fairness. Furthermore and perhaps most importantly, our work provides both heuristic justification and empirical evidence suggesting that a lower-bound on our measures often leads to bounded inequality in algorithmic outcomes; hence presenting the first computationally feasible mechanism for bounding individual-level inequality.

This paper proposes a new measure of fairness for classification and regression problems based on welfare considerations rather than inequality considerations.  This measure of fairness represents a convex constraint, making it easy to optimize for.  They experimentally demonstrate the tradeoffs between this notion of fairness and previous notions.  I believe this to be a pretty valuable submission.  A welfare-based approach over a inequality-based approach should turn out to be very helpful in addressing all sorts of concerns with the current literature.  It also provokes a number of questions to follow up on which, while disappointing that they are not addressed here, means that the community should take interest in this paper.  I am not at all convinced that this is really an 'individual' notion of fairness.  The requirement is to ensure only that the sum of individual utilities is sufficiently high, rather than enforcing that every utility is high (as would be the case if you forced \alpha=-\infty), or something similar.  Thus I think it is not particularly appropriate to call this the "first computationally feasible mechanism for bounding individual-level (un)fairness."  While on the subject of the abstract, the abstract states "our work provides both theoretical and empirical evidence suggesting that a lower-bound on our measures often leads to bounded inequality in algorithmic outcomes."  Empirical evidence is certainly provided, but to what theoretical evidence does this statement refer?  The connection to Atkinson's index?  I ended up waiting for generalization results that didn't exist in this paper, so that could be cleared up.   A few very small things: - It looks like there's overlapping notation between Pareto domination of the benefits and the welfare's total ordering (line 206) - "consistently" --> "consistent" (line 271) - Figure 2 has "accuracy" where the Dwork measure should be instead


Unsupervised Video Object Segmentation for Deep Reinforcement Learning
We present a new technique for deep reinforcement learning that automatically detects moving objects and uses the relevant information for action selection. The detection of moving objects is done in an unsupervised way by exploiting structure from motion. Instead of directly learning a policy from raw images, the agent first learns to detect and segment moving objects by exploiting flow information in video sequences. The learned representation is then used to focus the policy of the agent on the moving objects. Over time, the agent identifies which objects are critical for decision making and gradually builds a policy based on relevant moving objects. This approach, which we call Motion-Oriented REinforcement Learning (MOREL), is demonstrated on a suite of Atari games where the ability to detect moving objects reduces the amount of interaction needed with the environment to obtain a good policy. Furthermore, the resulting policy is more interpretable than policies that directly map images to actions or values with a black box neural network. We can gain insight into the policy by inspecting the segmentation and motion of each object detected by the agent. This allows practitioners to confirm whether a policy is making decisions based on sensible information. Our code is available at https://github.com/vik-goel/MOREL.

************************* Summary: ************************* This paper describes MOREL,  an approach for automatically segmenting objects for reinforcement learning. In particular, this work uses SfM-Net [1], which learns to predict optical flow of a single image, to segment the objects in a state, and then uses this object-mask for reinforcement learning. MOREL is evaluated on all 59 Atari games, where it outperforms the baselines in several environments.   ************************* Quality: ************************* This approach appears to be sound. Additionally, it is evaluated within several environments. The paper does not have enough discussion about weaknesses of the approach. Additionally, while there is a clear advantage of using MOREL over the baseline approaches, it would be better to have more than three averaged results.  ************************* Clarity: ************************* This paper is well-written. I did not have any problems understanding the approach or motivation. I did not even find any typos. I do have a couple of comments, though. In the background, the policy is defined w.r.t. histories, but this is not a common way to formulate policies. It is unclear why it is defined in this way in the paper. Additionally, r is never defined.   ************************* Originality: ************************* The main contribution of this work is using SfM-Net to segment images for reinforcement learning. This is similar to feature extraction, and other similar approaches should have been discussed or compared to. In general, the related work lacks discussion of related feature construction and computer vision techniques. Atari is simple and the features could be constructed using very basic computer vision techniques. In fact, Bellemare et al. [2] list a many ways to extract features for Atari, for example by using background subtraction. Additionally, DARLA [3] uses Beta-VAE to learn disentangled features, such as objects, for reinforcement learning, and deep spatial auto-encoders [4] also learn features that encode information about objects.  ************************* Significance: ************************* This work introduces an interesting way to segment objects for reinforcement learning. Vijayanarasimhan et al [1] demonstrate that Sfm-net can work on realistic environments, but it would still be useful to also evaluate MOREL within less trivial environments, particularly because the backgrounds in Atari games are stationary. This would make it clear that it is beneficial to segment objects. For example, do the masks even need to be learned? Can the objects be detected by computing optical flow of the previous and current frames or background subtraction? Can pre-trained features be used?  Additionally, the paper described few changes to SfM-Net. It would have been nice to see results for the unchanged network.  The main changes were a different loss for reconstruction, flow regularization, and a curriculum. These changes are incremental, and so the segmentation results from Figure 3 seem more like result from running SfM-Net on Atari, rather than any contribution from this work. This statement could be argued against by demonstrating that the adjustments to SfM-Net yield different or better results than the unchanged network.  Nevertheless, the approach performs well against the baselines, and I do believe it could be a useful approach since it would likely make it easier to learn about salient objects.   ************************* Comments ************************* The results do not demonstrate that this representation is more interpretable. Detecting objects does not give any insight into the decisions being made by the policy. Using attention might change this. Line 234: I disagree that the segmentation is remarkable since the approach is only evaluated on Atari. Additionally, the model is self-supervised. It is given optical flow and thus predicts optical flow. So the result does not seem too surprising. Why should we assume that moving objects are important?  ************************* Typos ************************* None, well done!  ************************* References: ************************* [1] Vijayanarasimhan, Sudheendra, et al. "Sfm-net: Learning of structure and motion from video." arXiv preprint arXiv:1704.07804 (2017). [2] Bellemare, Marc G., et al. "The arcade learning environment: An evaluation platform for general agents." Journal of Artificial Intelligence Research 47 (2013): 253-279. [3] Higgins, Irina, et al. "Darla: Improving zero-shot transfer in reinforcement learning." arXiv preprint arXiv:1707.08475 (2017). [4] Finn, Chelsea, et al. "Deep spatial autoencoders for visuomotor learning." arXiv preprint arXiv:1509.06113 (2015).  Update: I have read the review's and author's response. While I appreciate the rigorous studies done in the atari environment, I still believe the paper would be greatly improved if it demonstrated the benefits of the approach in environments with more difficult segmentation. This could still be done in simulation. As such, my score remains the same.


Trading robust representations for sample complexity through self-supervised visual experience
Learning in small sample regimes is among the most remarkable features of the human perceptual system. This ability is related to robustness to transformations, which is acquired through visual experience in the form of weak-or self-supervision during development. We explore the idea of allowing artificial systems to learn representations of visual stimuli through weak supervision prior to downstream supervised tasks. We introduce a novel loss function for representation learning using unlabeled image sets and video sequences, and experimentally demonstrate that these representations support one-shot learning and reduce the sample complexity of multiple recognition tasks. We establish the existence of a trade-off between the sizes of weakly supervised, automatically obtained from video sequences, and fully supervised data sets. Our results suggest that equivalence sets other than class labels, which are abundant in unlabeled visual experience, can be used for self-supervised learning of semantically relevant image embeddings.

This paper proposes a framework for representation learning that doesn't require any labeled data. Instead, priors on invariance to spatial transformations or local invariance in video is used to drive representation learning. This paper offers a lot of well thought out formalism and shows some promise on datasets like one-shot MNIST learning. The main downside is that MNIST is particularly suited for such transformations, and even going to a dataset like NORB, this clear edge over competing methods disappears. To me this is an accept because it gives me a lot ideas (and a fleshd out framework for expressing these ideas) and I think this will be true for the community in general.  Strengths. This paper offers a lot of thorough and precise formalism that are general and may find use in a variety of settings. The main idea of defining orbits is interesting. The experiments are thorough and interesting, and the paper is well written.  Weaknesses. It's hard to judge impact in real-world settings when most of the quantitative evaluations are on datasets not representative of complex natural images (e.g. MNIST and NORB). On MNIST, the method shows clear advantages over competing methods. However, even on NORB, where a lot of the deformations can't easily be parameterized, this advantage has turned into being only on par with other leading methods. I think the inclusion of the faces dataset  was important for this reason.  I was confused for a while what the exact orbit was for each dataset. I kept scanning the text for this. A table of all three datasets and a short note on how orbits were defined and canonical samples selected would make things a lot clearer.  Concurrent work. Similar ideas of representation learning through transformation priors have appeared in recent work. I don't think it takes away any novelty from this submission, since judging from the dates these is concurrent works. I just thought I would bring your attention to it: - https://openreview.net/pdf?id=S1v4N2l0- (ICLR 2018) - https://arxiv.org/pdf/1804.01552.pdf (CVPR 2018)  Minor comments. - eq. 6: what connects the orbit with this loss? I don't see the connection just yet - eq. 7: "x_q not in Oxq!=Oxi" What is this notation "set1 != set2" that seems to imply it forms another set (and not a true/false value)  line 136: Oxq \not= Oxi, again, I'm not sure about this notation. I understand what it means, but it looks odd to me. I have never seen this as part of set notation before. - eq. 8: where is x_p, x_q, x_c coming from? Shouldn't the summand be $(x_i, x_p, x_q) \in \mathcal{T}$? The canonical sample x_c is still unclear where it comes from. If x_c is the canonical instance for each orbit, then it also changes in the summation. This is not clear from the notation. - line 196: max unpooling transfers the argmax knowledge of maxpooling to the decoder. Do you use this behavior too? - Table 1: Should EX/NORB really be bold-faced? Is the diff between 0.59+/-0.12 really statistically significant from 0.58+/-0.11? - line 213: are all feature spaces well-suited for 1-NN? If a feature space is not close to a spherical Gaussian, it may perform poorly. If feature dimensions are individually standardized, it would avoid this issue. - It was a bit unclear how canonical samples were constructed on the face dataset ("least yaw displacement from a frontal pose"). This seems to require a lot of priors on faces and does not seem like purely unsupervised learning. Did the other competing methods require canonical examples to be designated?


Distributed Multi-Player Bandits -a Game of Thrones Approach
We consider a multi-armed bandit game where N players compete for K arms for T turns. Each player has different expected rewards for the arms, and the instantaneous rewards are independent and identically distributed. Performance is measured using the expected sum of regrets, compared to the optimal assignment of arms to players. We assume that each player only knows her actions and the reward she received each turn. Players cannot observe the actions of other players, and no communication between players is possible. We present a distributed algorithm and prove that it achieves an expected sum of regrets of near-O log 2 T . This is the first algorithm to achieve a poly-logarithmic regret in this fully distributed scenario. All other works have assumed that either all players have the same vector of expected rewards or that communication between players is possible.

This paper studies a distributed multi-armed bandits setting, where every round each of N players must choose one of K arms. If a player picks the same arm as some other player, they receive payoff zero; otherwise, they receive payoff drawn from some distribution (specific to that player and that arm). This can be thought of as learning a distributed allocation or matching from players to arms. The goal is to design a communication-free learning algorithm that maximizes the total overall utility of all the players (or alternatively, minimizes their regret with respect to the best fixed allocation).   The authors design an algorithm which receives total regret of O(log^2 T). Their algorithm can be thought of as a distributed "explore then exploit"-style algorithm (similar to e.g. epsilon-greedy). The algorithm begins with a phase where all players pick random arms to learn their individual distributions, and concludes with a phase where they all repeatedly pick what they have decided is the optimal allocation. The technically interesting part of the algorithm is the intermediate phase, where the arms coordinate amongst themselves (without communication, just by playing the game) to pick a good allocation where no arm is picked by more than one player (this they refer to as the "game of thrones"). They accomplish this by inducing a Markov process between the players of the game which converges to a state where all players are playing according to the optimal allocation. The authors also perform some simulations on synthetic data showing that this algorithm outperforms the case where all players are independently running UCB1.   This is a nice result. That said, it seems like the interesting part of this work is not really the learning component, but rather the coordination component -- in fact, the problem/solution still seems interesting even if all players know their exact expected rewards for each arm ahead of time. From a learning perspective, I would have preferred to see a UCB-style algorithm that explores and exploits at the same time (while somehow simultaneously coming to a consensus on what allocation to play).   Also, it seems important in the analysis that the distribution of rewards for an arm has no point mass at 0, since the way in which players estimate the expected reward for an arm is to just take the average over all non-zero pulls (zero-pulls are assumed to be collisions with other players). Is this necessary to the analysis? One can imagine natural settings where it is not possible to distinguish via the reward alone whether the reward is small because of the distribution or because other players are also playing that arm (either because the distribution has a point mass at 0 or because collisions don't decrease the reward all the way to 0). Does some variant of this algorithm work in this setting? (it is not obvious from the analysis in the paper).  The paper was well-written and easy to understand. I have not checked all the proofs in the supplementary materials, but the proof sketches appear sound.


A Lyapunov-based Approach to Safe Reinforcement Learning
In many real-world reinforcement learning (RL) problems, besides optimizing the main objective function, an agent must concurrently avoid violating a number of constraints. In particular, besides optimizing performance, it is crucial to guarantee the safety of an agent during training as well as deployment (e.g., a robot should avoid taking actions -exploratory or not -which irrevocably harm its hardware). To incorporate safety in RL, we derive algorithms under the framework of constrained Markov decision processes (CMDPs), an extension of the standard Markov decision processes (MDPs) augmented with constraints on expected cumulative costs. Our approach hinges on a novel Lyapunov method. We define and present a method for constructing Lyapunov functions, which provide an effective way to guarantee the global safety of a behavior policy during training via a set of local linear constraints. Leveraging these theoretical underpinnings, we show how to use the Lyapunov approach to systematically transform dynamic programming (DP) and RL algorithms into their safe counterparts. To illustrate their effectiveness, we evaluate these algorithms in several CMDP planning and decision-making tasks on a safety benchmark domain. Our results show that our proposed method significantly outperforms existing baselines in balancing constraint satisfaction and performance.

The paper focuses on the problem of safety in RL. The safety concept is expressed through a constraint on a cumulative cost. The authors leverage the framework of Constrained MDPs (CMDPs) to derive a novel approach where the Lyapunov function can be automatically constructed (instead of being manually defined). The core contribution is a method for constructing the Lyapunov function that guarantees the safety of the policy during training. In particular, safety can be efficiently expressed through a set of linear constraints. Leveraging this method, the authors showed how to modify several dynamic programming approaches to enforce safety. These algorithms do not have guarantees of convergence but the authors have empirically shown that they often learn near-optimal policies.  Comments ----------- The idea proposed in the article is very interesting. In particular, I believe that safety is an important topic to be explored in RL with a potentially high impact in real-world applications.  The introduction is well written and provides a nice overview of the literature about CMDPs.  On contrary, I believe that preliminaries can be made clearer. I think that it is necessary to state clearly the settings. It took me a while to figure it out that the considered scenario is the stochastic shortest path problem which is an instance of total cost problem. In particular, I was surprised by the fact that the Bellman (policy) operator is a contraction mapping (that is not in general true in undiscounted settings). After checking [Bertsekas, 1995] I realized that T is a contraction since, in your problem,  all stationary policies are proper [Bertsekas, 1995, Chap 2, Def 1.1, pag 80], that is, when termination is inevitable under all stationary policies. I suggest you to clearly state the settings and to put explicit references, e.g., in line 134 you should put page and prop/lem when references [7]. Am I correct? Moreover, while checking the proof in Sec C.3, I was not able to find the reference to Prop 3.3.1 and Prop 1.5.2 in [7]. Can you please provide page and or chapter? Please, also update the reference. Is it the volume 2?  I believe that the analysis can be generalized outside this settings. However, you should rely on different arguments since the Bellman operator is no more a contraction in generic undiscounted problems. Could you please comment on this? Is it possible to generalize to average cost problems?  Apart from that, I think that the presentation is fair and you acknowledge correctly strengths and weaknesses of the approach (e.g., the fact that algorithms do not enjoy optimality guarantees).  I was surprised by the fact that there is no comparison with safe methods that guarantee a monotonic policy performance improvement: e.g., in policy iteration [Kakade and Langford, 2002; Pirotta et al, 2013a] or policy gradient [Pirotta et al, 2013b, etc.]. Your approach is definitely different but, often, the constraint cost can be incorporated in the reward (it the case of your example).  The mentioned methods are guaranteed to learn policies that are always better than the baseline one. Then, if the baseline policy is good enough (satisfies the constraints reformulated in terms of bad reward) these approaches are guaranteed to recover safe policy over the entire learning process.  I think you should mention these works in the paper highlighting similarities and advantages of your approach. Please comment on this.  Please add a reference for the fact that the first hitting time is upper bounded by a finite quantity (you talk about standard notion).  I overall like the paper and I think it will be a good fit for NIPS subject to the fact that the authors will take into account the aforementioned comments about the presentation.  - Line 706: fnction -> function - Policy Distillation algorithm in page 24 contains a broken reference    [Bertsekas, 1995] D. P. Bertsekas, Dynamic Programming and Optimal Control, Vol. II. Athena Scientific ISBN: 1-886529-13-2 [Pirotta et al. 2013a] M Pirotta, M Restelli, A Pecorino, D Calandriello. Safe Policy Iteration, ICML 2013 [Pirotta et al. 2013b] M Pirotta, M Restelli, L Bascetta. Adaptive step-size for policy gradient methods, NIPS 2013  ------------------------ After feedback Thank you very much for the feedback. Now it is clear to me the difference between safe approaches and the proposed algorithm. I suggest you add this explanation to the paper. Hitting time: The assumption of boundness of the first hitting time is very restrictive. I underestimated it during the review because I was not sure about the meaning of the term "uniformly bounded". I checked the reference and uniform boundness of the first hitting time means that the time to reach the absorbing state is almost surely bounded for any stationary policy. I start noticing that due to the settings you considered, every stationary policy induces an absorbing Markov chain (by definition of proper policy, [Bertsekas, 1995, Chap 2, Def 1.1, pag 80]). For absorbing Markov chains it is known that the boundness of the first hitting time (with probability 1 the time to reach the absorbing state is bounded) implies that there are no loops in the chain (every state is visited at most once with probability 1) [see for example App. D of Fruit and Lazaric, Exploration-Exploitation in MDPs with Options, 2017]. You can assume this property but I would like to see this comment in the paper because I believe that the strong implication of this assumption is not well known in general. Moreover, you need to clearly distinguish the theoretical analysis from the empirical results because this assumption seems to be not satisfied in your experiments.


Learning Abstract Options
Building systems that autonomously create temporal abstractions from data is a key challenge in scaling learning and planning in reinforcement learning. One popular approach for addressing this challenge is the options framework [29]. However, only recently in [1] was a policy gradient theorem derived for online learning of general purpose options in an end to end fashion. In this work, we extend previous work on this topic that only focuses on learning a two-level hierarchy including options and primitive actions to enable learning simultaneously at multiple resolutions in time. We achieve this by considering an arbitrarily deep hierarchy of options where high level temporally extended options are composed of lower level options with finer resolutions in time. We extend results from [1] and derive policy gradient theorems for a deep hierarchy of options. Our proposed hierarchical option-critic architecture is capable of learning internal policies, termination conditions, and hierarchical compositions over options without the need for any intrinsic rewards or subgoals. Our empirical results in both discrete and continuous environments demonstrate the efficiency of our framework.

The paper extends the Option-Critic policy gradient theorems to multiple levels of abstractions (i.e. it allows options-over options at multiple levels). The results they present apply to execution models where action choices are conditioned on a vector [o^1, o_2, ..., o_N] representing abstract action choices at levels 1,2,...,N (1 being the highest). These choices are determined using policies \pi (o^j | o_{j-1}, o_{j-2}, ..., 1). Moreover, the conditioning is maintained until a termination signal, modeled in a similar way, is triggered at some level j, at which point [o_j , o_{j+1}, ..., o_N] are updated.  I consider this line of work very important, as it provides an investigation in the use of multiple levels of temporal abstractions, something that in most previous work was only alluded but never provided with clear derivations or empirical results. This paper provides these extensions. Although the paper is overall very well presented, there are some major technical details that are not properly described which makes me slightly skeptical of accepting the paper. See below for aspects that the authors should consider in improving the quality of the paper:  Major issues:  • From Eq (10), it seems that \beta(s, o^1, ..., o^N) is the prob. that at least one of the options terminate. For example, you use (1-\beta(s, o^1, ..., o^N)) for the event that none terminate.  Still, on L170 you give a different definition:  \beta(s, o^1, ..., o^{N-2}) is the prob. that o_{N-2} terminates given that o_{N-1} terminated and we were executing o^1, o_2, ..., o_{N-1}. This alternative definition is also used in the algorithm provided in the supplemental material. Although it makes no difference when N=2, it seems like a big inconsistency for N \geq 3. Please clarify the semantics of \beta to validate equation 10 and the Markov Chain in the supplemental material. • The paper argues for higher level abstractions, but nowhere in the experimental section do the authors try to illustrate the learned abstractions nor do they assess whether the resulting abstract options are degenerate (i.e. either never terminate or terminate immediately).  • L268- L270: Can you please add the details on how you ensured a fair comparison between the 2-level and multi-level architectures? I think this is a crucial aspect which should not be left out in the supplemental material. E.g. what was the numerical limit to put on the number of options at each level?  • as claimed in the abstract, the work "[learns abstract options] without the need for any intrinsic reward (IR) or subgoals", but it fails to explain why this is necessarily an advantage. I would argue that many of the problems associated with option-critic can be addressed by learning IR and subgoals (e.g. as demonstrated by work on Feudal Networks). The authors should clarify their perspective on the disadvantage of using IR and subgoals. •  Minor:  • nipick : what about "Hierarchical Option Critic" for a more concise title?   • L52: benefits • L74: Can you give some thoughts on learning both latent goal representations and options achieving these subgoals? Te statement on L74 seems to imply that these are orthogonal.  • L91: \pi: S \to (A \to [0,1]) for consistency with definition of P on L88. • L99:  P(s_t=s | s_0, \pi_\theta) -  state transitions are dependent on the policy • L110 - L129: You provide too much details coming from the Option-Critic paper - it would be much more useful to replace the policy gradient results (which are indirectly presented in 4.3) with a diagram that concisely illustrates the building blocks in Option-Critic as related to your work (e.g. the dependence between Q_\Omega, U, Q_U, V_\Omega, A_\Omega).  • Section 4 + supplemental work: consider using o_{1:k} instead of "o^1, o^2, ..., o^k". All results would read much easier.  • L185 (eq. 8) typo? Q_\Omega -> Q_U? • nipick: L351: remove "highly" + L352:  I don't think the experimental results support the statement as used in the conclusion. The results are interesting and illustrate in some cases the benefits of multiple levels of abstraction, but I would be reserved in using such superlatives.  ----After author feedback---- I would like to thank the authors for the additional feedback they provided, which includes clarifications to some of the issues I have raised. I still believe that the a proper qualitative analysis of the options learned at the higher levels is missing. I also find that a proper comparison to Feudal Networks is necessary for this work, and the author feedback is not very convincing on the many interesting aspects that Feudal Networks are using, such as intrinsic reward.


Generative Probabilistic Novelty Detection with Adversarial Autoencoders
Novelty detection is the problem of identifying whether a new data point is considered to be an inlier or an outlier. We assume that training data is available to describe only the inlier distribution. Recent approaches primarily leverage deep encoder-decoder network architectures to compute a reconstruction error that is used to either compute a novelty score or to train a one-class classifier. While we too leverage a novel network of that kind, we take a probabilistic approach and effectively compute how likely it is that a sample was generated by the inlier distribution. We achieve this with two main contributions. First, we make the computation of the novelty probability feasible because we linearize the parameterized manifold capturing the underlying structure of the inlier distribution, and show how the probability factorizes and can be computed with respect to local coordinates of the manifold tangent space. Second, we improve the training of the autoencoder network. An extensive set of results show that the approach achieves state-of-the-art performance on several benchmark datasets.

The problem of the study is outlier detection given only inlier training data. The high-level approach is to learn a density estimation function over the training data, and then filter out outliers using a learned threshold \gamma. They approximate the density function through a decomposition over the tangent space of the learned manifold near each given sample. To learn the manifold structure the authors use a variation of adversarial autoencoders. The evaluation is performed on MNIST, FashionMNIST, and COIL against a few baselines.  Overall, the paper is very well-written and easy to follow -- the presentation progresses coherently. However, there are several issues that I think must be addressed before the work is published.  At the heart of this work, the problem being addressed is that of density estimation. The learned function is then used to perform outlier detection. - There are several existing methods that are capable of performing density estimation. Autoregressive models such as PixelCNN, PixelCNN++, and the previous work, most of the recent compression techniques that use deep neural networks, and even a simple Variational Autoencoder can give us density estimates. There should be at least one of these density estimation methods to assess how well the proposed method approximates the density to determine whether the trouble of implementing such a complex technique is worth it. - The outlier detection problem is a quickly growing area which is not easy to track, but the authors should compare against easy-to-implement state-of-the-art methods such as ODIN [1] (ICLR 2018) at least. Thresholding the prediction of a predictive classifier, thresholding the reconstruction error of an AE that is trained with the same pipeline (without the additional loss functions) are two other easy to implement methods that they could compare against. - The presented evaluation is too weak. There are several measures in the outlier detection literature, from which the authors only report one or two. They don't adopt the same evaluation protocol either, yet, they use the numbers from other evaluations as a comparison. Furthermore, the evaluation is only performed on low-dimensional, low-complexity datasets MNIST, FashionMNIST, and COIL. There should be at least a CIFAR10 in the evaluation. It is common in the literature to also compare against CIFAR100 and downscaled imagenet.  Quality. The submission is technically sound. The idea is interesting and the work is executed and presented well and coherently. However, as mentioned earlier, at a high-level, the direction of the work should be slightly corrected to better position it within the existing literature.  Clarity. The paper is well-written and clear. The previous work could be significantly improved. In specific, I recommend adding the density estimation literature, and out-of-distribution detection literature as the method is addressing both of the problems. The mentioned work in the review could serve as a good starting point in these areas. The evaluation could also be improved as I discussed earlier.  Originality. I haven't seen this idea executed before. It is both novel and interesting. It would be influential if it improves the density approximation problem or performs outlier detection better (shown more strongly in evaluation). Even if it does not outperform the other methods, there still needs to be a visible connection and adequate comparison to position the work properly in the literature.  Significance. Unfortunately, I think, the work in its current state is unlikely to have an impact on the relevant areas.  References. [1] Liang, Shiyu, Yixuan Li, and R. Srikant. "Enhancing the reliability of out-of-distribution image detection in neural networks." arXiv preprint arXiv:1706.02690 (2017). -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Update after rebuttal. I think the author response has adequately addressed the raised concerns -- the paper would be a good submission with the proposed revisions. I have increased the overall score to 7.


Stochastic Cubic Regularization for Fast Nonconvex Optimization
This paper proposes a stochastic variant of a classic algorithm-the cubicregularized Newton method [Nesterov and Polyak, 2006]. The proposed algorithm efficiently escapes saddle points and finds approximate local minima for general smooth, nonconvex functions in onlyÕ(✏ 3.5 ) stochastic gradient and stochastic Hessian-vector product evaluations. The latter can be computed as efficiently as stochastic gradients. This improves upon theÕ(✏ 4 ) rate of stochastic gradient descent. Our rate matches the best-known result for finding local minima without requiring any delicate acceleration or variance-reduction techniques.

This submission is interested in stochastic nonconvex optimization problems, in which only stochastic estimates of the objective and its derivatives can be accessed at every iteration. The authors develop a variant of the cubic regularization framework, that only requires access to stochastic gradients and products of stochastic Hessians with vectors. Such a method is shown to reach a point at which the gradient norm is smaller than $\epsilon$ and the minimum Hessian eigenvalue is bigger than $-\sqrt{\rho \epsilon}$ in a number of stochastic queries (gradient or Hessian-vector product) of order of $\epsilon^{-3.5}$, which improves over the classical complexity of Stochastic Gradient Descent (SGD).  The problem of interest is clearly introduced, along with the possible advantages of using both stochastic estimates and a cubic regularization framework. The associated literature is correctly reviewed, and the authors even cite contemporary work that achieves similar complexity guarantees but rely on stochastic gradient estimates and variance reduction. It seems that these results could also have appeared in the table (Table 1) provided by the authors.   Section 3 of the submission contains its main results and a meta-algorithm that achieves the desired complexity rate, while Section 4 sketches the proof of the main result. The description of the algorithm and the properties of its subroutines are clearly presented. I would like to point out that the main algorithm resembles Algorithm 1 of [Carmon et Duchi, 2016], especially regarding the final application of the cubic subsolver: since the authors cite this reference, it seems that the algorithmic process cannot be considered as completely original.  Section 5 presents empirical results for the performance of the method, compared to two variants of SGD. The results are promising for the proposed algorithm, and several details about those experiments can be found in appendix. Those tests are undoubtedly relevant, however, one could also have expected a comparison with at least one variance-reduction method achieving a similar complexity bound, since those seem to be the only competitors on the theoretical side. I am thus surprised that the authors do not comment on such testing.  Overall, it is my opinion that this submission presents original and interesting results, that complement the existing landscape of algorithms in a meaningful way, They propose a method that relies on stochastic estimates together with an inexact solve of the cubic subproblem: their method is thus particularly appealing, although it is unclear whether this particular algorithm will outperform others possessing similar complexity guarantees. The submission is clearly written, and I particularly appreciate the balance between the main body of the submission and its appendices.  As a result, I will give this submission an overall score of 8.  Additional comments:  a) There appears to be several articles and conjunctions missing throughout the submission (e.g., "the" missing in "We also make following assumptions" L.117; "that" missing in "this will show if the cubic submodel" L.497).   b) In Table 1 page 3, I believe the footnote "1" on the bound for Natasha 2 should actually be a "2".  c) On Line 8 of Algorithm 2, I believe there is an extra $\eta$ in front of the $B[\Delta]$ term.  d) In Appendix E.1, it is said that a grid search was run over the hyperparameters of each method, but no information about the final configuration is provided. In particular, and contrary to Appendix E.2, the final adopted sample size is not given, yet this would be a relevant information for this submission.  Edit following author feedback:  I would like to point out that my comment on Appendix E was in favor of more details in Appendix E.1, not E.2 (in Appendix E.2, the final choice for gradient and Hessian batch sizes is clearly stated; this is not the case in Appendix E.1, where a range of options is given without specifying which option is the one used in the experiments). Apart from this misunderstanding, I am satisfied by the way the authors addressed my comments. I trust them to follow up on their response and incorporate a comparison with other second-order methods in their revised version, as this was also pointed out by another referee. Since all reviewers agreed on the overall score to give to this submission, I will maintain my initial assessment of the submission.


Adaptation to Easy Data in Prediction with Limited Advice
We derive an online learning algorithm with improved regret guarantees for "easy" loss sequences. We consider two types of "easiness": (a) stochastic loss sequences and (b) adversarial loss sequences with small effective range of the losses. While a number of algorithms have been proposed for exploiting small effective range in the full information setting, Gerchinovitz and Lattimore [2016]  have shown the impossibility of regret scaling with the effective range of the losses in the bandit setting. We show that just one additional observation per round is sufficient to circumvent the impossibility result. The proposed Second Order Difference Adjustments (SODA) algorithm requires no prior knowledge of the effective range of the losses, ε, and achieves an O(ε
expected regret guarantee, where T is the time horizon and K is the number of actions. The scaling with the effective loss range is achieved under significantly weaker assumptions than those made by Cesa-Bianchi and Shamir [2018]  in an earlier attempt to circumvent the impossibility result. We also provide a regret lower bound of Ω(ε √ T K), which almost matches the upper bound. In addition, we show that in the stochastic setting SODA achieves an O a:∆a>0
Kε 2 ∆a pseudo-regret bound that holds simultaneously with the adversarial regret guarantee. In other words, SODA is safe against an unrestricted oblivious adversary and provides improved regret guarantees for at least two different types of "easiness" simultaneously.

The paper presents an novel online algorithm for stochastic losses and adversarial losses with a small range under the assumption that they can observe an additional loss per round. In the adversarial setting, they provide a lower bound that almost matched their upper bound. In the stochastic setting, their regret bound remarkably does not depend on the rounds T.   Even though the setting described in the paper can be characterized by a sequence of feedback graphs, the authors do not cite or compare to the large body of literature of online learning with feedback graphs (i.e. "Online Learning with Feedback Graphs: Beyond Bandits" by Alon at al. 2015, "Online Learning with Feedback Graphs Without the Graphs" Cohen et al. 2016, "Leveraging Side Observations in Stochastic Bandits" by Caron et al. 2012).  Leveraging Side Observations in Stochastic Bandits by Caron et al. 2012 prove a lower bound of log(T) for stochastic losses when the feedback graph is fixed. The feedback graph used in the current paper is more complex than a fixed graph so it is surprising that they can attain a bound that does not depend on log T. I am not sure how the lower bound of Caron et al. can be reconciled with the stochastic bound in the paper, but it might be because in the current paper, they are allowed to chose which second arm they want to observe. Nevertheless, a thorough discussion and analysis is warranted here.   Another criticism is that they claim their assumptions are weak. Via their assumptions, they can bypass the impossibility result of Gerchinovitz and Lattimore [2016], but allowing an extra loss observation is essentially changing the setting and moreover they allow the algorithm to chose the second arm that they get to observe.  Adding extra loss observations even if it's just one more at each round can be hugely beneficial as shown in the above feedback graphs papers.   The paper is understandable, but not well written or polished.   Due to the lack of comparison to on-line learning with feedback graphs and the fact that Caron et al. 2012 have a lower bound that depends on log(T), I cannot vote to accept unless these issues are addressed by the authors in the rebuttal.   ----------------------------------------------------------------------------------------------- The authors have adequately addressed my concerns and hence I increased their overall score.


Credit Assignment For Collective Multiagent RL With Global Rewards
Scaling decision theoretic planning to large multiagent systems is challenging due to uncertainty and partial observability in the environment. We focus on a multiagent planning model subclass, relevant to urban settings, where agent interactions are dependent on their "collective influence" on each other, rather than their identities. Unlike previous work, we address a general setting where system reward is not decomposable among agents. We develop collective actor-critic RL approaches for this setting, and address the problem of multiagent credit assignment, and computing low variance policy gradient estimates that result in faster convergence to high quality solutions. We also develop difference rewards based credit assignment methods for the collective setting. Empirically our new approaches provide significantly better solutions than previous methods in the presence of global rewards on two real world problems modeling taxi fleet optimization and multiagent patrolling, and a synthetic grid navigation domain.

The paper tackles a multi-agent credit assignment problem, an egregious problem within multi-agent systems by extending existing methods on difference rewards for settings in which the population of the system is large. Though the results are relevant and lead to an improvement for large population systems, the contribution is nonetheless limited to a modification of existing techniques for a specific setting which seemingly requires the number of agents to be large and for the agents to observe a count of the agents within their neighbourhood. The results of the paper enable improved credit assignment in the presence of noise from other agents' actions, an improved baseline leading to reduced variance and, in turn, better estimates of the collective policy gradient (under homogeneity assumptions). The analysis of the paper applies to a specific setting in which the reward function has a term that is common to all agents and therefore is not decomposable. The extent to which this property is to be found in multi-agent systems, however, is not discussed. The setting is also partially observable multi-agent system - an instance of which is motivated by the authors. This, as the authors claim, implies that the agents need not observe the full state but need only make local observations for which the problem is formulated using a decentralised POMDP framework. The claims of the paper are supported by some theoretical results though some important details on which the theory rely seem to be omitted. Additionally, the analysis of the paper is accompanied by some experiments that showcase the benefits of the approach. A weakness of the general method is that the approach seems to alleviate computational complexity during execution of a learned policy but during the learning phase, however, the method seems not to alleviate the problem of explosive complexity in the number of agents which may cause tractability issues for large populations. Furthermore, the method seems reliant on the population size being large for the approximations to hold though the paper does not include a discussion on when the population size is reasonably large for the method to be valid. Finally, though the contribution is useful for the given setting, it is not clear how often the central assumption of non-decomposability occurs in multi-agent systems. (edited) There are also some other important questions that if addressed would improve the paper: In particular, though the setting of the paper is said to be a partially observable - it seems that 'distant' agents do not have an immediate effect on the agents’ rewards since the rewards depend on variables that are defined locally  (though they may interact with the agent at some later time). In this sense each agent's observations fully capture all relevant state information - it doesn't seem as though extra information concerning the agent's immediate reward or state transition can be inferred by including global information.  Secondly, the authors introduce the notion of agent types as a way of introducing heterogeneity into the formalism. The notion of agent types, however, seems inconsistent since it is assumed that the agents have the same policy which would differ for different agent types if type is any meaningful parameter (i.e. determines the agents' rewards/state transition probabilities). Additionally, heterogeneity from agent types also seems not to fit with the idea of a global critic.  Lastly, the advantage function (15) as used in the cited paper is used in a team game setting when agents have the same reward function. There are a number of details on which the theory is built that are omitted - in particular:  1. The authors do not state conditions under which sufficient statistic is actually sufficient (is this derived from application of the law of large numbers?) - this also seems to impose conditions of homogeneity on the agents - without a clear statement of the conditions under which this holds, some other statements are hard to evaluate e.g. line 163: "which makes learning scalable even with very large number of agents" and "Crucially, the above computation is independent of agent population M"- it seems that these statements in fact only hold in cases when M is large. This also arises in lines 195-196 where M must be large for the approximation to be valid as well as in the proof B1.  2. The authors state that "Previous work shows that count tables are the sufficient statistic for planning" - this seems to be fairly fundamental, it would be of great benefit to the reader to have a citation and a reference to the result for this remark. The paper would also benefit from a discussion on the solution admitted by the method. Specifically, though the setting is non-cooperative since each agent maximises their individual rewards the examples discussed nonetheless suggest the authors seek to target cooperative or team games. Though cooperative and team-based games can be tackled in a decentralised way,  the stable points in non-cooperative game setting (i.e. Markov game) are described by Nash equilibria (Markov-perfect Nash equilibria) which are in general inefficient from a social welfare perspective. The authors ought to include a discussion on how they expect the payoffs from the learned policies which seem to be generated by Nash equilibrium strategies to perform in relation to the system goal in general and, possibly how they might overcome challenges of inefficiency from non-cooperative challenges.    3. What is the meaning of d - can the authors give an example?  4. Additionally, the authors state that "We assume that Qw is differentiable in all input parameters". This doesn't seem to be a reasonable assumption given that one of the inputs n^{sa} is a discrete integer count.    The reason for my score is the following. The paper is clearly written leading to a coherent exposition and its objectives are clearly stated. The contribution seems however only incremental given the existing (and cited) methods on difference rewards and in the absence of clear statements when the method is most applicable. The crucial element, however, is about the correctness of the method. I have two major concerns: i) about the differentiability of Q_w* and ii) about small population sizes. There's also a (minor) issue about the claim that the method can work with heterogeneous populations, where the method, which seems to depend on one single policy being learned, may break down. i) The assumption of differentiability of Q_w*, which appears to be (likely) invalidated in the current setting as one of its inputs is a discrete integer variable. This seems to me to be a concern as differentiation here is ill-defined. The authors have not addressed this issue in their rebuttal.   ii) The derivations of the Q function estimates rely on large populations (e.g. eqs. (5), (9)), and use previous results (e.g. citations [4] and [6] on the appendix) but those papers (beside having many notational inconsistencies) also rely on large populations.  Overall, the paper is well written and easy to follow, but I  have major concerns with the correctness of the method. Also, reading the previous papers that this paper heavily relies on, the contribution looks very incremental.


Provable Variational Inference for Constrained Log-Submodular Models
Submodular maximization problems appear in several areas of machine learning and data science, as many useful modelling concepts such as diversity and coverage satisfy this natural diminishing returns property. Because the data defining these functions, as well as the decisions made with the computed solutions, are subject to statistical noise and randomness, it is arguably necessary to go beyond computing a single approximate optimum and quantify its inherent uncertainty. To this end, we define a rich class of probabilistic models associated with constrained submodular maximization problems. These capture log-submodular dependencies of arbitrary order between the variables, but also satisfy hard combinatorial constraints. Namely, the variables are assumed to take on one of -possibly exponentially many -set of states, which form the bases of a matroid. To perform inference in these models we design novel variational inference algorithms, which carefully leverage the combinatorial and probabilistic properties of these objects. In addition to providing completely tractable and well-understood variational approximations, our approach results in the minimization of a convex upper bound on the log-partition function. The bound can be efficiently evaluated using greedy algorithms and optimized using any first-order method. Moreover, for the case of facility location and weighted coverage functions, we prove the first constant factor guarantee in this setting -an efficiently certifiable e/(e − 1) approximation of the log-partition function. Finally, we empirically demonstrate the effectiveness of our approach on several instances.

The authors present an algorithm for approximate inference in exponential family models over the bases of a given matroid.  In particular, the authors show how to leverage standard variational methods to yield a provable approximation to the log partition function of certain restricted families.  This is of interest as 1) these families can be difficult to handle using standard probabilisitic modeling approaches and 2) previous bounds derived from variational methods do not necessarily come with performance guarantees.  General comments:  Although the approach herein would be considered variational approximation methods, the term variational inference is more commonly used for a related but different optimization problem.  There are lots of other variational approaches that yield provable upper/lower bounds on the partition function.  Some effort should be made to cite these.  A few to help the authors get started (see citations therein for more):  Tamir Hazan and Amnon Shashua. "Norm-product belief propagation: Primal-dual message-passing for approximate inference." IEEE Transactions on Information Theory 56, no. 12 (2010): 6294-6316.  Meltzer, T., Globerson, A. and Weiss, Y., 2009, June. Convergent message passing algorithms: a unifying view. In Proceedings of the twenty-fifth conference on uncertainty in artificial intelligence (pp. 393-401). AUAI Press.  Nicholas Ruozzi. Beyond log-supermodularity: lower bounds and the Bethe partition function. Uncertainty in Artificial Intelligence (UAI), July 2013.   Nicholas Ruozzi.  The Bethe partition function of log-supermodular graphical models. Advances in Neural Information Processing Systems (NIPS), December 2012.  Typos: Line 222, "which is easily seen to concave" -> "which is easily seen to be concave"