BRITS: Bidirectional Recurrent Imputation for Time Series
Time series are ubiquitous in many classification/regression applications. However, the time series data in real applications may contain many missing values. Hence, given multiple (possibly correlated) time series data, it is important to fill in missing values and at the same time to predict their class labels. Existing imputation methods often impose strong assumptions of the underlying data generating process, such as linear dynamics in the state space. In this paper, we propose a novel method, called BRITS, based on recurrent neural networks for missing value imputation in time series data. Our proposed method directly learns the missing values in a bidirectional recurrent dynamical system, without any specific assumption. The imputed values are treated as variables of RNN graph and can be effectively updated during backpropagation. BRITS has three advantages: (a) it can handle multiple correlated missing values in time series; (b) it generalizes to time series with nonlinear dynamics underlying; (c) it provides a data-driven imputation procedure and applies to general settings with missing data. We evaluate our model on three real-world datasets, including an air quality dataset, a healthcare dataset, and a localization dataset for human activity. Experiments show that our model outperforms the state-of-the-art methods in both imputation and classification/regression.

Post-rebuttal comment: The authors responded to some of my concerns. In particular, they addressed my question about the utility of this model to do imputation independent of the supervision task. According to their response, the imputed data produced by BRITS is superior in terms of downstream classification performance than forward-filled data, demonstrating utility. If this experiment can be extended to include comparisons from the other imputation methods then the paper will be strengthened. On this assumption, I change my score from a 5 to 6.  Summary of work: They describe a RNN-based framework for performing imputation and simultaneous classification of time series. The RNNs are trained to predict the next (or previous) value of the time-series , with a consistency requirement between forward and backward RNNs. The architecture makes use of masking variables to denote missing values, and time deltas since the last observed value. The latter are used to decay parts of the hidden state, and to weight the relative contributions to the imputed value from other variables, or from the historical prediction of that variable. Since imputation is performed at the same time as classification, they report both imputation performance and AUROC on three real-world datasets.  Explanation of overall score: I think this approach is solid, but the analyses are slightly lacking, limiting the general interest and applicability of the approach.  Explanation of confidence score: I am reasonably familiar with the related work, and there isn’t much mathematics to check.  Quality:  * The proposed model and variants are evaluated on three datasets for imputation, and two for imputation+binary classification. * I like the analysis in figure 4/appendix C, this is a useful experiment. * That the “correlated recurrent imputation” helps much over the independent version ((B)RITS v. (B)RITS-I) for imputation is only evident on the healthcare dataset - in the other cases, the difference is of the order of 0.1 percentage points or so (and RITS-I confusingly even outperforms RITS on the Human Activity dataset). Depending on the nature of the missingness this doesn’t seem entirely surprising - the difference between the independent and correlated prediction will depend on: 1) how correlated the variables actually are, and 2) how many non-missing values are observed alongside the data-point to be imputed. It would be interesting to see if these factors could explain the relative performance of BRITS v. BRITS-I on the different datasets. * How is the performance of GRU-D for imputation measured if it doesn’t impute missing data explicitly? * In applied machine learning, imputation is usually done as a precursor step to e.g. a supervised learning problem. I’m glad to see that performance on subsequent tasks is included in this work, but I would like to see more detail here. As indicated by the relatively poor imputation performance of GRU-D (although this may be related to the fact that it doesn’t perform imputation, as per my previous question), but the high classification performance, it would be interesting to see more about how imputation accuracy actually impacts downstream classification. I think to do this in a fair way you would need to separate the imputation and classification tasks - get each model to first perform imputation on the dataset, and then train the same classifier architecture (e.g. a random forest) on each imputed version of the dataset, and compare performance. This would enable the imputation performed by the non-RNN baselines to be compared. You could then also compare how training the RNN to classify at the same time as impute (as is done already) presumably improves classification performance, and potentially changes the imputed data. My intuition here is that certain imputation strategies may appear poor by absolute error, but may actually be adequate for the types of downstream tasks considered. * Highlighting a sub-part of the previous point - I think it would be interesting independent of the above investigation to see how much of an effect the classification task has on the imputation performance - that is, to turn off L_out in the loss. It would be an interesting result (and perhaps a useful contribution to the ongoing discussion around unsupervised learning) to demonstrate that having a task can improve imputation performance - and what would be doubly interesting would be to show that the resulting imputed data is still useful for *other* tasks! * We see empirically that BRITS outperforms GRU-D and MRNN, and the use of feature correlations is given as an explanation, but (B)RITS-I also outperforms both models, so something else must be going on. Is there something else about the (B)RITS architecture that’s providing an advantage? * One question: is it feasible to run this model to perform imputation in real-time? * In summary, I think the experiments here are interesting, but there are many more potentially interesting analyses which could be run.  Clarity:  * I like how this paper was laid out - the RITS-I -> BRITS-I -> BRITS progression made it quite easy to see how the model is constructed. * I think it would be useful to highlight where data is missing at random, or not-at-random in these experiments. I think it’s not at random in the Air Quality dataset, but at random in the others? The descriptions (e.g. figure 1, example 1) make it sound like the primary form of missingness is missing in blocks, which is slightly misleading. * Slight inconsistency: in the GRU-D paper, the AUROC on mortality in the PhysioNet data is reported at 0.8424 ± 0.012, which would make it comparable to the performance of BRITS (0.850 ± 0.002), but in this submission it is reported at 0.828 ± 0.004. * Minor: I think that errors should not be neglected when choosing which result to put in boldface (see last two rows, second column in Table 2 - these results are within error of each other). * Regarding reproducibility, I greatly appreciate that the authors provided an anonymous code repository, however the details of how the other RNN models were trained are missing.  Originality:  * It would be helpful if the other (GRU-D, MRNN) methods were described in a little more detail to make clear which elements of this model are original.  Significance:  * Solving problems on time-series data with missing data is quite important, so this work is significant in that sense. * However, I feel the analyses are slightly limited, making it hard to conclude what exactly about this architecture is providing benefit, or how generally useful it is for imputation (for example, in cases where the classification problem is not known ahead of time, or the classification architecture will not also be an RNN).


Bilevel Distance Metric Learning for Robust Image Recognition
Metric learning, aiming to learn a discriminative Mahalanobis distance matrix M that can effectively reflect the similarity between data samples, has been widely studied in various image recognition problems. Most of the existing metric learning methods input the features extracted directly from the original data in the preprocess phase. What's worse, these features usually take no consideration of the local geometrical structure of the data and the noise that exists in the data, thus they may not be optimal for the subsequent metric learning task. In this paper, we integrate both feature extraction and metric learning into one joint optimization framework and propose a new bilevel distance metric learning model. Specifically, the lower level characterizes the intrinsic data structure using graph regularized sparse coefficients, while the upper level forces the data samples from the same class to be close to each other and pushes those from different classes far away. In addition, leveraging the KKT conditions and the alternating direction method (ADM), we derive an efficient algorithm to solve the proposed new model. Extensive experiments on various occluded datasets demonstrate the effectiveness and robustness of our method.

Summary: The authors propose a bilevel method for metric learning, where the lower level is responsible for the extraction of discriminative features from the data based on a sparse coding scheme with graph regularization. This effectively detects their underlying geometric structure, and the upper level is a classic metric learning approach that utilizes the learned sparse coefficients. These two components are integrated into a joint optimization problem and an efficient optimization algorithm is developed accordingly. Hence, new data can be classified based on the learned dictionary and the corresponding metric. In the experiments the authors demonstrate the capabilities of the model to provide more discriminative features from high dimensional data, while being more robust to noise.  Quality: The technical contribution of the paper seems to be pretty solid. In particular, the argumentation about the feature extraction and why this is beneficial for the metric learning, is pretty convincing. Also, the final objective function is directly interpretable. In addition, the derivation of the optimization algorithm is as clean as possible, and nicely presented. Furthermore, the converge analysis of the algorithm is discussed as well. So the proposed framework is sufficiently complete. However, I have two comments:  1) The graph regularizer in Eq.16 utilizes the weights q_i, which are not defined properly in the text. Especially, given that the Laplacian is constructed during training in a supervised way.   2) Could you provide some information considering the efficiency of the model? Consider for instance the training time.  Clarity: The paper is very well-written and nicely organized. Also, it provides neatly all the necessary information to the reader. The related work seems to be discussed properly. I have few minor comments:  1) In Fig.1 it would be nice if you could give a better explanation of the x-axis. To my undestanding these are the samples i=1,...n categorized, but this is not direcly communicated by the figure.   2) Does the construction of the Laplacian in a supervised way implies that you construct the adjacency matrix based on the given labels?  3) In Fig.2 you could have used log-scale for the y-axis.  Originality: I am not an expert in the field, but the related work seems to be properly cited. Also, by taking into account the related work as presented by the authors, the proposed model seems to be novel enough. Even if the bilevel concept with the graph regularized sparse coding as an integrated feature extractor has been utilized in previous models, in the current paper it is considered into the metric learning framework. Also, the provided optimization algorithm has been developed specifically for the proposed model.   Significance: The idea of integrating a meaningful feature extractor in the metric learning task, is very interesting and makes a lot of sense. Especially, I like the fact that in the proposed model this combination is implemented naturally. Also, the developed optimization algorithm is remarkable, since it solves a rather complicated problem. The improvement upon the results of the experiments is considerable, which shows that the proposed approach is particularly useful.   General comments: The general idea of bilevel models is pretty appealing. In particular, the lower level is designed in such a way that it provides meaningful features into the upper level, which solves the main problem. In this paper, the discriminative features that capture the underlying geometric structure of the data are utilized such that to learn a Mahalanobis distance for classification. The provided optimization algorithm that solves the joint problem is very interesting as well, since the problem is challenging. The experimental results show that, indeed, extracting meaningful features from the data, can lead to significant improvements.   ------------------------------------------------ After reading the rebuttal my final comments are that the submission seems to be solid, and also, considering the replies of the authors, I feel that they answer sufficiently well all the comments of the reviewers. So I will keep my score and vote for acceptance.


First-order Stochastic Algorithms for Escaping From Saddle Points in Almost Linear Time
In this paper, we consider first-order methods for solving stochastic non-convex optimization problems. The key building block of the proposed algorithms is firstorder procedures to extract negative curvature from the Hessian matrix through a principled sequence starting from noise, which are referred to NEgative-curvatureOriginated-from-Noise or NEON and are of independent interest. Based on this building block, we design purely first-order stochastic algorithms for escaping from non-degenerate saddle points with a much better time complexity (almost linear time in the problem's dimensionality) under a bounded variance condition of stochastic gradients than previous first-order stochastic algorithms. In particular, we develop a general framework of first-order stochastic algorithms with a secondorder convergence guarantee based on our new technique and existing algorithms that may only converge to a first-order stationary point. For finding a nearly second-order stationary point x such that ∇F (x) ≤ and ∇ 2 F (x) ≥ − √ I (in high probability), the best time complexity of the presented algorithms is O(d/ 3.5 ), where F (·) denotes the objective function and d is the dimensionality of the problem. To the best of our knowledge, this is the first theoretical result of first-order stochastic algorithms with an almost linear time in terms of problem's dimensionality for finding second-order stationary points, which is even competitive with existing stochastic algorithms hinging on the second-order information.

This paper proposes an efficient algorithm, Neon, to extract the negative-curvature of the Hessian matrix. Neon is very useful, which can be taken as a build block to convert existing convert existing stationary point finding algorithms into local minimum finding algorithms. This is because although the gradient of the points around saddle point is very small, there still exists a direction to decrease the function value by updating the variable along the negative-curvature of its Hessian matrix. In this way, Neon can be easily integrated with existing stationary point finding algorithms, such as SGD, accelerated SGD, SCGC, etc. Actually, the authors theoretically showed that SGD, SCGC, Natasha and SVRG can be combined with Neon and compared with existing works, such as noisy SGD and SGLD, the achieved computational complexity has much better dependence on the dimension d of the problem. Specifically, the complexity of noisy and SGLD rely on O(d^p) (p>=4), while the complexity in this work is only O(d). For the optimization accuracy, the best complexity in this work makes about improvement O(1/epsilon^0.75). Compared with Hessian vector product based algorithm, Neon only need to use first order information and is thus more practical.   To my best knowledge, this work seems to be the first work which firstly use first-order gradient to compute the negative-curvature for escaping saddle points. Based on this work, there are some follow-up/independent works, such as Neon2 proposed by Allen-Zhu et al.  So I think that this paper has very strong novelty and makes great contributions to optimization area, especially for escaping saddle points.    However, the discussion of related works is not sufficient. For example, the differences between Neon and Neon2, AGD proposed by Jin Chi are not very clear, since they share similar techniques and the authors only slightly mentioned the differences.   The authors claim that their work is the first one which only linearly depends on the problem dimension d. I wonder whether existing works, such as Natasha2, SNCG although they involve the second information, e.g. Hessian vector product, also depend on the problem dimension d polynomially or higher.  Finally, as for the experiments, the authors only show the convergence behavior of Neon-SGD, Neon+-SGD and noisy SGD in Fig. 2. However, these plots only demonstrated that these algorithms can converge but cannot prove they can escape saddle points. A better comparison may be that adding an algorithm, e.g. SGD, and looking at SGD can achieve the same optimization accuracy with Neon-SGD, Neon+-SGD and noisy SGD.


Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agents
Evolution strategies (ES) are a family of black-box optimization algorithms able to train deep neural networks roughly as well as Q-learning and policy gradient methods on challenging deep reinforcement learning (RL) problems, but are much faster (e.g. hours vs. days) because they parallelize better. However, many RL problems require directed exploration because they have reward functions that are sparse or deceptive (i.e. contain local optima), and it is unknown how to encourage such exploration with ES. Here we show that algorithms that have been invented to promote directed exploration in small-scale evolved neural networks via populations of exploring agents, specifically novelty search (NS) and quality diversity (QD) algorithms, can be hybridized with ES to improve its performance on sparse or deceptive deep RL tasks, while retaining scalability. Our experiments confirm that the resultant new algorithms, NS-ES and two QD algorithms, NSR-ES and NSRA-ES, avoid local optima encountered by ES to achieve higher performance on Atari and simulated robots learning to walk around a deceptive trap. This paper thus introduces a family of fast, scalable algorithms for reinforcement learning that are capable of directed exploration. It also adds this new family of exploration algorithms to the RL toolbox and raises the interesting possibility that analogous algorithms with multiple simultaneous paths of exploration might also combine well with existing RL algorithms outside ES.

This submission proposes methods to enhance exploration in the Evolution Strategies framework introduced by Salimans et al. (ES), a black box optimization method that can leverage massive hardware resources to speedup training of RL agents, by introducing novelty seeking objectives from evolution/genetic algorithms literature. The balance between exploration and exploitation is a fundamental problem in RL, which has been thoroughly studied for many DRL methods, and is very interesting to explore it for alternative methods like ES. I find NSRA-ES particularly interesting, as it naturally handles the trade-off between exploration and exploitation, and will likely motivate new research in this direction. One of the weaknesses of the approach is the definition of the behavior characterization, which is domain-dependent, and may be difficult to set in some environments; however, the authors make this point clear in the paper and I understand that finding methods to define good behavior characterization functions is out of the scope of the submission.  The article is properly motivated, review of related work is thorough and extensive experiments are conducted. The methods are novel and their limitations are appropriately stated and justified. The manuscript is carefully written and easy to follow. Source code is provided in order to replicate the results, which is very valuable to the community. Overall, I believe this is a high quality submission and should be accepted to NIPS.   Please read more detailed comments below:  - The idea of training M policies in parallel is somewhat related to [Liu et al., Stein Variational Policy Gradient], a method that optimizes for a distribution of M diverse and high performing policies. Please add this reference to the related work section.  - The update of w in NSRA-ES somehow resembles the adaptation of parameter noise in [Plapper et al., “Parameter Space Noise for Exploration”, ICLR 2018]. The main difference is that the adaptation in Plappert et al. is multiplicative, thus yielding more aggressive changes. Although this is not directly compatible with the proposed method, where w is initialized to 0, I wonder whether the authors tried different policies for the adaptation of w. Given its similarity to ES (where parameter noise is used for structured exploration instead of policy optimization), I believe this is a relevant reference that should be included in the paper as well.  - It seems that the authors report the best policy in plots and tables (i.e. if f(theta_t) > f(theta_{t+k}), the final policy weights are theta_t). This is different to the setup by Salimans et al. (c.f. Figure 3 in their paper). I understand that this is required for methods that rely on novelty only (i.e. NS-ES), but not for all of them. Please make this difference clear in the text.  - Related to the previous point, I believe that section 3.1 lacks a sentence describing how the final policy is selected (I understand that the best performing one, in terms of episodic reward, is kept).  - In the equation between lines 282 and 283, authors should state how they handle comparisons between episodes with different lengths. I checked the provided code and it seems that the authors pad the shorter sequence by replicating its last state in order to compare both trajectories. Also, the lack of a normalization factor of 1/T makes this distance increase with T and favors longer trajectories (which can be a good proxy for many Atari games, but not necessarily for other domains). These decisions should be understood by readers without needing to check the code.  - There is no caption for Figure 3 (right). Despite it is mentioned in the text, all figures should have a caption.  - The blue and purple color in the plots are very similar. Given the small size of some of these plots, it is hard to distinguish them -- especially in the legend, where the line is quite thin. Please use different colors (e.g. use some orangish color for one of those lines).  - Something similar happens with the ES plot in Figure 2. The trajectory is quite hard to distinguish in a computer screen.  - In SI 6.5, the authors should mention that despite the preprocessing is identical to that in Mnih et al. [7], the evaluation is slightly different as no human starts are used.   - In SI 6.6, the network description in the second paragraph is highly overlapping with that in the first paragraph.    ----------------------------------------------------------------------------------------------------------------------------------------------------------  Most of my comments had to do with minor modifications that the authors will adress. As stated in my initial review, I vote for accepting this submission.


Adversarially Robust Generalization Requires More Data
Machine learning models are often susceptible to adversarial perturbations of their inputs. Even small perturbations can cause state-of-the-art classifiers with high "standard" accuracy to produce an incorrect prediction with high confidence. To better understand this phenomenon, we study adversarially robust learning from the viewpoint of generalization. We show that already in a simple natural data model, the sample complexity of robust learning can be significantly larger than that of "standard" learning. This gap is information theoretic and holds irrespective of the training algorithm or the model family. We complement our theoretical results with experiments on popular image classification datasets and show that a similar gap exists here as well. We postulate that the difficulty of training robust classifiers stems, at least partially, from this inherently larger sample complexity.

The paper analyses the robustness of machine learning algorithms to adversarial examples. For two particular artificial data distributions of binary classification data---overlapping Gaussians and overlapping Bernoulli distributions---the paper analyzes the sample complexity required to achieve robustness against adversarial examples. This sample complexity is compared to classical ones for the generalization error. The theoretical findings are supported by experiments on datasets that share properties with the artificial data distributions.   The quality of the paper is good, the theoretical parts are sound and well-explained in the appendix. Even though the analyzed data distributions are very limited, the results are still very insightful; in particular, because they hold for arbitrary learning algorithms. The empirical evaluation is insightful and properly related to the theoretical findings.   The paper is well-written and clear. The theoretical results are undertandable, their proofs are comprehensible. I have some minor comments to improve the clarity at the end of this review.   The main contribution of the paper are robust sample complexity bounds for two specific data distributions for binary classification for arbitrary learning algorithms (and model classes). This adds considerably to the understanding of robustness against adversarial examples. The empirical evalutation support these theoretical results. In contrast to the other theoretical findings the results on thresholding appear to be a bit trivial though: if you map every perturbation of a binary vector back to a binary vector using thresholding, then trivially the perturbations have no effect as long as they do not change the sign of elements of the binary vector.  The paper is a useful contribution to the community, because it extends the understanding of robustness against adversarial examples. Since the theoretical analysis holds for arbitrary learning algorithms, it emphasizes that adversarial examples are a general problem for learning algorithms that can be tackled by using more training examples. However, since max-margin classifiers are known to be more robust against adversarial examples, it would be great to discuss them in relation to especially deep neural networks in more detail.  The author's response has addressed my concerns about the thresholding approach, as well as my remark to further investigate max-margin classifiers.  Detailed Comments:  - The sample complexity of robust generalization is larger by sqrt(d), i.e., sub-linear in the dimension, not polynomial. - I would have liked the authors to explain more on the relation of the sample complexity of max-margin models to the robust sample complexity. - In theorem 4, I think \widehat{w} = yx should be \widehat{w} = yx / \|x\|_2, same in Theorem 5. - Line 160: the reference Corollary 23 should be Corollary 22. - In Theorem 4 and 8 it should be stated that achieving this bound with a single example is only possible because of the symmetrie of the positive and negative class distributions (at first glance, the result was a bit surprising). - I can see some benefit in shortening the formulas in Theorem 4 and Theorem 5 using constants c, c_1, c_2. However, since c_1 = 1/32 and c_2 = 64, they could also be included. Same for c= (5\sqrt{\log 1/\beta})^-1 for \beta = 0.01 (i.e., c = 5\sqrt(2)), where even the desired error bound \beta is part of the constant. - Some restrictions on possible variable assignments are not given explicitely: e.g., in Corollary 19, the error is assumed to be < 1, in Corollary 22 the \epsilon-Radius is assumed to be <= 1/4. - In the introduction, when introducing l_\infty perturbations, I suggest noting that this means robustness against adversarial examples generated within the l_\infty hypercube around a given example. - In the motivating example in the introduction: since Figure 1 shows accuracies, I suggest not using the term error---even though the two are related---since it is a bit confusing at first. Also, it might be interesting to also see the standard train accuracy.


Middle-Out Decoding
Despite being virtually ubiquitous, sequence-to-sequence models are challenged by their lack of diversity and inability to be externally controlled. In this paper, we speculate that a fundamental shortcoming of sequence generation models is that the decoding is done strictly from left-to-right, meaning that outputs values generated earlier have a profound effect on those generated later. To address this issue, we propose a novel middle-out decoder architecture that begins from an initial middle-word and simultaneously expands the sequence in both directions. To facilitate information flow and maintain consistent decoding, we introduce a dual self-attention mechanism that allows us to model complex dependencies between the outputs. We illustrate the performance of our model on the task of video captioning, as well as a synthetic sequence de-noising task. Our middleout decoder achieves significant improvements on de-noising and competitive performance in the task of video captioning, while quantifiably improving the caption diversity. Furthermore, we perform a qualitative analysis that demonstrates our ability to effectively control the generation process of our decoder.

This paper describes a middle-out decoding scheme for the encoder-decoder frameworks, and its application to a synthetic symmetric task and the video captioning task. The main idea is that the decoder starts from an externally provided middle word (either an oracle, or a word predicted using a classifier over the vocabulary), and then alternates between expanding to the left and to the right. The two directions are connected through a self attention mechanism which uses two heads, one over the previous decoder hidden states, and one over the decoder inputs. The use of two attention heads for self attention is claimed as a novel contribution, but seems to be more of a design decision. The use of two heads instead of one is never tested directly. Strong results are shown on a synthetic task, and they are able to match the baseline on image captioning while increasing diversity. They can also easily control the output by changing the provided middle word.   This is a clear paper with an interesting main idea and sound experimental design. I’m not sure the idea is sufficiently developed to warrant publication at this stage, but it is very close. I’m also not sure it’s a very good idea, which is affecting my grade somewhat, though I am trying my best to judge the work primarily based on its evaluation of this idea.  At its core, middle-out decoding re-introduces some amount of pipelining into the encoder-decoder architecture. The system designer must decide how the middle word will be selected from the reference sentence (it isn’t the exact middle word by token count, but some high-information content word like a main verb or the subject or or object noun phrase), and they must decide how that middle word will be predicted before decoding begins. Those decisions will have a large impact on the ceiling for the complete system that will be built around them. I consider this pipeline to be a pretty steep cost. What we are buying with this cost is controllability and diversity. I do not see controllability as a large gain. Controllable output has been studied in NMT, and I’ll provide some references below. And the diversity effect isn’t compared against other methods of introducing diversity (again, I’ll provide references). So, I’m not sure if this cost is worth what we gain. Hence my middling score.  There are a number of things the authors could do to make this a stronger paper:  Implement the proposed future work and use a strong, proven action classification architecture to predict the middle word, and beat the baseline of Seq2Seq + Attention + Self-Attention. I suspect that if this had happened, I would not be focusing so much on the side-benefits. Another way to potentially improve your score would be to try incorporating the middle word choice into the beam, so with a beam of 5, the system starts with the 5-best choices for the middle word from the current model. That is, if you are not already doing so (it isn’t clear from the paper).  Compare to known methods for controlling the words in the output sequence by altering the beam search: https://arxiv.org/abs/1804.06609 https://arxiv.org/abs/1704.07138 I suspect you will win here, as you are altering both the search and the model, while the above techniques alter only the search, but without experiments it’s hard to know. Providing both the embedding of the constraint-word to the decoder while also constraining the search to output the word would be an interesting and novel way to extend the work cited above to change both the model and search.  Compare to a known methods for introducing diversity, for example (though you can probably do better than these, there may well be techniques specific to the image captioning domain that I am unaware of): https://arxiv.org/abs/1611.08562 https://arxiv.org/abs/1605.03835  Other, smaller concerns:  Equation (3) has a circular dependency on h^d_t, you probably meant h^d_{t-1}.  Do you share encoders for the middle word classifier and the middle-out decoder? If so, please specify how you balance the two loss functions. If not, it might be worth considering to strength the middle-word classifier.  Your results on the synthetic data are impressive, but it would be worthwhile to specify how many more parameters are used with this new model. Likewise for the captioning experiments.  ===  Thanks very much for addressing my comments in your response. Given these additional experiments and comments, I will gladly raise my scores. Using a sampled oracle to simulate a higher accuracy classifier was particularly clever. Good luck with your future research.  ===


An Off-policy Policy Gradient Theorem Using Emphatic Weightings
Policy gradient methods are widely used for control in reinforcement learning, particularly for the continuous action setting. There have been a host of theoretically sound algorithms proposed for the on-policy setting, due to the existence of the policy gradient theorem which provides a simplified form for the gradient. In off-policy learning, however, where the behaviour policy is not necessarily attempting to learn and follow the optimal policy for the given task, the existence of such a theorem has been elusive. In this work, we solve this open problem by providing the first off-policy policy gradient theorem. The key to the derivation is the use of emphatic weightings. We develop a new actor-critic algorithm-called Actor Critic with Emphatic weightings (ACE)-that approximates the simplified gradients provided by the theorem. We demonstrate in a simple counterexample that previous off-policy policy gradient methods-particularly OffPAC and DPG-converge to the wrong solution whereas ACE finds the optimal solution.

After rebuttal: The author addresses some of my questions in the rebuttal.  Overall, I think the paper can use better presentation of the theorems and contrast with previous results on semi-gradient theorems. For now the theory might be quite limited in its scalability to larger problems, it might be interesting to explore further how one can combine it with scalable methods in the future. I increase my rating a bit. ====== Summary: The paper proposes a new off-policy gradient form as well as a new procedure to compute the gradient using emphatic weighting. The paper also empirically compares the proposed method with DPG algorithms on a simple three state MDP problem, showing that the proposed method can learn optimal policies while DPG is stuck at local optimal. Questions: 1. Line 96. dot v_s is the gradient of v_s with respect to states or parameters? The states are discrete here so I presume it is parameters? Can authors write down the mathematical form of the gradient in the paper. 2. The author should make it more clear that the LHS of Proposition 1 corresponds to the gradient of interest. 3. What is semi-gradient? The paper did not seem to clarify what this definition is. For example, the update of DPG does follow the gradient (with some technical details to drop an additional term), does the ‘semi-‘ come from dropping such a term? General Advice: 1. Previous work. Though the paper mentions previous works on offPAC and DPG on off-policy optimization, the comparison is not made obvious and clear. Importantly, the paper did not write the mathematical equations for DPG (for example), and make it a bit difficult for direct comparison. 2. Notations and presentation. There is much notation in the paper that is not quite directly clear. I think it definitely helps to clarify interpretations of some terms in (for example) LHS of proposition1.  3. Experiments are a bit weak. The experiments could include probably some more conventional MDP systems to illustrate that the new method does not deteriorate compared with DPG/offPAC because of the difficulty in weight estimate. Current experiments only consider the three MDP system and seem a bit weak. Also, is it possible to carry out function approximation in such a setup to make the algorithm scale and applicable to more realistic system?


On the Local Hessian in Back-propagation
Back-propagation (BP) is the foundation for successfully training deep neural networks. However, BP sometimes has difficulties in propagating a learning signal deep enough effectively, e.g., the vanishing gradient phenomenon. Meanwhile, BP often works well when combining with "designing tricks" like orthogonal initialization, batch normalization and skip connection. There is no clear understanding on what is essential to the efficiency of BP. In this paper, we take one step towards clarifying this problem. We view BP as a solution of back-matching propagation which minimizes a sequence of back-matching losses each corresponding to one block of the network. We study the Hessian of the local back-matching loss (local Hessian) and connect it to the efficiency of BP. It turns out that those designing tricks facilitate BP by improving the spectrum of local Hessian. In addition, we can utilize the local Hessian to balance the training pace of each block and design new training algorithms. Based on a scalar approximation of local Hessian, we propose a scale-amended SGD algorithm. We apply it to train neural networks with batch normalization, and achieve favorable results over vanilla SGD. This corroborates the importance of local Hessian from another side.

#### Summary  The authors attempt to characterize neural network architectures and training methods conducive to successful training. They propose that backpropagation with respect to a loss function is equivalent to a single step of a "back-matching propagation" procedure in which, after a forward evaluation, we alternately optimize the weights and input activations for each block to minimize a loss for the block's output. The authors propose that architectures and training procedures which improve the condition number of the Hessian of this back-matching loss are more efficient and support this by analytically studying the effects of orthonormal initialization, skip connections, and batch-norm. They offer further evidence for this characterization by designing a blockwise learning-rate scaling method based on an approximation of the backmatching loss and demonstrating an improved learning curve for VGG13 on CIFAR10 and CIFAR100.  #### Quality  The authors assembled a good set of data points to make the case for investigating backmatching propagation procedures, but the strength of this case may be insufficient for a complete work at NIPS. The authors acknowledged that one of the assumptions used for the residual connections case may be too strong and that empirical validation is lacking. Additionally, the scale-amended SGD experiment results appear to rely on individual training runs. For experiments at this scale, it seems reasonable to expect some measure of variance from multiple trials.  #### Clarity  -   If I understood the paper's case correctly, the claim that improving the training efficiency requires or implies a well-conditioned local Hessian is not mathematically strict and rather depends on the collection of observations listed above. If this is the case, the authors should consider softening some assertions in the abstract.  -   I believe equation (12) somewhat overloads F_b; would it be correct to rewrite this as $z_b = F_b(W_b, z_{b-1}) = z_{b-1} = \phi_b(W_b, z_{b-1})$?  -   In Section 4, it's not clear how m_{b, W} and m_{b, z} are computed for Algorithm 2.  #### Originality  Variants of back-matching propagation are explored in prior works but these did not apply this procedure to study the efficiency of vanilla backpropagation.  #### Significance  A general characterization of efficiency for deep learning architectures and training procedures would be very valuable, and the proposal offered here appears to be worth further study.


Acceleration through Optimistic No-Regret Dynamics
We consider the problem of minimizing a smooth convex function by reducing the optimization to computing the Nash equilibrium of a particular zero-sum convexconcave game. Zero-sum games can be solved using online learning dynamics, where a classical technique involves simulating two no-regret algorithms that play against each other and, after T rounds, the average iterate is guaranteed to solve the original optimization problem with error decaying as O(log T /T ). In this paper we show that the technique can be enhanced to a rate of O(1/T 2 ) by extending recent work [22,25] that leverages optimistic learning to speed up equilibrium computation. The resulting optimization algorithm derived from this analysis coincides exactly with the well-known NESTEROVACCELERATION [16] method, and indeed the same story allows us to recover several variants of the Nesterov's algorithm via small tweaks. We are also able to establish the accelerated linear rate for a function which is both strongly-convex and smooth. This methodology unifies a number of different iterative optimization methods: we show that the HEAVYBALL algorithm is precisely the non-optimistic variant of NESTEROVACCELERATION, and recent prior work already established a similar perspective on FRANKWOLFE [2, 1].

This paper shows that Nesterov’s accelerated gradient descent algorithms can be interpreted as computing a saddle point via online optimization algorithms. A convex optimization problem is transformed to be a minmax problem by the Fenchel dual, the solution of which is then approximated via online optimization algorithms.   This paper can be a significant contribution to the optimization community. I would say that this is one of the most natural interpretations of Nesterov’s accelerated gradient methods. The use of weighted regrets and (optimistic) FollowTheLeader (instead of follow the regularized leader) are a little bit artificial but acceptable. The latter is perhaps easier to accept, if the authors point out that \ell_t is strongly convex due to smoothness of f.   A possible issue is boundedness of the radius D in Section 4. To use Theorem 2 and Corollary 1 to prove the accelerated convergence rate, one has to provide an upper bound of the radius D. It is not immediate to me whether D is actually bounded or not in Section 4. (In Section 4.2, there is a constraint set K, but boundedness of K is not mentioned.)  Other comments:  1. ln 135: It seems that \ell_t(y) should be equal to -g(x_{t - 1}, y) instead of -g(x_t, y), according to Algorithm 1. Please check.  2. ln 4, Algorithm 2: The x in \nabal h_t (x) is not defined, though this is not really relevant.  3. Theorem 3: You need to specify \gamma to use Theorem 2 and Corollary 1.


A Smoother Way to Train Structured Prediction Models
We present a framework to train a structured prediction model by performing smoothing on the inference algorithm it builds upon. Smoothing overcomes the non-smoothness inherent to the maximum margin structured prediction objective, and paves the way for the use of fast primal gradient-based optimization algorithms. We illustrate the proposed framework by developing a novel primal incremental optimization algorithm for the structural support vector machine. The proposed algorithm blends an extrapolation scheme for acceleration and an adaptive smoothing scheme and builds upon the stochastic variance-reduced gradient algorithm. We establish its worst-case global complexity bound and study several practical variants. We present experimental results on two real-world problems, namely named entity recognition and visual object localization. The experimental results show that the proposed framework allows us to build upon efficient inference algorithms to develop large-scale optimization algorithms for structured prediction which can achieve competitive performance on the two real-world problems.

Overview: This paper proposes an accelerated variance-reduction algorithm for training structured predictors. In this approach the training objective is augmented with a proximal term anchored with a momentum point (eq (3)), the loss is smoothed using Nesterov's smoothing method (adding entropy or L2 to the dual), and a linear-rate solver (SVRG) is applied to the resulting objective in the inner loop. This achieves accelerated convergence rates for training.   Comments: * I think that the connection to structured prediction is somewhat weak. In particular, the analysis uses the finite sum and smoothability of the training objective. The first is not specific to structured prediction, and the latter has been studied extensively in previous work, including for learning (see next point). Inference is assumed to be an atomic operation performed via calls to an inference oracle -- assuming access to efficient top-k and expectations computation, and there is no leveraging of instance-level structure.  * I don't think it's fair to say that smooth inferene oracles are a contribution of this paper (line 40). Entropic smoothing was used much earlier, for example: - Convex Relaxation Methods for Graphical Models: Lagrangian and Maximum Entropy Approaches, Johnson 2008. - Fast and smooth: Accelerated dual decomposition for MAP inference, Jojic et al. 2010. - [Adaptive smoothing] Efficient MRF Energy Minimization via Adaptive Diminishing Smoothing, Savchynskyy et al. 2012. Also in the context of learning (SSVMs), e.g.: - Blending Learning and Inference in Conditional Random Fields; Hazan et al. JMLR 2016. L2 smoothing for inference was used more recently, for example: - Smooth and Strong: MAP Inference with Linear Convergence, Meshi et al. 2015. - [For deep structured models] From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification, Martins and Astudillo 2016.  * Top-K oracles may sometimes be more expensive than max oracles (also true for marginal oracles). It’s worth providing more details on that to clarify when it’s not a good idea to use this type of smoothing.  * In the experiments (Fig 2), the accelerated methods don't seem to have an advantage over plain SVRG in terms of test performance.   Minor comments: * Lines 62-63: the remark about the loss being decomposable is not really necessary since nothing is assumed about the score (line 58). Instead, perhaps better to directly extend the assumption on the *oracle* to include (2). Otherwise, score decomposition may be mentioned.  * Line 81: it would be good to clarify that u_2 depends on mu via notation, e.g., u_2(y;\mu) or u_\mu(y).  * Line 194 [typo]: 'set reduced set'.  ================= After author response: To better place this work in context of existing literature on structured prediction, I suggest you down-tone the contribution on smoothing of inference and cite a few works on Nesterov smoothing of inference objectives. I also think you should add a discussion on complexity of the required oracles making it clear that smoothing often requires more expensive oracles.


Amortized Inference Regularization
The variational autoencoder (VAE) is a popular model for density estimation and representation learning. Canonically, the variational principle suggests to prefer an expressive inference model so that the variational approximation is accurate. However, it is often overlooked that an overly-expressive inference model can be detrimental to the test set performance of both the amortized posterior approximator and, more importantly, the generative density estimator. In this paper, we leverage the fact that VAEs rely on amortized inference and propose techniques for amortized inference regularization (AIR) that control the smoothness of the inference model. We demonstrate that, by applying AIR, it is possible to improve VAE generalization on both inference and generative performance. Our paper challenges the belief that amortized inference is simply a mechanism for approximating maximum likelihood training and illustrates that regularization of the amortization family provides a new direction for understanding and improving generalization in VAEs.

This paper puts forward the idea that we should in certain cases regularize the generative model in VAEs in order to improve generalization properties. Since VAEs perform maximum likelihood estimation, they can in principle exhibit the same overfitting problems as any other maximum likelihood model. This paper argues that we can regularize the generative model by increasing the smoothness of the inference model. The authors consider the Denoising VAE (DVAE) as a means of achieving such regularization. In the special case where the encoder is an exponential family, they show that the optimum natural parameters for any input data can be expressed as a weighted average over the optimum parameters for the data in the training set. Simlarly for a decoder model in the exponential family, the optimum choice of mean parameters for any latent code is a weighted average over sufficient statistics of training data. Experiments consider evaluate the test set log marginal likelihood for MNIST, Omniglot, and Caltech 101 with the DVAE, a convex combination between DVAE and VAE, and a third smoothing strategy based on regularization that bounds the magnitude of encoder weights.  I think that this is a well-written paper that provides some useful insights about the interplay between encoder and decoder expressivity in VAE. Equations (14), (15) and (19) in particular provide a really nice intuition for the behavior of optimal encoders and decoders. The experiments are reasonable. I was not able to find anything amiss with the proofs (albeit on a cursory read). My only complaint about this paper is that there are some issues of clarity in the discussion of experiments approved (see questions below). Overall, this seems like a welcome addition to ongoing efforts to better understand the generalization properties of VAEs. I would be happy for this paper to appear.   Comments and Questions   - I'm not entirely sure what to make of the results in Table 1. While it seems intuitive that regularizing the inference model would improve the test set log marginal likelihood, it is less intuitive that doing so would also decrease Δ_inf, which one would naively expect to *increase*. Could the authors comment on this.  - I was not quite able to follow the definition of the regularization parameter. This should probably be discussed in the main text, rather than the appendix.   - When the authors write  "Our main experimental contribution in this section is the verification that increasing the number of importance samples results in less underfitting when the inference model is over-regularized."   It sounds a bit like they are saying that increasing the number of IWAE sample decreases susceptibility to underfitting. Is this in fact what they mean to say, or do these results simply validate Proposition 3?   - While perhaps not the point per se of this paper, equation (19) provides some intuitions that are probably worth pointing out in expliclity. If we apply (19) to a standard VAE with Bernoulli likelihood, then the mean value of each pixel will be a weighted average over pixel values of examples in the training data encode to similar z values. My read of this is that, for an optimal decoder, "interpolation" really means "averaging over nearest neighbors in the latent space".   - For a normal VAE, the mutual information I(x;z) in the inference model tends to saturate to log n (where n is the size of the training data). This suggests that the weights q(x^(i) | z) would be pretty close to 0 for all (i) except the nearest neighbor for a normal VAE (since each x^(i) encode to non-overlapping regions in z space). In other words, for a normal VAE, an optimal decoder could effectively be doing nearest-neighbor matching in the latent space. As you increase the regularization, presumably the generative model averages over a larger number of neighbors. It would be interesting to look at how the expected entropy of q(x^(i) | z) changes with the amount of regularization (the exponent of this entropy would effectively give you something like "the number of neighors" that an optimal likelihood model would average over).   - The posteriors in Figure 2 of cited ref [10] look pretty significantly non-Gaussian to me. I unfortunately lacked time to read ref [10] in detail, but I was not immediately able to find support for the claim that "it has still been observed empirically that the true posteriors tend to be approximately Gaussian by the end of training as a result of posterior regularization induced by the variational family". Could the authors either clarify, or walk back this claim?   - The notation F(Q) does not make sense to me. In this notation would the domain of F somehow be the set of all possible variational families? Given that Q is a specific family of variational distributions (in practice a parametric family like a Gaussian with diagonal covariance), and F(Q) is a family of functions that map from X to Q, it is not clear how you would even apply the same F to two different families Q and Q'. Writing F_Q, or perhaps just F would be clearer, since F is in practice specific to a family Q. The same argument applies to G(P).  - How are the errors in Table 1 defined? Is this an error over independent restarts? If so, how many restarts? Could the authors figure out a way to reformat the table so the erros are not shown in tiny fonts? I understand that we're all using screens to read papers these days, but I still think that fonts should be legible in print.   - Does Table 1 show results for the regularization parameters listed in Table 7?   - Do Figures 1 a 2 show the same IW-SVI approximation of log p_θ(x) as Table 1?  - Why does the Caltech panel of Figure 1 have error bars, whereas the Omniglot and MNIST panels do not?  - q(x^(i) | z) is never explicitly defined, as far as I can tell.   - The acronym "AIR" results in an unfortunate collision with "Attend, Infer, Repeat", which at this point is pretty well known.


SEGA: Variance Reduction via Gradient Sketching
We propose a randomized first order optimization method-SEGA (SkEtched GrAdient)-which progressively throughout its iterations builds a variancereduced estimate of the gradient from random linear measurements (sketches) of the gradient. In each iteration, SEGA updates the current estimate of the gradient through a sketch-and-project operation using the information provided by the latest sketch, and this is subsequently used to compute an unbiased estimate of the true gradient through a random relaxation procedure. This unbiased estimate is then used to perform a gradient step. Unlike standard subspace descent methods, such as coordinate descent, SEGA can be used for optimization problems with a non-separable proximal term. We provide a general convergence analysis and prove linear convergence for strongly convex objectives. In the special case of coordinate sketches, SEGA can be enhanced with various techniques such as importance sampling, minibatching and acceleration, and its rate is up to a small constant factor identical to the best-known rate of coordinate descent.

In this paper, the authors propose a randomized first order optimization method (SEGA) which progressively builds a variance reduced estimate of the gradient from random linear measurements of the gradient. The proposed method (or class of methods - depending on the sketch matrix and metric used) updates the current estimate of the gradient through a sketch-and-project operation using new gradient information and the past estimate of the gradient.  - The first half of the paper is very well written, easy to follow and well motivated. However, the quality of the paper deteriorates after page 6. The paper has minor typos and grammatical mistakes that can be corrected easily.  — Line 29: now draw a two simple -> now draw two simple — Line 36: methods as stochastic gradient -> methods such as stochastic gradient — Line 83: For the illustration, -> To illustrate this, — Line 83: shows a the evolution -> shows the evolution — Line 87: baseline choice -> baseline choices — Line 106: no proximal setting, coordinate -> no proximal term and coordinate — Line 115: eigen-properties or matrices -> eigen-properties of matrices — Line 148: analogy -> analog (and other instances of this) — Line 152: fresult -> result — Line 158: ary -> Corollary — Line 165: in sense -> in the sense — Line 181: verify claim -> verify the claim — Line 192: a construct -> a constructed problem — Line 193: system comparing -> system compared  - The Experiments section seems weak, primarily because all problems solved are constructed problems. The experiments are well though out to highlight certain algorithmic features of the method, however, several details are missing (e.g., what is the dimension n of the problems solved?), comparison with more methods would strengthen the claims made and experiments on real ML problems would highlight the merits (and limitations) of SEGA. — Section 5.1: —— The authors should mention in what settings different cost scenarios arise for solving the linear systems. —— The authors should compare the methods on problems that are not quadratic. — Section 5.2: —— How was the finite difference interval set for the gradient estimation used in SEGA? Did the authors use forward differences (n function evaluations per iteration)?  ——The authors should compare against other zero order optimization methods (e.g., model based trust region method, finite difference quasi-Newton methods or Nelder-Mead). —— Again, the authors should compare the methods on problems that are not quadratic. —— In this section do the authors assume that they have access to true (non-noisy) function evaluations? — Section 5.3: —— How is beta chosen in the accelerated variant of the method? —— In practice, on real world problems, how would one chose the subspaces to use in subspaceSEGA? The authors should make a comment about this in the paper. — The authors claim that “theory supported step sizes were chosen for all experiments.” These step sizes depend on problem specific parameters such as Trace(M), minimum and maximum eigenvalues of different matrices and other quantities. How were these quantities calculated?   - Other minor issues and questions: — The authors should mention clearly that it is not necessary to compute the full gradient in Algorithm 1. — “SEGA can be seen as a variance reduced coordinate descent.” Is it not the case that SEGA is more general than that? If certain sketch matrices are chosen then it is a coordinate descent type method, but with other sketches this is not the case.  — “potentially more expensive iteration”: it would be helpful for the reader if the authors provided some examples. — “total complexity”: do the authors mean iteration complexity? — What is alpha_0 in Corollary 3.4? — Line 128: “Corollary 4.3 and Corollary 4.3”? — Remark 4.4: “Therefore, SEGA also works in a non-convex setting.” I believe this statement is too strong. SEGA works for a certain class of non-convex function that satisfy the PL inequality (these function have unique minimizers). — Corollary 4.5: Is the Lyapunov function used here the same as in Theorem 4.2? — Table 1: What is the cause of the constant differences in the results of CD and SEGA? Is this an artifact of the analysis? Is this realized in practice? — The authors should consider adding a Final Remarks section to summarize their contributions and to aid the reader.  - I believe the generality of the algorithms the authors propose (different sketch matrices) and the special cases of algorithms that they recover, as well as the wide range of potential problem classes that can be solved by the proposed methods, makes this a paper of interest for the ML and Optimization communities. However, there are certain issues and questions with the paper (mentioned above) that should be addressed. As such, I marginally recommend this paper for publication at NIPS.  I have read the author rebuttal and stand by my initial assessment of the paper.


Adversarial Attacks on Stochastic Bandits
We study adversarial attacks that manipulate the reward signals to control the actions chosen by a stochastic multi-armed bandit algorithm. We propose the first attack against two popular bandit algorithms: ✏-greedy and UCB, without knowledge of the mean rewards. The attacker is able to spend only logarithmic effort, multiplied by a problem-specific parameter that becomes smaller as the bandit problem gets easier to attack. The result means the attacker can easily hijack the behavior of the bandit algorithm to promote or obstruct certain actions, say, a particular medical treatment. As bandits are seeing increasingly wide use in practice, our study exposes a significant security threat.

This paper presents adversarial attacks against multi-arm bandit algorithms e-greedy and UCB. This paper is very well written;                 the problem is well-motivated; the approach is sound and well explained through the guidance from an oracle approach in an                 less realistic setting to the real setting. I enjoy reading this paper.                 I list some of the aspects that can be improved below and hope they can help the authors to improve the paper. But, again, I                think the paper is publishable even in the current version. First, it would be better if the threat model can be further                 justified, i.e., why the attacker can manipulate the rewards arbitrarily? It would be good if the authors can map the attacker                to real settings.                 Second, I don't find how the attacks can be mitigated if Bob is aware of the existence of the attacker. This would be an                 interesting discussion to see to what can be an equilibrium between the attacker and the defender.                 Third, which is related to the above aspects, the authors only consider two bandit algorithms, leaving a large space of other                bandit algorithms not discussed. Thus, it is unclear whether Bob can simply choose another algorithm to defend against the                attacks? Would there be any universal results that can be said about the attacker without relying on the concrete bandit                 algorithm chosen by the defender? The authors admit that the work is still in progress, and many more bandit algorithms will                be discussed; I hope, in the future versions, there could be some general conclusions presented in par of these concrete                bandit algorithms, which might provide more tight results.


PAC-learning in the presence of evasion adversaries
The existence of evasion attacks during the test phase of machine learning algorithms represents a significant challenge to both their deployment and understanding. These attacks can be carried out by adding imperceptible perturbations to inputs to generate adversarial examples and finding effective defenses and detectors has proven to be difficult. In this paper, we step away from the attack-defense arms race and seek to understand the limits of what can be learned in the presence of an evasion adversary. In particular, we extend the Probably Approximately Correct (PAC)-learning framework to account for the presence of an adversary. We first define corrupted hypothesis classes which arise from standard binary hypothesis classes in the presence of an evasion adversary and derive the Vapnik-Chervonenkis (VC)-dimension for these, denoted as the adversarial VC-dimension. We then show that sample complexity upper bounds from the Fundamental Theorem of Statistical learning can be extended to the case of evasion adversaries, where the sample complexity is controlled by the adversarial VC-dimension. We then explicitly derive the adversarial VC-dimension for halfspace classifiers in the presence of a sample-wise norm-constrained adversary of the type commonly studied for evasion attacks and show that it is the same as the standard VC-dimension. Finally, we prove that the adversarial VC-dimension can be either larger or smaller than the standard VC-dimension depending on the hypothesis class and adversary, making it an interesting object of study in its own right.

This paper develops a PAC framework for learning binary functions in the presence of evasion adversaries. Let X be the domain, and suppose we have an unknown function f : X -> {0,1} to be learned, and we also have a hypothesis class H. Moreover, there is a "closeness" relationship defined on pairs in X, so any pair of points in X are either "close" or "far." We also have an unknown distribution P on X. For a hypothesis h, its loss is defined as follows: first we pick a data point x in X according to P. The true label of this point is f(x). If there is a point y "close" to x such that h(y) differs from f(x), then we say x is classified incorrectly under the adversary (the intuition is that, the adversary can slightly perturb the point x into point y to fool the classifier). The loss of a classifier is the probability that a random point x is classified incorrectly under the adversary.  Now that we have a loss function, we can ask the standard question in learning theory: given a finite labelled sample from the unknown distribution P, can we pick a hypothesis whose loss is close to being optimal? When there is no adversary, the answer to this question is known, and the sample complexity is characterized by the VC-dimension of the hypothesis class. In this paper, a notion of "adversarial VC-dimension" has been defined, and it is proved that if this dimension is finite, then learning is possible, and a bound is given in terms of this dimension (Theorem 1), which resembles the bound on the sample complexity in terms of VC-dimension in the non-adversarial setting. The proof involves carefully reducing the problem to an appropriate adversary-free learning problem, and using a Rademacher complexity analysis.  Then the paper proceeds to understanding the connection between adversarial VC-dimension and standard VC-dimension. An example is given where the former can be arbitrarily larger than the latter. However, it is shown that if the closeness relationship is given by a norm-derived metric, then the VC-dimension of half-space classifiers do not change by adding an adversary. This is rather surprising.  I am not familiar with the literature on adversarial learning theory, so I cannot compare the framework and results with the previous ones.  Pros: + a clean framework and analysis for binary classification in the presence of adversaries is introduced, which is likely to open doors to new, interesting research. This would be of significance interest to the statistical and learning theory community.  + the framework and proofs are novel and mathematically elegant.  + the paper is well written.  Cons: - the authors have shown that finite adversarial VC-dimension implies learnability. Is the reverse also true? Does learnability under adversary imply finite adversarial VC-dimension? (Note that this is the easy direction in binary classification without adversaries, but it is not addressed in the current paper.)  Questions and comments for the authors: * Line 89: add index n for hat(h)  (compare with line 104) * 7th line in Table 1: delete "adversarial" * Line 103: "we say" learning is possible if etc. * Line 104: if there is an algorithm that, "upon receiving n training examples", etc. * Line 104: add "as n -> infinity". * Line 105: Rewrite it as: The learner does not have access to the true distribution P, but can approximate it with the empirical distribution. * Line 108: ERM -> AERM * Line 113: be "a" learning algorithm * In Lemma 1, you assume the learning algorithm does not know R. This assumption is without loss of generality, but please explain it in the paper. * Line 131: compute -> define * Display after Line 151: the dot should be replaced with "such that" or ":" * In Definition 4, explain the connection between H and F. * Line 175: C -> tilde(C) * Elaborate on Footnote 3: how can this be improved via chaining? What is the obtained result? * Line 191: "is" is missing * Line 203: what do you mean by "other variants of halfspace classifiers"? * Line 218: inside the formula, replace the dot with "we have" == after reading other reviews and authors' response: please accommodate reviewers' comments and also explain other theoretical work on adversarial learning and how your work differs from them.


Symbolic Graph Reasoning Meets Convolutions
Beyond local convolution networks, we explore how to harness various external human knowledge for endowing the networks with the capability of semantic global reasoning. Rather than using separate graphical models (e.g. CRF) or constraints for modeling broader dependencies, we propose a new Symbolic Graph Reasoning (SGR) layer, which performs reasoning over a group of symbolic nodes whose outputs explicitly represent different properties of each semantic in a prior knowledge graph. To cooperate with local convolutions, each SGR is constituted by three modules: a) a primal local-to-semantic voting module where the features of all symbolic nodes are generated by voting from local representations; b) a graph reasoning module propagates information over knowledge graph to achieve global semantic coherency; c) a dual semantic-to-local mapping module learns new associations of the evolved symbolic nodes with local representations, and accordingly enhances local features. The SGR layer can be injected between any convolution layers and instantiated with distinct prior graphs. Extensive experiments show incorporating SGR significantly improves plain ConvNets on three semantic segmentation tasks and one image classification task. More analyses show the SGR layer learns shared symbolic representations for domains/datasets with the different label set given a universal knowledge graph, demonstrating its superior generalization capability.

This paper is about a Symbolic Graph Reasoning (SGR) layer which can be injected between convolutional layers. The SGR layer employs a knowledge graph that allows the neural network to leverage semantic constraints derived from human knowledge.   The SGR is structured in 3 modules: 1) local-to-semantic voting module which maps local features to semantic nodes (i.e. the nodes of the knowledge graph) and outputs a D^c dimensional feature vector for each semantic node in the graph, 2) a graph reasoning module in which the semantic node features are concatenated with word-vector  features (derived from linguistic embeddings) projected to lower dimensionality linear layer. Finally, each semantic node representation is replaced by the average of the node representations of its neighbors (using the dot product with the row-normalized adjacency matrix of the knowledge graph). 3) the semantic-to-local mapping module reconstructs the local features starting from the  global representation of the semantic nodes in produced in the previous module.  The paper include experiments with the SGR module on the semantic segmentation datasets such as Coco-Stuff, ADE20K and PASCAL-Context and on the classification dataset CIFAR-100. The experimental evaluation is quite convincing and I find impressive that the performance of SGR/SGR 2-layer can get very close to the one of DenseNet using 7.5M/8.1M of parameters instead of 25.6M (see Table 5).  With respect to the novelty, the closest work is the one on Dynamic-structured semantic propagation networks [27] The author claim that "Our work takes an important next step beyond prior approaches in that it directly incorporates the reasoning over external knowledge graph into local feature learning, called as Symbolic Graph Reasoning (SGR) layer." However, also [27] can incorporate a graph and yields results which are a bit lower on Coco-stuff.  Strengths: + The paper is well written and easy to read. + The performance of the method is impressive and does not seem to be tailored to the specific tasks addressed.  Weaknesses: - It not clear what is novel compared to [27], which also incorporate a graph and yields similar results which are a bit lower on Coco-stuff  Questions: Q1)  Since [27] is missing in Table 2. Is there a reason because [27] can not be run on Pascal-Context? Q2) In [27] we can see experiments on the Cityscape and the Mapillary datasets would SGR work on those datasets? Q3) "The very recent DSSPN [27] directly designs a network layer for each parent concept. However, this method is hard to scale up for large-scale concept set and results in redundant predictions for pixels that unlikely belongs to a specific concept." Q3.1 Could you provide evidence of a large-scale concept set in which your method surpasses [27]? Q3.2 Is there a relationship between the size of the concept set and the number object classes that we want to segment? Q4) Could you clarify what is meant with the term "reasoning" in SGR? In particular, could we interpret the dot product of the row-normalized adjacency matrix with the semantic node features as a forward chaining step in an inference engine?   Minor Issues: - "people wear a hat and play guitar not vice-versa;" why not? - "orange is yellow color." orange is orange not yellow, unless I am missing something. - "Comparison with the state-of-the-arts" -> "Comparison with the state of the art"  —————— My concerns are all well addressed in the rebuttal. My opinion of the paper is increased.


Lifted Weighted Mini-Bucket
Many graphical models, such as Markov Logic Networks (MLNs) with evidence, possess highly symmetric substructures but no exact symmetries. Unfortunately, there are few principled methods that exploit these symmetric substructures to perform efficient approximate inference. In this paper, we present a lifted variant of the Weighted Mini-Bucket elimination algorithm which provides a principled way to (i) exploit the highly symmetric substructure of MLN models, and (ii) incorporate high-order inference terms which are necessary for high quality approximate inference. Our method has significant control over the accuracy-time trade-off of the approximation, allowing us to generate any-time approximations. Experimental results demonstrate the utility of this class of approximations, especially in models with strong repulsive potentials.

The paper presents a lifted version of the weighted mini-bucket elimination algorithm.   Exploiting symmetries within probabilistic inference is important. In particular, with respect to the present paper, the standard approach for inference in probabilistic formalisms with first-order constructs is lifted variable elimination (LVE) for single queries. To handle multiple queries efficiently, the lifted junction tree algorithm (LJT) employs a first-order cluster representation of a model and LVE as a subroutine. The present paper extends this family of approach towards a lifted variant of  weighted mini-bucket. That is, it shows how to employ its tighter bounds for probabilistic inference. This is interesting.  And yes, the paper mentions major related work. However, it doe not discuss some important related work at a sufficient level of details. Consider e.g. [5, 13] but also   Prithviraj Sen, Amol Deshpande, Lise Getoor: Bisimulation-based Approximate Lifted Inference. UAI 2009: 496-505  In terms of the bound used, as it is closely related to counting numbers, the authors should clarify the connection to [13] and related approaches in more details.  This line of research has also shown that symmetries can be exploited for concave  energies and for variational optimization of them. This discussion is missing.  Also, the authors should mention  Seyed Mehran Kazemi, David Poole: Knowledge Compilation for Lifted Probabilistic Inference: Compiling to a Low-Level Language.  KR 2016: 561-564   David B. Smith, Parag Singla, Vibhav Gogate: Lifted Region-Based Belief Propagation.  CoRR abs/1606.09637 (2016)   As far as I see it, the underlying theory presented, next to the join operations, are very similar if not identical. The authors should develop where (on the underlying algebraic level) the differences are. As far as I see it, the present paper introduces a novel approach to compute counting numbers, which could then be used e.g. in Theorem 9 of [13].  This is also related to the asymmetric case presented. The sequential case is interesting but not discuss in detail. The unary evidence case is very much related to the domain graph presented by Bui et al. (see above) as well as by   Martin Mladenov, Kristian Kersting, Amir Globerson: Efficient Lifting of MAP LP Relaxations Using k-Locality.  AISTATS 2014: 623-632  Overall, however, the contribution and also the particular combination appears to be very interesting. The experimental evaluation falls a little bit too short as no real world data has been considered. Would be great to add something evaluations here.   Nevertheless, overall, the paper makes a nice progress for lifted inference. The main downside is the weak discussion of related work, which makes it hard to see the novel contributions. This is also reflected in the fact that now summary of the results are presented. The experimental evaluation is too short for the claims raised in the intro, but still kind of fine. I am confident that this will work on other models, too.   Furthermore, there are forthcoming IJCAI 2018 papers such as   Tanya Braun, Ralf Möller Parameterised Queries and Lifted Query Answering IJCAI 2018  They introduce the idea of parameterised queries as a means to avoid groundings,  applying the lifting idea to queries. Parameterised queries enable LVE and LJT  to compute answers faster, while compactly representing queries and answers. Would be great if the authors can mention them, although the work was indeed done independently.


Non-Ergodic Alternating Proximal Augmented Lagrangian Algorithms with Optimal Rates
We develop two new non-ergodic alternating proximal augmented Lagrangian algorithms (NEAPAL) to solve a class of nonsmooth constrained convex optimization problems. Our approach relies on a novel combination of the augmented Lagrangian framework, alternating/linearization scheme, Nesterov's acceleration techniques, and adaptive strategy for parameters. Our algorithms have several new features compared to existing methods. Firstly, they have a Nesterov's acceleration step on the primal variables compared to the dual one in several methods in the literature. Secondly, they achieve non-ergodic optimal convergence rates under standard assumptions, i.e. an O 1 k rate without any smoothness or strong convexity-type assumption, or an O 1 k 2 rate under only semi-strong convexity, where k is the iteration counter. Thirdly, they preserve or have better per-iteration complexity compared to existing algorithms. Fourthly, they can be implemented in a parallel fashion. Finally, all the parameters are adaptively updated without heuristic tuning. We verify our algorithms on different numerical examples and compare them with some state-of-the-art methods.

Summary:   The paper proposed 4 new variants of Augmented Lagrangian methods, which called NAPALA (non-ergodic alternating proximal augmented Lagrangian algorithms) to solve non-smooth constrained convex optimization problems under different problem structure assumptions. The first algorithm only requires f and g to be convex but neither necessarily strongly convex nor smooth. Especially, its parallel variant (8) has more advantages. The second algorithm requires either f or g to be strongly convex. Its variant (12) also allows to have parallel steps. The author(s) could derive the convergence rate at the last iterate (non-ergodic), which are the state-of-the-art results. Numerical experiments also show good overall performance of the proposed algorithms compared to other algorithms.   Comments:   In general, I really like the results of the paper since it could achieve the convergence rate at the last iterate (non-ergodic sense), which is very useful for sparse and low rank optimization or image processing. In the non-strongly convex case, there exist many algorithms that can achieve O(1/k) convergence rate, but such a rate is often in an ergodic sense. Note that ergodic sequences possibly break the sparsity and low rankness in sparse and low-rank optimization, respectively. Especially, in image processing, ergodic sequences may destroy the sharp edge structure of images. Although the convergence rate of ADMM (O(1/k)) and its accelerated versions is widely studied in the literatures, this paper proposes 4 novel algorithms which I have not seen in the literature. Compared to existing methods in the literature, the parallel variant (8) seems to have several advantages while achieving O(1/k)-non-ergodic optimal rate under only convexity and strong duality. Algorithm 2 allows using different sequence to achieve either ergodic or nonergodic rate on y, but nonergodic rate on x. It also linearizes the y-subproblem compared to ADMM. I believe that the results of the paper are non-trivial and have significant theoretical contribution to optimization and machine learning community.   Besides the above mentioned positive points, I also have the following comments:   1) I saw that you are able to achieve the convergence rate of O(1/k^2) in the case that either f or g is strongly convex (section 4.1). However, when you do parallel step, you need f and g are strongly convex (together) to have the rate O(1/k^2). What do you think about the convergence rate when either f or g is strongly convex in this parallel step? Do you think it is still possible to achieve O(1/k^2)? Have you tried to do experiments to see the behavior?  2) Could you please explain more clearly about Section 4.3? What should we do if f and/or g are smooth?   3) A minor comment: I think your algorithms could also expand to coordinate descent. What is your opinion about this?


On the Convergence and Robustness of Training GANs with Regularized Optimal Transport
Generative Adversarial Networks (GANs) are one of the most practical methods for learning data distributions. A popular GAN formulation is based on the use of Wasserstein distance as a metric between probability distributions. Unfortunately, minimizing the Wasserstein distance between the data distribution and the generative model distribution is a computationally challenging problem as its objective is non-convex, non-smooth, and even hard to compute. In this work, we show that obtaining gradient information of the smoothed Wasserstein GAN formulation, which is based on regularized Optimal Transport (OT), is computationally effortless and hence one can apply first order optimization methods to minimize this objective. Consequently, we establish theoretical convergence guarantee to stationarity for a proposed class of GAN optimization algorithms. Unlike the original non-smooth formulation, our algorithm only requires solving the discriminator to approximate optimality. We apply our method to learning MNIST digits as well as CIFAR-10 images. Our experiments show that our method is computationally efficient and generates images comparable to the state of the art algorithms given the same architecture and computational power.

SUMMARY  The authors investigate the task of training a Generative Adversarial Networks model based on optimal transport (OT) loss. They focus on regularized OT losses, and show that approximate gradients of these losses can be obtained by approximately solving regularized OT problem (Thm 4.1). As a consequence, a non-convex stochastic gradient method for minimizing this loss has a provable convergence rate to stationarity (Thm 4.2). The analysis also applies to Sinkhorn losses. The authors then explore numerically the behavior of a practical algorithm where the dual variable are parametrized by neural networks (the theory does not immediately apply because estimating the loss gradient becomes non-convex).  QUALITY I find that the point of view adopted by the authors is relevant. Training GAN is known to suffer from instabilities and here the authors provide theoretical tools to understand how to obtain gradients of the loss with a bounded error. While this relation between regularization and stability is not new in optimization/statistics, it seems to be invoked with proofs in this specific application for the first time. Yet, the authors provide numerical results on simple datasets which are interpretable.   CLARITY The article is well written, well organized, and easy to read.   ORIGINALITY The theoretical results are new in this setting but rely on well known principles. The numerical algorithm is, I believe, a minor variation of previously proposed Wasserstein GAN algorithms.  SIGNIFICANCE The paper does not provide a strikingly new point of view, but make reliable progress in the direction of robust training for GANs.  SPECIFIC REMARKS - l.130 "for practical considerations the dual functions belong to the set of parametric functions": this is likely to considerably change the (geometrical and statistical) properties of the distance and is instrumental. I would be in favor that the community as a whole emphasize more on the differences between these "parametrized Wasserstein" and the classical Wasserstein distances (I just state this as an opinion). - l.167 "although pi may not be a feasible transport plan": it is feasible, as a solution to Eq.(4). - l.174 again, I think the switch to parametric dual function cannot be reduced to a mere computational trick; - theorem 3.1 it seems that there is redundancy in the second and third conditions. The definition of L-Lipschitz smooth could maybe be given? - proof of Th.3.1 (in Appendix C): why is the 1-norm of plans appearing (Jensen's inequality should be explicitly invoked I guess) (l.449 typo  after nabla). - l.200 typo "the its" - l.272 I'd say that neural network falls also in the class of parametric approach. - l.273 I didn't understand why gradients are unbiased, is there any result supporting this claim ? This is a very important claim. - l.292 this is a very (!) strong conjecture... in any case, Lemma 4.3 is not supporting this conjecture since it is just a sufficient stationarity condition. - l.299 again, it is not predicted by Lemma 4.3 which does not characterize stationary points in general.  [UPDATE AFTER REBUTTAL I have read the reviews and authors' response. As reviewer #2, I thought that the setting was that of arbitrary probability measures (including "continuous" marginals with densities), as implied by the notations of the paper. I am not entirely convinced by the comment in author's response stating that "discrete distributions are the case in practice where distributions are represented by their samples" because the smoothness should hold for the "expected" risk, although the algorithm only uses samples. Looking at the proof, it is true that it lacks some details required to correctly deal with the continuous case (details to apply Danskin’s theorem, choices of topologies). Some technical difficulties might appear although I do not expect the results to be different. The authors in the rebuttal claim that they can adapt the proof to fix these issues. Therefore, I still lean towards acceptance although I am slightly less convinced than before, because of this confusion between continuous and discrete setting.]


Robust Subspace Approximation in a Stream
We study robust subspace estimation in the streaming and distributed settings. Given a set of n data points
and an integer k, we wish to find a linear subspace S of dimension k for which
is minimized, where dist(S, x) := min y∈S ∥x − y∥ 2 , and M (·) is some loss function. When M is the identity function, S gives a subspace that is more robust to outliers than that provided by the truncated SVD. Though the problem is NP-hard, it is approximable within a (1 + ϵ) factor in polynomial time when k and ϵ are constant. We give the first sublinear approximation algorithm for this problem in the turnstile streaming and arbitrary partition distributed models, achieving the same time guarantees as in the offline case. Our algorithm is the first based entirely on oblivious dimensionality reduction, and significantly simplifies prior methods for this problem, which held in neither the streaming nor distributed models.

This paper studies the use of oblivious dimensionality reduction for approximating a point set with a subspace. It gives a sketching and solve algorithm that reduces both the number of points, and the dimension that they are in, to numbers close to the dimensional of the goal subspace. It then empirically demonstrates that the proposed methods perform better than SVDs.  While subspace approximation is important, I’m doubtful of the value of this work for several reasons. First, unlike previous results on oblivious subspace embeddings that introduced new tools, this paper appears to be almost entirely applying existing tools to a mathematically natural variant of the problem. It does not discuss the connections and applications related to this problem. The experiments are also restricted on small synthetic data, and does not compare against natural heuristics such as simulated annealing and local methods. The experiment section also does not discuss the parameters that the projections are invoked with, and the role of table (b) is not very well explained.  Overall, while this paper contains good theoretical ideas, it can benefit from a more thorough set of experiments on practically motivated datasets (although good steps in this direction were taken in the Author Feedback), as well as a more thorough treatment of the importance of the problem studied. On the other hand, due to the fundamental importance of subspace approximation problems in numerical linear algebra, I would not object to this paper appearing.


Contextual Combinatorial Multi-armed Bandits with Volatile Arms and Submodular Reward
In this paper, we study the stochastic contextual combinatorial multi-armed bandit (CC-MAB) framework that is tailored for volatile arms and submodular reward functions. CC-MAB inherits properties from both contextual bandit and combinatorial bandit: it aims to select a set of arms in each round based on the side information (a.k.a. context) associated with the arms. By "volatile arms", we mean that the available arms to select from in each round may change; and by "submodular rewards", we mean that the total reward achieved by selected arms is not a simple sum of individual rewards but demonstrates a feature of diminishing returns determined by the relations between selected arms (e.g. relevance and redundancy). Volatile arms and submodular rewards are often seen in many real-world applications, e.g. recommender systems and crowdsourcing, in which multi-armed bandit (MAB) based strategies are extensively applied. Although there exist works that investigate these issues separately based on standard MAB, jointly considering all these issues in a single MAB problem requires very different algorithm design and regret analysis. Our algorithm CC-MAB provides an online decision-making policy in a contextual and combinatorial bandit setting and effectively addresses the issues raised by volatile arms and submodular reward functions. The proposed algorithm is proved to achieve O(cT 2α+D 3α+D log(T )) regret after a span of T rounds. The performance of CC-MAB is evaluated by experiments conducted on a realworld crowdsourcing dataset, and the result shows that our algorithm outperforms the prior art.

This paper studies the stochastic contextual combinatorial multi-armed bandit with volatile arms (i.e. arms can become available or unavailable in each round) and submodular rewards (i.e the reward observed for the selected arms is not a sum of their individual rewards, but a diminishing function of their returns, dependent on the relationship between selected arms).  The authors very clearly introduce the reader to the problem and to the key relevant topics (the contextual, combinatorial and volatile bandits, and submodular functions). They successfully explain their main contribution (a contextual combinatorial multi-armed bandit algorithm compatible with submodular reward functions and volatile arms), by explicitly stating the novelty of their work: "[17] considers a special submodular reward function that depends on a linear combination of individual rewards and assumes a constant set of arms which is very different from the volatile arms in our work".  As such, the significance of their work is important, since they consider (a) general submodular reward functions (as in equation (1)), and (2) allow for arms to dynamically become available or unavailable, all within the contextual-combinatorial bandit setting.  Their proposed algorithm is based on ideas from the contextual bandit with similarity information literature. They divide the arms into different groups based on their context, learn the expected quality/reward for each group of arms, and assume that arms with similar context have similar properties (as in Lipschitz bandits via the Holder condition).  The authors propose a heuristic to explore under-explored groups (hypercubes in space), in combination with the greedy algorithm (1) to solve the submodular function maximization.  The key parameters of their algorithm are the hypercube partition sizes $h_T$ and the control policy $K(t)$, which they specify in connection with their theoretical regret bound. Some insights into the tightness of the bound, and its connection to the stochastic pattern of the arm arrival are provided. The authors have provided in their rebuttal further comments on the impact of context space dimension in the regret bound, which would be interesting to add to the final manuscript.      However, it is unclear how realistic the budget assumption in corollary 1 is. The authors argue in their rebuttal that it is realistic in some practical applications (such as crowdsourcing). However, it is likely that in other applications, having a smaller budget than the available arms at every time instant would not apply. Thus, it would be interesting if the authors include such discussion on their final manuscript. Nevertheless, further analysis (both theoretical and empirical) of realistic arm arrival patterns would increase the impact of their work.  The evaluation section focuses on a real problem of crowdsourcing, which is specifically re-casted to fit their problem formulation. They conclude that (a) contextual-combinatorial bandits perform better than non-contextual ones, (2) not considering submodularity incurs an obvious reward loss (because the underlying reward is indeed submodular); and (3) increasing the budget reduces regret (no learning required). These results are not surprising and in some sense expected, since the problem set up is specifically targeted to the designed framework.  On the one hand, the authors have agreed to extend the evaluation of their algorithm to longer runs (as only 200 time slots are shown in Figure 3). By doing so, they can empirically evaluate the presented regret bound (as it is not clear from the provided figures whether it is sublinear, or how it compares to the theoretical bound). On the other, I would still encourage the authors to consider simulated scenarios where they can empirically evaluate the impact of stochastic pattern of the arm arrivals.  Finally, their algorithm is based on static partitions of the space (hypercubes $h_t$). I thank the authors for the insights provided in their rebuttal regarding (a) the incurred algorithm complexity and (b) extending the partition to be non-homogeneous and/or dynamic. As such, I encourage the authors to add them to the final manuscript.