Infinite-Horizon Gaussian Processes
Gaussian processes provide a flexible framework for forecasting, removing noise, and interpreting long temporal datasets. State space modelling (Kalman filtering) enables these non-parametric models to be deployed on long datasets by reducing the complexity to linear in the number of data points. The complexity is still cubic in the state dimension m which is an impediment to practical application. In certain special cases (Gaussian likelihood, regular spacing) the GP posterior will reach a steady posterior state when the data are very long. We leverage this and formulate an inference scheme for GPs with general likelihoods, where inference is based on single-sweep EP (assumed density filtering). The infinite-horizon model tackles the cubic cost in the state dimensionality and reduces the cost in the state dimension m to O(m 2 ) per data point. The model is extended to online-learning of hyperparameters. We show examples for large finite-length modelling problems, and present how the method runs in real-time on a smartphone on a continuous data stream updated at 100 Hz.

This submission proposes to use the stationary distribution of the stochastic function under the GP model for the prediction of streaming data. The required matrices follow the DARE, and the authors propose to solve for these using standard matrix libraries. The method is extended to non-Gaussian likelihoods. Hyper-parameter adjustments based on mini-batches are proposed. Experimental results on four real data sets are given.  [Quality] This paper gives a new approximation approach to the state-space formulation of Gaussian processes, and it is especially useful for streaming data using processes with comparatively shorter length scales. However, the "on-line learning of hyper parameter" (abstract, L34 and S3.3) is much more appropriately termed as re-estimation of hyper parameter based on mini-batches. This approach to hyper-parameter adjustments is trivial and not interesting --- I suggest that the authors move this part to the appendix and devote more space to improve the clarity of the paper (see points below). The results in sections 4.2 to 4.4 are convincing applications of the proposed approximation.  [Clarity] The paper is very unclear in certain critical aspects. 1. L25 --- the term "kernel components" is vague and not defined. 2. The term "boundaries" is used at several places but not well-defined at all. 3. For ADF, the authors should also cite the work by Casto and Opper's Sparse Online Gaussian processes, 2002. 4. L69. The authors need to expand on "scale poorly in long time-series models by still needing to fill the extending domain", since this seems to be an important motivation of the approach. 5. L72. Need to be more precise for "an optimal representation (without approximations) is rewriting". 6. L129. What are the function names in some of the matrix libraries? 7. L155-157: The "convolutional interpolation" needs to be expanded on --- the way it is used here is for $\gamma$, which is different from the way it is applied in [33]. 8. In section 4.1, the details of the experiments, for example, the example functions are not given in the main paper nor the supplementary material. The short forms in Table 1 needs to be explained. Also, are the MAE on the hyperparameters and not the prediction? 9. L225: The text needs to explain more how "Fig. 5 shows that .. reproduce".  That is, what is being compared in order to show this? 10. L244: How are the separate learning rates used?  [Originality] Section 3 (other than 3.3) is original: the use of the stationary distribution, and the use of DARE to find the stationary solution.  [Significance] I deem is work to be of significance, especially when the such streaming data is becoming more common.  Minor points [I would have given an 8, if not for so many of these minor points] a. There should be a better wording than saying "data are very long". b. L29 statepaces c. L39: there is be a more appropriate reference for general non-Gaussian likelihoods than [18], which focuses purely on binary classification. d. The word "latent" is used in various places as a noun, when it is an adjective. e. L66. genral f. L68. they they g. Section 2.1 seems to be based almost entirely on [19] or [26] --- suggest the authors to cite these here. h. L118: convereges j. L155: insipired k. L250: This only applies to stationary GPs.  [Comments on author's reply] Unfortunately, I have become more confused with the clarification that "kernel components" are the summands in additive kernels, and with the reply the R2 from lines 6 to 11. First, the paper have not given any explicit example or experiments on "additive kernels". Second, I am not sure how much Markovian structure is retained after kernel addition, so the state-space representation may in this case become sub-optimal.


Early Stopping for Nonparametric Testing
Early stopping of iterative algorithms is an algorithmic regularization method to avoid over-fitting in estimation and classification. In this paper, we show that early stopping can also be applied to obtain the minimax optimal testing in a general non-parametric setup. Specifically, a Wald-type test statistic is obtained based on an iterated estimate produced by functional gradient descent algorithms in a reproducing kernel Hilbert space. A notable contribution is to establish a "sharp" stopping rule: when the number of iterations achieves an optimal order, testing optimality is achievable; otherwise, testing optimality becomes impossible. As a by-product, a similar sharpness result is also derived for minimax optimal estimation under early stopping. All obtained results hold for various kernel classes, including Sobolev smoothness classes and Gaussian kernel classes.

Summary: This is a very interesting paper that provides an alternative approach to nonparametric testing. Instead of imposing a penalty to encourage bias-variance tradeoff, the paper proposed to do an “early stopping” to achieve that. The reason why this approach would work is that when applying a gradient ascent/descent approach, even the optimal solution will overfit the data, on the way to the optimal, there is a sweet spot where the variability has been removed a lot while the bias is still not too large (have not yet overfit the data).  Overall, this is a nice and well-written paper and its idea is worth spreading in the community of machine learning and statistics.    Detail comments: 1. In Theorem 3.1 and soon after it, there is a rule for testing H0 using the asymptotic distribution.  However, if I understand it correctly, this rule will be asymptotically valid when both t and n goes to infinity.  Will the stopping rule T^* diverges when n goes to infinity?  Otherwise it may not really control the asymptotic type-1 error.   2. Although the power analysis has been done for the rejection rule, the type-1 error at t=T^* was not clear. In particular, the stopping rule T^* also depends on the data. So Theorem 3.1 may not be enough to guard the type-1 error. Is there any theoretical results about the control of type-1 error?  There might be two possible ways to do that. One is to derive the asymptotic distribution at t=T^* when n goes to infinity.  The other is via the optional stopping theorem.   3. Will this method be applicable to estimation problem? I guess the main challenge is to find the right time to stop the gradient ascent so the stopping has to be re-defined. Maybe the idea of Lepski’s adaptive rule could be useful in this case.


Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels
Deep learning with noisy labels is practically challenging, as the capacity of deep models is so high that they can totally memorize these noisy labels sooner or later during training. Nonetheless, recent studies on the memorization effects of deep neural networks show that they would first memorize training data of clean labels and then those of noisy labels. Therefore in this paper, we propose a new deep learning paradigm called "Co-teaching" for combating with noisy labels. Namely, we train two deep neural networks simultaneously, and let them teach each other given every mini-batch: firstly, each network feeds forward all data and selects some data of possibly clean labels; secondly, two networks communicate with each other what data in this mini-batch should be used for training; finally, each network back propagates the data selected by its peer network and updates itself. Empirical results on noisy versions of MNIST, CIFAR-10 and CIFAR-100 demonstrate that Co-teaching is much superior to the state-of-the-art methods in the robustness of trained deep models.

This paper proposes and empirically investigates a co-teaching learning strategy that makes use of two networks to cross-train data with noisy labels for robust learning. Experiments on MNIST, CIFAR10 and CIFAR100 seem to be supportive. The paper is well structured and well written, but technically I have following concerns.  1. Since the proposed co-teaching strategy is motivated by co-training, one can tell the similarity between the two. Typically, in a semi-supervised learning setup by co-training, the labeled and unlabeled data is known. It starts with the labeled data and improves two models based on the confidence of the unlabeled data. In the proposed co-teaching setup, however, whether a label is clean or noisy is unknown. So co-teaching starts with some "clean" labels and gradually builds two good models to deal with the noisy labels. Otherwise, co-teaching works very similar to co-training.  2. Since the quality of the labels is unknown, co-teaching has to estimate a few variables, for instance, the confidence of the labels and the noise rate. The former is done by so-called "small-loss", which is argued to give rise to "support vectors" and the latter is simply assumed to be known ($\tau$) in the experiments. This is one of my concerns. First of all, it is not clear how the estimate of $\tau$ would impact the performance of co-teaching. It would be helpful to show some results. Second, there is an implicit assumption behind co-teaching. That is the clean and noisy labels are uniformly distributed across mini-batches. In practice, both assumptions may not hold. Therefore, it is important to show the results with estimated $\tau$.  3. "dealing with extremely noisy labels" is claimed to be the focus of this paper. However, it is not clear what noisy rate is considered "extremely noisy". It may be a reasonable speculation that noise rate >50% is extremely noisy. Furthermore, why does co-teaching only work well for extremely noisy labels while not for slightly noisy labels? Wouldn't it be reasonable to show its behavior through a spectrum of noise rate, say, from clean, 10%, 20%, all the way to 80% (in a controlled manner, for instance, only on one dataset)?  4. Co-training has a very good theory backing it. The proposed co-teaching in this paper is basically empirical observations. It would be nice to come up with some similar analysis as co-training.  Overall, I find this paper clearly written. The idea is interesting but significance of the reported work is not overwhelming. There are some fundamental issues regarding the approach and the experimental results that need to be clarified.


Inexact trust-region algorithms on Riemannian manifolds
We consider an inexact variant of the popular Riemannian trust-region algorithm for structured big-data minimization problems. The proposed algorithm approximates the gradient and the Hessian in addition to the solution of a trust-region sub-problem. Addressing large-scale finite-sum problems, we specifically propose sub-sampled algorithms with a fixed bound on sub-sampled Hessian and gradient sizes, where the gradient and Hessian are computed by a random sampling technique. Numerical evaluations demonstrate that the proposed algorithms outperform state-of-the-art Riemannian deterministic and stochastic gradient algorithms across different applications.

This paper proposed the inexact TR optimization algorithm on manifold, in which the analysis largely follows from the euclidean case [27,28]. They also considered the subsampling variant of their method for finite-sum problems and the empirical performance on PCA and matrix completion problems.   1. It seems to me the convergence analysis follows exactly from the works [27,28] except for some different notation. At least this paper doesn't show what is the technical challenge for the Riemannian case.   2. For the experiments, both MC and PCA problems can be solved efficiently by Euclidean method like GD/SGD. What are the advantages of using Riemannian optimization method?   3. In appendix, line 410, C should not depend on \delta_g or \delta_H otherwise it will depend on \eps_g and \eps_H.  Overall, it is a good paper with very interesting results. The presentation can be further improved.  First, show some real motivation for doing inexact RTR method. Right now the intro is a little bit general and vague.  Second, show the technical challenges/difficulties of obtaining the iteration complexity results and the analysis should be elaborated a little bit. Third, show some background/examples on how to compute the gradient and Hessian on manifold, i.e. how you can do it efficiently in practice for certain problems.


Foreground Clustering for Joint Segmentation and Localization in Videos and Images
This paper presents a novel framework in which video/image segmentation and localization are cast into a single optimization problem that integrates information from low level appearance cues with that of high level localization cues in a very weakly supervised manner. The proposed framework leverages two representations at different levels, exploits the spatial relationship between bounding boxes and superpixels as linear constraints and simultaneously discriminates between foreground and background at bounding box and superpixel level. Different from previous approaches that mainly rely on discriminative clustering, we incorporate a foreground model that minimizes the histogram difference of an object across all image frames. Exploiting the geometric relation between the superpixels and bounding boxes enables the transfer of segmentation cues to improve localization output and vice-versa. Inclusion of the foreground model generalizes our discriminative framework to video data where the background tends to be similar and thus, not discriminative. We demonstrate the effectiveness of our unified framework on the YouTube Object video dataset, Internet Object Discovery dataset and Pascal VOC 2007.

The paper is good, presents a new idea that achieves excellent results at the level of performance. I'm familiar with these types of techniques but I can not quite understand why has been used the histogram metric rather than some gradient-based technique (e.g., Gradient Edge Detection) to differentiate an object across all image.  The submission present a video/image segmentation and localization framework for learns to localize and segment common objects in images and videos using a discriminative clustering model. The originality of the work is focused on: 1) Introduce a foreground/beckground discriminative model nased on histogram matching term. This foreground model are incorporated into discriminative clustering one and optimizated together. 3) Show a novel mechanism to exploit spatial relation between superpixel and bounding box in an unsupervised way.  The technical content of the paper appears to be correct but the authors did not highlight well both the strengths and weaknesses of their work.  It's generally well-written and structured clearly. Maybe it's too structured in the paragraph 6. In this case, I recommend making the text more fluid.   I don't have any minor remarks and typos.


Multi-Task Zipping via Layer-wise Neuron Sharing
Future mobile devices are anticipated to perceive, understand and react to the world on their own by running multiple correlated deep neural networks on-device. Yet the complexity of these neural networks needs to be trimmed down both withinmodel and cross-model to fit in mobile storage and memory. Previous studies squeeze the redundancy within a single model. In this work, we aim to reduce the redundancy across multiple models. We propose Multi-Task Zipping (MTZ), a framework to automatically merge correlated, pre-trained deep neural networks for cross-model compression. Central in MTZ is a layer-wise neuron sharing and incoming weight updating scheme that induces a minimal change in the error function. MTZ inherits information from each model and demands light retraining to re-boost the accuracy of individual tasks. Evaluations show that MTZ is able to fully merge the hidden layers of two VGG-16 networks with a 3.18% increase in the test error averaged on ImageNet and CelebA, or share 39.61% parameters between the two networks with < 0.5% increase in the test errors for both tasks. The number of iterations to retrain the combined network is at least 17.8× lower than that of training a single VGG-16 network. Moreover, experiments show that MTZ is also able to effectively merge multiple residual networks.

The paper considers the following problem: given a collection of pretrained deep models for different tasks, how to obtain a shared/joint compressed model? Authors work under the assumption that all models have same architecture and propose an approach to identify hidden units across corresponding layers in these models that can be merged together. The criterion for merging looks at the errors introduced in the pre-activations of the units when their corresponding incoming weights are merged -- hidden unit pairs (for the case of two pretrained networks) which result in lowest error are merged. Experiments with LeNets on MNIST and VGG-16 on Imagenet and CelebA are reported to show the effectiveness of the method.   Strengths:  The problem setting considered in the paper seems novel as well as reasonable from a practical perspective.   Weaknesses: The empirical evaluation of the proposed method is not comprehensive enough in my view -  (i) some natural baselines are not considered, eg, the paper starts with trained theta_1... theta_n and merges them in a shared model (theta) - a natural alternative is start with a shared model (either fully shared or usual branching architectures) and train it jointly on all tasks -- how does it perform in comparison to the proposed method on the error vs model-size plane?  (ii) Experiments in Sec 4.1 (two LeNet models, both trained on MNIST) do not convey much information in my view. I am missing the point in merging two models trained on the same task (even with different seeds). (iii) Experiments in sec 4.2 are done the VGG 16 which is a quite heavy model to begin with. It's hard to judge the significance of the results due to lack of baselines. Authors should show model size and accuracy numbers for some recently proposed multitask deep models (for the same set of tasks - imagenet and celebA) to make it easier for the reader.  (iv) All experiments and even the formulation in 3.3 is for two models. How does the method extend to more than two? (v) Algorithm 1 refers to d(w^A, w^B) and f(w^A, w^B). Where are they defined in Sec 3.3? I couldn't find a definition for these.  (vi) The paper claims that retraining needed after the merging is quite fast. What is the time needed for merging? From Algorithm 1, it looks like the method needs L forward passes for all training samples to merge L layers (as after merging a layer 'i', the activations for layer 'i+1' need to be recomputed)?  =================================== I looked at the author response and I still feel many of my concerns are not addressed satisfactorily, particularly the following points that I raised in my initial review.  (i) Natural baseline: I am not fully convinced about the reasons the authors cite for not including this baseline in the first draft. If the results of the baseline were worse than MTZ, it would've been still useful to include the results in the paper and discuss/explain why it performs worse than MTZ in the case of fully shared model. I think this is an interesting observation on its own which deserves discussion, since one would expect joint training of fully shared model to perform better than the two step optimization process of first training individual models and then merging them.   (ii) Same task with different random seeds: I am willing to buy the rationale given in the author response for this. However I would've still liked to see experiments for this setting using more recent architectures that authors use for experiments in Sec 4.2 with same datasets (ideally, ResNets or something more recent..). It is not clear why the paper uses LeNets + mnist.   (iii) More recent architectures and other competitive baselines: Author response has some results on MTZ applied on ResNets. Again, the fully shared baseline is missing from the table. My other point about comparing with recent multitask approaches (which also result in a compressed model) still remains unaddressed. It is difficult to assess the effectiveness of the proposed method (in terms of the trade-off it provides b/w final model size and accuracy) in the absence of any comparison. The absence of any baselines is perhaps the weakest point of the current draft in my view.   (iv) (v) I am fine with the explanation provided in the author response for these.   (vi) Number of fwd passes: If I understand correctly, to apply the approach with a single forward pass, one needs to store *all* the activations of previous layer in the memory so that these can be fwd-propagated after zipping the current layer. I couldn't find details in the paper on how many input examples are used for the zipping process. Author response just mentions "Moreover, only a small part of the whole training dataset is needed to generate an accurate enough Hessian matrix as mentioned in [8].", without giving details on number of examples used in the experiments. Algorithm 1 "input" on the other hand mentions "training datum of task A and B" which gives a feeling that full data is used during MTZ.


Bayesian Semi-supervised Learning with Graph Gaussian Processes
We propose a data-efficient Gaussian process-based Bayesian approach to the semisupervised learning problem on graphs. The proposed model shows extremely competitive performance when compared to the state-of-the-art graph neural networks on semi-supervised learning benchmark experiments, and outperforms the neural networks in active learning experiments where labels are scarce. Furthermore, the model does not require a validation data set for early stopping to control over-fitting. Our model can be viewed as an instance of empirical distribution regression weighted locally by network connectivity. We further motivate the intuitive construction of the model with a Bayesian linear model interpretation where the node features are filtered by an operator related to the graph Laplacian. The method can be easily implemented by adapting off-the-shelf scalable variational inference algorithms for Gaussian processes.

## [Updated after author feedback] Thank you for the feedback. The explanation in L32-L34 is very helpful. I hope you will add it to the main paper. I am happy that figure 2 now has error bars and the note on the small errors on the SOPT-* models is informative. It is good to see that the SOPT-* results are indeed significantly different. I hope, however, that you will also update table 1 in the main paper to show the standard deviations on the 10 random trials.  ## Summary The paper presents a Gaussian process-based model for semi-supervised learning on graphs (GGP). The model consists of a novel covariance function, which considers nodes in the 1-hop neighbourhood. The authors further present two distinct views on this new covariance function, namely that of a kernel mean embedding and that of a Bayesian linear model relating to the graph Laplacian. The latter is remarked as being particularly interesting since it can be viewed in terms of a spectral filter, which can be designed to have attractive properties. Finally, the authors cast the model in a variational inference framework with inducing inputs, thus allowing the model to scale to large graphs.  ## Quality The paper is of good technical quality. The authors provide discussions and explanations along the way, providing good insights for the reader.  The experiments section show a very fair comparison with graph convolutional neural networks by replicating an experiment where these are known to work well. The GGP model is thus compared to nine other methods in a semi-supervised setting. While I like this set-up, I would prefer to have uncertainties on the classification accuracies. There are several competitive methods, all showing promising accuracies, and it is not clear how significant these results are. An additional, interesting experiment shows the method's performance in an active learning setting. This is an interesting experiment, but I would again like to see errorbars on the plots in figure 2 corresponding to the standard deviation over the 10 trials performed.  ## Clarity The paper reads well and is easy to follow. The structure makes sense, and figure 1 is very helpful in understanding the method. One section that confused me, however, is lines 156-163. While I understand what you are saying, I am not sure what the implications for your model should be.  ## Originality As the authors mention in the (quite extensive) related work section, GPs on graphs have been studied before. The novelty lies in the covariance function, which is simple, yet shown to be effective. By presenting two different views on the developed covariance function, the authors further define interesting directions for future research, which is valuable information for the community.  ## Significance I think this paper constitutes a nice addition to the literature on graph Gaussian processes. The proposed covariance function is easy to interpret, yet powerful, and the authors point out interesting research directions. While not groundbreaking novel, it is solid paper that fits NIPS.


FRAGE: Frequency-Agnostic Word Representation
Continuous word representation (aka word embedding) is a basic building block in many neural network-based models used in natural language processing tasks. Although it is widely accepted that words with similar semantics should be close to each other in the embedding space, we find that word embeddings learned in several tasks are biased towards word frequency: the embeddings of high-frequency and low-frequency words lie in different subregions of the embedding space, and the embedding of a rare word and a popular word can be far from each other even if they are semantically similar. This makes learned word embeddings ineffective, especially for rare words, and consequently limits the performance of these neural network models. In this paper, we develop FRequency-AGnostic word Embedding (FRAGE) which is a neat, simple yet effective way to learn word representation using adversarial training. We conducted comprehensive studies on ten datasets across four natural language processing tasks, including word similarity, language modeling, machine translation, and text classification. Results show that with FRAGE, we achieve higher performance than the baselines in all tasks.

*Summary:  This paper presents a simple regularization technique for word embeddings. The core idea is adding an adversarial loss which is recently widely used in many NLP papers as mentioned in this paper submission. The authors define two frequency-based domains: major words and rare words. The proposed approach is easy to use and seems helpful in improving accuracy of several NLP tasks.  *Strengths:  - The frequency-based analysis on word embeddings is interesting and motivates this work well.  - The method is easy to follow, although we have to additionally tune a few hyperparameters (the frequency threshold and the coefficient for the additional loss).  - The method leads to consistent improvements upon the baselines on the NLP tasks used in their experiments.  *Weaknesses:  - Figure 1 and 2 well motive this work, but in the main body of this paper I cannot see what happens to these figures after applying the proposed adversarial training. It is better to put together the images before and after applying your method in the same place. Figure 2 does not say anything about details (we can understand the very brief overview of the positions of the embeddings), and thus these figures could be smaller for better space usage.  - For the LM and NMT models, did you use the technique to share word embedding and output softmax matrices as in [1]? The transformer model would do this, if the transformer implementations are based on the original paper. If so, your method affects not only the input word embeddings, but also the output softmax matrix, which is not a trivial side effect. This important point seems missing and not discussed. If the technique is not used, the strength of the proposed method is not fully realized, because the output word embeddings could still capture simple frequency information.  [1] Inan et al., Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling, ICLR 2017.  - There are no significance test or discussions about the significance of the score differences.  - It is not clear how the BLEU improvement comes from the proposed method. Did you inspect whether rare words are more actively selected in the translations? Otherwise, it is not clear whether the expectations of the authors actually happened.  - Line 131: The authors mention standard word embeddings like word2vec-based and glove-based embeddings, but recently subword-based embeddings are also used. For example, fasttex embeddings are aware of internal character n-gram information, which is helpful in capturing information about rare words. By inspecting the character n-grams, it is sometimes easy to understand rare words' brief properties. For example, in the case of "Peking", we can see the words start from a uppercase character and ends by the suffix "ing", etc. It makes this paper more solid to compare the proposed method with such character n-gram-based methods [2, 3].  [2] Bojanowski et al., Enriching Word Vectors with Subword Information, TACL. [3] Hashimoto et al., A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks, EMNLP 2017.  *Minor comments:  - Line 21: I think the statement "Different from classic one-hot representation" is not necessary, because anyway word embeddings are still based on such one-hot representations (i.e., the word indices). An embedding matrix is just a weight matrix for the one-hot representations.  - Line 29: Word2vec is not a model, but a toolkit which implements Skipgram and CBOW with a few training options.  - The results on Table 6 in the supplementary material could be enough to be tested on the dev set. Otherwise, there are too many test set results.   * Additional comments after reading the author response Thank you for your kind reply to my comments and questions. I believe that the draft will be further improved in the camera-ready version. One additional suggestion is that the title seems to be too general. The term "adversarial training" has a wide range of meanings, so it would be better to include your contribution in the title; for example, "Improving Word Embeddings by Frequency-based Adversarial Training" or something.


Meta-Reinforcement Learning of Structured Exploration Strategies
Exploration is a fundamental challenge in reinforcement learning (RL). Many current exploration methods for deep RL use task-agnostic objectives, such as information gain or bonuses based on state visitation. However, many practical applications of RL involve learning more than a single task, and prior tasks can be used to inform how exploration should be performed in new tasks. In this work, we study how prior tasks can inform an agent about how to explore effectively in new situations. We introduce a novel gradient-based fast adaptation algorithm -model agnostic exploration with structured noise (MAESN) -to learn exploration strategies from prior experience. The prior experience is used both to initialize a policy and to acquire a latent exploration space that can inject structured stochasticity into a policy, producing exploration strategies that are informed by prior knowledge and are more effective than random action-space noise. We show that MAESN is more effective at learning exploration strategies when compared to prior meta-RL methods, RL without learned exploration strategies, and task-agnostic exploration methods. We evaluate our method on a variety of simulated tasks: locomotion with a wheeled robot, locomotion with a quadrupedal walker, and object manipulation.

This paper introduces a metalearning approach that discovers RL exploration strategies that work well across a particular distribution of MDPs. Furthermore, due to the metalearning approach used, the exploration behaviors can be adapted to the problem at hand quickly as the agent gains experience in the MDP.  It is shown that this approach significantly outperforms other metalearning strategies such as MAML (which metalearns initial policy parameters), as well as random exploration.  This appears to be due to the fact that the proposed method conditions on a per-episode latent variable that encourages temporally coherent exploration throughout the episode, rather than per-step randomness.  Strengths:  + The core idea of per-episode temporally-consistent exploration is excellent and much needed in the RL community.  Per-step random search will never scale well to complex problems.  This is an idea that will be useful to the community well beyond this particular instantiation of it.  + Most exploration techniques are just glorified random or uniform exploration (e.g. intrinsic motivation, diversity metrics, etc). Instead, this approach uses the learned latent space to discover a space of useful exploratory behaviors.  + Metalearning seems to be the right tool for this.  The technical development of the MAESN algorithm is both sound and insightful, as is the analysis of why approaches that are limited to time-invariant noise don’t work.  + The paper is written extremely clearly, being both technically thorough, and providing good high-level overviews of tricky concepts before diving it.  + The evaluation is quite thorough as well.  It is shown that the MAESN works significantly better than existing state-of-the-art approaches on a number of robot domains and that MAESN leads to sensible, temporally consistent exploration. There is also a solid intuitive explanation given of the results + analysis.  + I liked that early “learning to learn” papers were cited in addition to more recent metalearning papers.  Too often metalearning is talked about as though it is a brand new idea, so it is good to see some older works getting credit.   Weaknesses:  - No significant weaknesses.  There could have been a larger diversity in domains (e.g. non-robotics domains such as scheduling, etc.).    Other thoughts:  - There may be something I’m missing, but I’m wondering why the KL divergence term was used instead of just clamping the pre-update w_i to the prior, since they converge to that anyway.  Does it just make it harder to learn if there isn’t some flexibility there during learning?  - There are a few other related papers that would be good to cite.  This work is related in spirit to some prior work on searching for reward functions that speed up learning on a distribution of MDPs by essentially encouraging certain types of exploration.  Some examples:  @article{niekum2010genetic,   title={Genetic programming for reward function search},   author={Niekum, Scott and Barto, Andrew G and Spector, Lee},   journal={IEEE Transactions on Autonomous Mental Development},   volume={2},   number={2},   pages={83--90},   year={2010},   publisher={IEEE} }  @inproceedings{sorg2010internal,   title={Internal rewards mitigate agent boundedness},   author={Sorg, Jonathan and Singh, Satinder P and Lewis, Richard L},   booktitle={Proceedings of the 27th international conference on machine learning (ICML-10)},   pages={1007--1014},   year={2010} }


Image-to-image translation for cross-domain disentanglement
Deep image translation methods have recently shown excellent results, outputting high-quality images covering multiple modes of the data distribution. There has also been increased interest in disentangling the internal representations learned by deep methods to further improve their performance and achieve a finer control. In this paper, we bridge these two objectives and introduce the concept of crossdomain disentanglement. We aim to separate the internal representation into three parts. The shared part contains information for both domains. The exclusive parts, on the other hand, contain only factors of variation that are particular to each domain. We achieve this through bidirectional image translation based on Generative Adversarial Networks and cross-domain autoencoders, a novel network component. Our model offers multiple advantages. We can output diverse samples covering multiple modes of the distributions of both domains, perform domainspecific image transfer and interpolation, and cross-domain retrieval without the need of labeled data, only paired images. We compare our model to the state-ofthe-art in multi-modal image translation and achieve better results for translation on challenging datasets as well as for cross-domain retrieval on realistic datasets.
In this paper, we combine the disentanglement objective with image-to-image translation, and introduce the concept of cross-domain disentanglement. The aim is to disentangle the domain specific factors from the factors that are shared across the domains. To do so, we partition the representation into three parts; the shared part containing information that is common to both domains, and two exclusive parts, which only represent those factors of variation that are particular to each domain (see example in figure 1). Cross-domain disentanglement for image-to-image translation has several advantages that allow for applications that would otherwise not be feasible: (i) Sample diversity: we can generate a distribution of images conditioned on the input image, whereas most image-to-image architectures can only generate deterministic results [23,49]. Our approach is similar to the recent work of Zhu et al. [50], although we explicitly model variations in both domains, whereas they only consider variations in the output domain; (ii) Cross domain retrieval: we can retrieve similar images in both domains based on the part of the representation that is shared between the domains, and, unlike [4], we do not require labeled data to learn the shared representation; (iii) Domain-specific image transfer: domain-specific features can be transferred between images; and (iv) Domain-specific interpolation: we can interpolate between two images with respect to domain-specific features.
Our model is based on bidirectional image translation across domains, using a pair of Generative Adversarial Networks (GANs) [17]. We enforce a disentangled structure in the learned representation through an adequate combination of multiple losses and a new network component called crossdomain autoencoder. We demonstrate the disentanglement properties of our method on variations on the MNIST dataset [26], and apply it to bidirectional multi-modal image translation in more complex datasets [3,38], achieving better results than state-of-the-art methods [23,50] due to the finer control and generality granted by our disentangled representation. Additionally, we outperform [50] in cross-domain retrieval on realistic datasets [23,45]. Our code and models are publicly available at https://github.com/agonzgarc/cross-domain-disen.

In overall, I think this paper proposes a nice framework to learn the disentangled features. The authors addressed my questions in the rebuttal. And I suggest the author add the original BicycleGAN (with skip connections) results to the final version if the paper is accepted. Besides, I also suggest the author soften the claim about GRL loss or explain when the GRL loss can improve the performance and when not.  My final rating is accept. ======================================== This paper proposes a framework to disentangle the domain-exclusive factor and domain-shared factors for paired image-to-image translation task.   Pros: * The proposed cross-domain autoencoder pipeline is an interesting way to implement the domain-shared constraint. It seems a good alternative to the weight-sharing technique which is used in previous image-to-image translation and domain adaptation works. * Empirical experiments show that the proposed method can learn the disentangled features successfully.  Cons: * In the abstract line 13, it is mentioned that the proposed method can perform cross-domain retrieval without the need of labeled. I am confused about this setting because the proposed method does need paired data for training, right? Besides, in line 269, it is mentioned that “Moreover, we do not need image labels, as opposed to specialized approaches such as [13].” However, [13] does not need paired data. So, what does the image labels mean here? * In line 28, it is mentioned that “In spite of the current popularity of these models, the learned representation (the output of the encoder), has not been studied.” I do not agree with this strong statement, since there are many related works on disentangled representation learning just not for image translation specifically, e.g. 1) Disentangled Representation Learning GAN for Pose-Invariant Face Recognition, in CVPR’17; 2) Disentangled person image generation, in CVPR’18. * In line 298, the authors remove the skip connections in [5,12] in order to do a fair comparison. I do not think it is a fair setting. Because the skip connections are very important for the architecture used in [5,12], while the proposed method cannot utilize the skip connections.   Given that this paper is about the disentangled image-to-image translation, it is suggested that some recent work on disentangled representation learning can be included in the related work, e.g.,   [a] K. Bousmalis, G. Trigeorgis, N. Silberman, D. Krishnan and D. Erhan. “Domain separation networks.” In NIPS, 2016. [b] L. Ma, Q. Sun, S. Georgoulis, L. Van Gool, B. Schiele and M. Fritz. “Disentangled person image generation.” In CVPR, 2018. [c] Y.C. Liu, Y.Y. Yeh, T.C. Fu, S.D. Wang, W.C. Chiu and Y.C. Frank Wang. “Detach and Adapt: Learning Cross-Domain Disentangled Deep Representation.” In CVPR, 2018.


An Improved Analysis of Alternating Minimization for Structured Multi-Response Regression
Multi-response linear models aggregate a set of vanilla linear models by assuming correlated noise across them, which has an unknown covariance structure. To find the coefficient vector, estimators with a joint approximation of the noise covariance are often preferred than the simple linear regression in view of their superior empirical performance, which can be generally solved by alternating-minimizationtype procedures. Due to the non-convex nature of such joint estimators, the theoretical justification of their efficiency is typically challenging. The existing analyses fail to fully explain the empirical observations due to the assumption of resampling on the alternating procedures, which requires access to fresh samples in each iteration. In this work, we present a resampling-free analysis for the alternating minimization algorithm applied to the multi-response regression. In particular, we focus on the high-dimensional setting of multi-response linear models with structured coefficient parameter, and the statistical error of the parameter can be expressed by the complexity measure, Gaussian width, which is related to the assumed structure. More importantly, to the best of our knowledge, our result reveals for the first time that the alternating minimization with random initialization can achieve the same performance as the well-initialized one when solving this multi-response regression problem. Experimental results support our theoretical developments.

This paper studies the multi-response regression model, and in particular, the alternating minimization algorithm for the problem . In contrast to prior work, this paper makes the following improvements: 1. It does not require the resampling assumptions that usually abound in showing results on alternating procedures. With good initialization, the procedure without resampling is able to achieve what is usually the minimax rate of such problems. 2. It shows that an arbitrarily initialized problem will still exhibit convergence, but with a worse rate. 3. It allows for the unknown parameter to lie in a possibly non-convex set, and for the noise to be sub-Gaussian.  The major technique is that of generic chaining, which allows the authors to prove bounds that hold uniformly over all iterates.  I liked the paper and the result overall seems quite interesting, in particular points 1 and 2 above. It seems as though the techniques could also translate to other problems. While I did not read the proofs very carefully, they seem quite reasonable overall given the deterministic conditions that the authors posit. On the negative side, the fact that proof for Alternating Minimization are known to exist without re-sampling means that the paper would benefit from a comparison of the current techniques to those used in the past; if the techniques themselves are not so different, then analyzing AltMin in this particular setting without resampling is not particularly impactful or insightful.  Concerns: 1. The authors could de-emphasize point 3 above. It does not particularly impress me, since the authors require an exact solution to each iteration of the parameter estimation problem anyway, irrespective of non-convexity, and this can only be provided in some settings. Similarly, most Gaussian upper bounds can be translated to sub-Gaussian upper bounds since the required behavior of the stochastic process is similar in the two cases. 2. It would be good to compare these results to other results on alternating minimization that do not require resampling (e.g., see the references below).   Minor: Line 131: The function could be misinterpreted. Consider stating as d_1 (respectively d_2) to signify two clauses.  [1] I. Waldspurger, “Phase retrieval with random Gaussian sensing vectors by alternating projections” IEEE Trans. on Inf Theory, 2018. [2] M. Soltanolkotabi. Algorithms and theory for clustering and nonconvex quadratic programming. PhD thesis, Stanford University, 2014.  Post-rebuttal: I have read the response, and maintain my score. This is a strong paper with interesting results.


Learning to Navigate in Cities Without a Map
Navigating through unstructured environments is a basic capability of intelligent creatures, and thus is of fundamental interest in the study and development of artificial intelligence. Long-range navigation is a complex cognitive task that relies on developing an internal representation of space, grounded by recognisable landmarks and robust visual processing, that can simultaneously support continuous self-localisation ("I am here") and a representation of the goal ("I am going there"). Building upon recent research that applies deep reinforcement learning to maze navigation problems, we present an end-to-end deep reinforcement learning approach that can be applied on a city scale. Recognising that successful navigation relies on integration of general policies with locale-specific knowledge, we propose a dual pathway architecture that allows locale-specific features to be encapsulated, while still enabling transfer to multiple cities. A key contribution of this paper is an interactive navigation environment that uses Google Street View for its photographic content and worldwide coverage. Our baselines demonstrate that deep reinforcement learning agents can learn to navigate in multiple cities and to traverse to target destinations that may be kilometres away. The project webpage http://streetlearn.cc contains a video summarizing our research and showing the trained agent in diverse city environments and on the transfer task, the form to request the StreetLearn dataset and links to further resources. The StreetLearn environment code is available at https://github.com/deepmind/streetlearn.

This paper presents an application of a deep reinforcement learning model to a new navigation task based on real-world images taken from Google Street View. The paper is well written and easy to read. The task is interesting, the results are impressive and show that an external memory is not even required in order to achieve a good performance.  My main concerns are that: a) The model cannot work on maps it has not seen during training, and requires to be fine-tuned on new maps. Also, the point of doing transfer between multiple cities does not seem obvious, as the model still needs to be trained (tuned) on the new map and does not perform as well as a single-city model. b) The model is evaluated on the training maps themselves, but using held-out areas. The size of these held-out areas seem to have a significant impact on the model which suggests that the model is overfitting on the training maps.  A couple of comments / questions:  1) I’m not sure I understood how the landmarks are selected to define the goal, even after looking at the Appendix. Are the landmarks fixed and always the same across a city? The example Figure 2. a) shows an example with 5 landmarks used to represent the location of a particular goal, and I initially thought that given a location, g_t was computed using some nearby landmarks. But it turns out that the landmarks you consider are always the same (all of them) and not only the ones nearby?  2) I found line 138 "the agent receives a higher reward for longer journeys" a bit confusing. It sounds like the agent is given more reward for taking paths longer than necessary. It is not totally clear to me why you would want to give more reward for distant goals (in the end you want to find distant goals as much as the nearby ones). Instead I would have found it natural to give a higher reward to the agent if it is able to find the goal quickly.  3) Section 4.2 suggests that curriculum learning is an important part of the training procedure, but Figure 3) c) indicates that ER alone (with the appropriate range) converges faster and to a better final performance. Is that right? Any reason to keep using curriculum learning for this task?  4) I was surprised by how much the auxiliary task of heading is helping the model given the simplicity of this task. Any intuition of why it helps that much? Doesn’t the loss of this task converge to zero very quickly?  5) Figure 5: did you observe that during training, the agent tends to avoid the held-out areas naturally? Or, given a random goal location, it will navigate through it using the held-out areas as much as the other ones? The gap of performance between “fine” and “coarse” grid size is quite significant and I’m wondering if this can be an issue resulting from the agent not being familiar with the held-out areas at all.  6) When the model is trained on some cities and fine-tuned on a new city, how long does it take to fine tune it compared to training a model from scratch on the new city alone? I think this is an important result to know. The experiments show that a single-city network performs better than the ones trained jointly and fine-tuned, so if fine-tuning is not faster, I don’t see the point in training anything else than single-city networks.  7) Line 74 and line 172, Lample & Chaplot should be cited along with Mirowski et al. and Jaderberg et al., as they were the firsts to publish results showing the benefits of training with auxiliary tasks.


Contamination Attacks and Mitigation in Multi-Party Machine Learning
Machine learning is data hungry; the more data a model has access to in training, the more likely it is to perform well at inference time. Distinct parties may want to combine their local data to gain the benefits of a model trained on a large corpus of data. We consider such a case: parties get access to the model trained on their joint data but do not see each others individual datasets. We show that one needs to be careful when using this multi-party model since a potentially malicious party can taint the model by providing contaminated data. We then show how adversarial training can defend against such attacks by preventing the model from learning trends specific to individual parties data, thereby also guaranteeing party-level membership privacy.

**Review updated post feedback** Thank you to the authors for the clarifications included in their feedback, I updated my review accordingly below.  This submission sets out to study poisoning attacks in the context of collaborative ML wherein the attacker is a malicious participant that poisons some of the training data to encode a specific correlation between input attributes and output labels. The submission then considers the robustness of this attack in the face of adversarial training (in the sense of GANs).  The threat model's presentation can be improved:  * The feedback clarified which specific learning setting is considered in this submission (collaborative but not distributed).  * Furthermore, it would be beneficial to expand upon the distinction between contamination attacks and other existing poisoning attacks. My suggestion would be to stress that contamination attacks do not require a specific trigger to be added to the input presented at test time. Examples relating contamination attacks to fairness issues with respect to some of the input attributes are helpful as well to help position this paper within the existing body of literature.  * The submission has ties to work published in security conferences on property inference [a] and [14]. I think there are differences but the present submission could better articulate them and perhaps add some more detailed comparison points than what is currently included in the related work paragraph.  Moving beyond the lack of clarity regarding the threat model, adversarial capabilities assumed in the rest of the manuscript and required to mount the attack could be made more explicit. This aspect was clarified in the author feedback. Indeed, notation potentially misleads readers into thinking that both the ManipulateData and TrainModel procedures are part of the attacker pipelines.   It would have been interesting to see a discussion of how the adversary chooses contaminated attributes and how contamination accuracy relates to the existence of sensitive attributes in a given dataset.  In Section 5.2, it is proposed to apply adversarial training to defend against contamination attacks. However, the evaluation does not consider any adaptive attack strategies to answer the following question: how could adversaries adapt their strategy if they were aware of the defense being deployed? The author feedback addresses this point using guarantees of the mini-max game convergence. However, given that game convergence relies (empirically) on the choice of constant c, it would be beneficial to expand upon this in the manuscript or supplementary material.   Additional minor details: * Abstract uses vague terms. * What is g’s “own objective function” at l205 of p6? * Membership inference attacks are only discussed in the very last paragraph of the manuscript. They could be introduced earlier to help readers position the paper in existing literature.  * Typo line 208: "memership inference attack"  [a] G. Ateniese, L. V. Mancini, A. Spognardi, A. Villani, D. Vitali, and G. Felici. Hacking smart machines with smarter ones: How to extract meaningful data from machine learning classifiers. International Journal of Security and Networks, 2015.


The Limit Points of (Optimistic) Gradient Descent in Min-Max Optimization
Motivated by applications in Optimization, Game Theory, and the training of Generative Adversarial Networks, the convergence properties of first order methods in min-max problems have received extensive study. It has been recognized that they may cycle, and there is no good understanding of their limit points when they do not. When they converge, do they converge to local min-max solutions? We characterize the limit points of two basic first order methods, namely Gradient Descent/Ascent (GDA) and Optimistic Gradient Descent Ascent (OGDA). We show that both dynamics avoid unstable critical points for almost all initializations. Moreover, for small step sizes and under mild assumptions, the set of OGDA-stable critical points is a superset of GDA-stable critical points, which is a superset of local min-max solutions (strict in some cases). The connecting thread is that the behavior of these dynamics can be studied from a dynamical systems perspective.

The paper studies stability of gradient and optimistic gradient dynamics for C^2 saddle-point problems.  The main contribution of the paper can be summarized in two results (stated in the inclusion following line 83): - local saddles are stable for GDA (under Assumption 8.1) - stable equilibria of GDA are also stable for OGDA. (note that results on unstable critical points were previously known).  Quality: The results are interesting, and the paper is well written. There are some typos in the proofs, but I believe these are omissions that can be corrected, rather than major flaws.  Significance: I would love to see further discussion of the consequences of this result, and its relevance to the NIPS community, both theoreticians and practitioners. For example, do these results suggest that GDA should be preferred to OGDA (since the latter has a larger equilibrium set)? Does the analysis extend to time-varying step sizes? In the concluding remarks, the authors mention that some saddle points (local min max) can be better than others. Can the authors expand on how the quality of an equilibrium can be measured?  Originality: It takes some effort to tease out the contributions of the paper from existing results. Existing results should be cited but not part of the main paper (they can potentially be moved to the appendix) in favor of an expanded discussion of Theorem 1.10 and its consequences. - Theorems 1.11, 2.2 and 3.2 (that a dynamical system a.s. does not converge to an unstable equilibrium when the Hessian is invertible) are known. Note that the particular form of the dynamics is irrelevant to the result, only the fact that the spectral radius of the Jacobian at the equilibrium is greater than one. I don't think one can claim this is a contribution of this paper. - Similarly, Corollary 2.3 and 3.3 are immediate consequences of these known results, and are stated without discussion. I do not see what they bring to the picture. - The literature review needs to be expanded. Similar techniques have been used to prove stability of gradient algorithms (e.g. [1]), and for GAN training [2], which seems particularly relevant, since they use similar techniques (analyzing the Jacobian at the equilibrium, albeit for continuous time dynamics) and this paper is motivated by GAN training.  Typos and minor suggestions: - Line 18: "compact and concave subsets": do you mean convex? - In the analysis of OGDA dynamics, what is the advantage of introducing the functions F and F'? One can simply express the dynamics (and the proof of Lemma 3.1 in the appendix) in terms of f. Defining F, F' introduces unnecessary additional notation. - Second experiment (4.2): how is this experiment related to the results of the paper? - Statement of Theorem B.1: what is E? What is Dg(x^*), is this the Jacobian? Please use notation consistent with the rest of the paper. - Please define "C^r embedded disk" in Theorem B.1. - In the proof of Theorem 2.2 and 3.2, you can extract a finite sub-cover simply by the compactness assumption. Why mention second-countability? - In the proof of Lemma 2.4: I + a H should be I + \alpha H. The proof only provides an upper bound on the eigenvalues of H. Note that this is not sufficient, a lower bound is also needed to conclude. - Proof of Lemma 2.7: Incomplete sentence "Hence we conclude that Re(\lambda)". There is also a missing factor 2 in the expression of the eigenvalue magnitude.  [1] L. Lessard, B. Recht and A. Packard. Analysis and Design of Optimization Algorithms via Integral Quadratic Constraints. SIAM Journal on Optimization - 26(1): 57-95 [2] V. Nagarajan and J. Zico Kolter, Gradient descent GAN optimization is locally stable. NIPS 2017.  ======== post rebuttal Thank you for your responses, and for the corrections to the proofs. Regarding Theorem 2.2, 3.2 and Corollaries 2.3 and 3.3: I agree that proving the invertibility of the Jacobian for these particular dynamics is new, but the rest of the argument is identical to previous results, and this minor contribution hardly justifies stating two lemmas, two theorems and two corollaries, especially when much of this space can be used to improve the discussion of the new results. I strongly encourage the authors to move these to the supplement, and clarify that these mostly follow existing arguments.


Step Size Matters in Deep Learning
Training a neural network with the gradient descent algorithm gives rise to a discrete-time nonlinear dynamical system. Consequently, behaviors that are typically observed in these systems emerge during training, such as convergence to an orbit but not to a fixed point or dependence of convergence on the initialization.
Step size of the algorithm plays a critical role in these behaviors: it determines the subset of the local optima that the algorithm can converge to, and it specifies the magnitude of the oscillations if the algorithm converges to an orbit. To elucidate the effects of the step size on training of neural networks, we study the gradient descent algorithm as a discrete-time dynamical system, and by analyzing the Lyapunov stability of different solutions, we show the relationship between the step size of the algorithm and the solutions that can be obtained with this algorithm. The results provide an explanation for several phenomena observed in practice, including the deterioration in the training error with increased depth, the hardness of estimating linear mappings with large singular values, and the distinct performance of deep residual networks.

This paper analyzes the effect of the choice of step length for the (full-batch) gradient method applied to feed-forward linear networks and two-layer (one hidden layer) ReLU networks from a discrete-time dynamical system perspective. In particular, the bounds on the steplength characterize which minima the algorithm can converge to when certain steplengths are used in the algorithm. A more in-depth analysis is also provided for the case of symmetric positive definite matrices, in which the maximum steplength can be used when the parameters are initialized with the identity matrix.  Overall, the paper provides novel bounds on the step length necessary to converge to a certain subset of stationary points. This corresponds with my understanding that practitioners are indeed selecting their initialization and learning rate schedules for SGD to specifically converge to particular local minima that have better generalization properties.   Although these theoretical results are very interesting, they are motivated improperly, in my opinion. I expand on this point below:  1. Explanation of Training Error Behavior  Coming from a background in optimization rather than dynamical systems, the first contribution (line 80) highlighting the reason the training error stops decreasing is already well understood from an optimization perspective, in my opinion. In particular, the convergence of the gradient method for strongly convex functions with fixed step length requires Lipschitz continuity of the gradient (or smoothness). This makes some claims (such as the claim on large singular values in line 263-264) trivial. If the smoothness condition does not hold, one may have to diminish the step length in order to converge to the solution.   In the more realistic case in which we are applying the stochastic (sub)gradient method, we know that it indeed requires a diminishing steplength (typically non-summable but square-summable) in order to converge to the solution; otherwise, the algorithm only is guaranteed to converge to a neighborhood of the solution. See Bottou, et al. (2018) for a review on the convergence analysis of stochastic gradient methods. In this sense, the claim that the (stochastic) gradient method is converging to a periodic orbit in dynamical systems is similar to the results of only having guarantees to converge to a neighborhood of the solution, which is already sufficient to explain the plots from He, et al. (2016).  Rather than motivating the theoretical results from this angle (which I would consider to be uninteresting and well-known from optimization), I would suggest the authors take the approach of claiming to provide theory that better elucidates the connection between the choice of step length and the subset of local optima the algorithm can converge to, from a dynamical systems perspective. This is the part of the paper that I would consider most novel and interesting, specifically towards understanding the training of neural networks.  2. Stochastic vs. Deterministic  Unfortunately, the paper fails to address how the use of stochasticity in the stochastic gradient method could impact these results for the choice of step length. Instead, the paper only analyzes the case for the full gradient method, which is not representative of what is used in practice (which also takes away from the first contribution). If these results claim to establish the relationship between the choice of step length for training neural networks, the paper must address the lack of stochasticity of the method considered in the analysis and use these results as a theoretical building block towards understanding the stochastic gradient method for training neural networks.  3. Relationship with Lipschitz Continuity and Smoothness  Lastly, I would suggest the authors provide a more in-depth investigation of the connection between these results and the local Lipschitz and smoothness constants of the objective function. I’m curious how these results may be related to the typical convergence analysis of the gradient method in optimization. For instance, in the first example (line 36), the function considered does not have a global smoothness constant; hence, the gradient method would require a diminishing step length in order to converge from any initialization. This is clearly reflected in the example.  Minor Comments and Questions: - Should always say “the gradient method” or “the gradient descent algorithm”, not “the gradient descent”. - The networks considered are not relevant to practice, but this is OK given that the analysis of neural networks have mainly been limited to linear and ReLU feed-forward networks. - The paper is well-written.  Summary:  In summary, this paper provides some elucidating theoretical results on the set of local minima that the gradient method will converge to for certain choices of step size for simple feed-forward networks. This could be motivated better from a different angle as described earlier, without the comment explaining the convergence of the method to a periodic orbit or neighborhood of the solution, which is well-known in optimization. Because of its theoretical novelty, I believe that this well-written paper is indeed worthy of publication, with some minor edits and perhaps a major change in approach and introduction.  ================================ Post-Author Response:  I thank the authors for their well-crafted response, particularly for clarifying the comment made in Lines 263-264. In light of the changes proposed in the response regarding the introduction and additional remarks, I have raised my score to a 7.


Provable Gaussian Embedding with One Observation
The success of machine learning methods heavily relies on having an appropriate representation for data at hand. Traditionally, machine learning approaches relied on user-defined heuristics to extract features encoding structural information about data. However, recently there has been a surge in approaches that learn how to encode the data automatically in a low dimensional space. Exponential family embedding provides a probabilistic framework for learning low-dimensional representation for various types of high-dimensional data [20]. Though successful in practice, theoretical underpinnings for exponential family embeddings have not been established. In this paper, we study the Gaussian embedding model and develop the first theoretical results for exponential family embedding models. First, we show that, under mild condition, the embedding structure can be learned from one observation by leveraging the parameter sharing between different contexts even though the data are dependent with each other. Second, we study properties of two algorithms used for learning the embedding structure and establish convergence results for each of them. The first algorithm is based on a convex relaxation, while the other solved the non-convex formulation of the problem directly. Experiments demonstrate the effectiveness of our approach.

Summary: The paper analyzes Gaussian embedding in the exponential-family embedding framework of Rudolph et al NIPS'16. The primary contribution of the paper is to obtain convergence guarantees for two estimation procedures considered in the paper for Gaussian embeddings. The context of each node induces a graph structure over the node and the authors (somewhat) relate convergence with properties of the graph (e.g. the spectral norm of the adjacency matrix).  Clarity and Originality: The paper is clearly written and easy to follow. A contribution of the paper is to view the exponential-family embedding model as a graphical model. I have the following questions: 1) How are repeated observations handled? For instance in the word embedding case how are repeated words handled? It seems like each node of the graph corresponds to a unique word? If this is the case, then the flexibility of the exponential-family embeddings model is greatly reduced. 2) The joint distribution over the nodes is not a graphical model: (i) its not a directed graphical model since there can be loops in the graph (ii) its not an undirected graphical model becuase the joint distribution is not a Gibbs distribution i.e. doesn't factor over maximal cliques. 3) How do you leverage the results from the literature of graphical models for estimation? 4) The estimation procedure is essentially the loss function from Rudolph et al specialized for the Gaussian case. However, it is strange to ignore the unknown covariances matrix during estimation --- this is reasonable if the Gaussians are isotropic which does not appear to be the case here.  Minor comments: In Lemma 4.1 additional constants \kappa_mu and \kappa_L don't need to be introduced. (4.2) can be directly stated in terms of c_min and c_max.  Originality and Significance: The primary contribution of the paper is the theoretical analysis of Gaussian embeddings. The paper makes a number of simplifying assumptions in order to facilitate theoretical analyses. However the assumption that the embedding matrices (V_i's) are the same is problematic. This greatly reduces the flexibility of exponential-family embeddings model. Experimental evidence needs to be provided for the usefulness of the simplified model. I am also not convinced if Assumption SC holds in any reasonable setting. Its very unlikely that the spectral norm of the \Sigma_{col}^{1/2}, which is a pm x pm matrix, is constant --- the spectral norm grows with both the number of variables m and the dictionary size p. Also, where does this constant (\rho) show up in the bound in Theorem 4.2? This is arguably the weakest part of the paper. While the analysis is promising, it is preliminary and the authors should further exploit different properties of the graph to provide bounds for the spectral norm of A and \Sigma_{col}. The authors should additionally provide empirical evidence through simulation experiments on the effect of the covariance matrices on estimation.  === Post rebuttal === I think the authors for clarifying some of the issues in the rebuttal. Here is my final assessment of the paper after going through the rebuttal, the paper, other reviews and some related work.  The primary novelty of the paper is to view the exponential family embedding model as an instance of the mixed graphical model framework of Chen et al (2014). This allows them to show that the conditional distributions in the Gaussian embedding model induce a valid joint distribution, parameterized by a covariance matrix \Sigma_{col}, over the data (x_j). Due to this the authors are able to obtain theoretical guarantees for two estimation procedures considered in the paper by imposing suitable assumptions on the matrix \Sigma_{col}. The other contribution of the paper is to develop examples to motivate and justify their assumptions. On second reading I think the examples are reasonable --- the identity covariance matrices used in the examples is a common simplification used in the literature for exponential family embeddings.  However, both the convex and non-convex method and their analysis is pretty standard. The analysis of the convex method is an invocation of the restricted strong convexity framework of Negahban and Wainwright and all the authors need to show is that the quadratic objective is strongly convex --- which is straight forward. The analysis of the non-convex method is again a straight-forward invocation of the results of Yu et al (2018).  The analysis for the Gaussian case would be complete if the authors do the following: 1) Keep the link function in the analysis and ideally state their results with respect to general link functions, 2) keep the embedding and context matrix separate --- the matrix V_j and V_k respectively in Eq 2.2.This would make the results relevant and faithful to the Gaussian embedding case introduced in Rudolph et al.


Learning Deep Disentangled Embeddings With the F-Statistic Loss
Deep-embedding methods aim to discover representations of a domain that make explicit the domain's class structure and thereby support few-shot learning. Disentangling methods aim to make explicit compositional or factorial structure. We combine these two active but independent lines of research and propose a new paradigm suitable for both goals. We propose and evaluate a novel loss function based on the F statistic, which describes the separation of two or more distributions. By ensuring that distinct classes are well separated on a subset of embedding dimensions, we obtain embeddings that are useful for few-shot learning. By not requiring separation on all dimensions, we encourage the discovery of disentangled representations. Our embedding method matches or beats state-of-the-art, as evaluated by performance on recall@k and few-shot learning tasks. Our method also obtains performance superior to a variety of alternatives on disentangling, as evaluated by two key properties of a disentangled representation: modularity and explicitness. The goal of our work is to obtain more interpretable, manipulable, and generalizable deep representations of concepts and categories.
The literature on deep embeddings (Chopra et al., 2005;Yi et al., 2014a;Schroff et al., 2015;Ustinova & Lempitsky, 2016;Song et al., 2016;Vinyals et al., 2016;Snell et al., 2017) addresses the problem of discovering representations of a domain that make explicit a particular property of the domain instances. We refer to this property as class or category or identity. For example, a set of animal images might be embedded such that animals of the same species lie closer to one another in the embedding space than to animals of a different species. Deep-embedding methods are trained using a class-aware oracle which can be queried to indicate whether two instances are of the same or different class. Because this paradigm can handle an arbitrary number of classes, and because the complete set of classes does not have to be specified in advance-as they would be in an ordinary classifier-deep embeddings are useful for few-shot learning. A small set of examples of novel classes can be projected into the embedding space, and an unknown instance can be classified by its proximity to the embeddings of the labeled examples.
Similar to deep embeddings, the literature on disentangling attempts to discover representations of a set of instances, but rather than making explicit a single property of the instances (class), the goal is to make explicit multiple, independent properties, which we refer to as factors. For example, a disentangled representation of animals might include factors indicating its size, length of its ears, and whether it has feet or fins. We will later be more rigorous in defining a disentangled representation, but for now we operate with the informal notion that the factors form a compositional or distributed representation such that with relatively few factors and relatively few values of each factor, the factor values can be recombined to span the set of instances. Disentangling has been explored using either a fully unsupervised procedure (Chen et al., 2016;Higgins et al., 2017) or a semi-supervised 32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montréal, Canada.
procedure in which a factor-aware oracle can be queried to specify a factor along with sets of instances partitioned by factor value (Reed et al., 2014;Kulkarni et al., 2015;Karaletsos et al., 2015;Reed et al., 2015).
Despite their overlapping and related goals, surprisingly little effort has been made to connect research in deep embeddings and disentangling. There are two obvious ways to make the connection. First, a factor-aware oracle might be used to train deep embeddings (instead of a class-aware oracle), and hopefully disentangled representations would emerge. Second, a class-aware oracle might be used to train disentangled representations (instead of a factor-aware oracle), and hopefully an embedding suitable for few-shot learning would emerge. We primarily pursue the former approach, but briefly explore the latter as well.
In the next section, we propose a deep-embedding method that is suitable for both few-shot learning of novel classes and for disentangling factors. After describing the algorithm and showing that it obtains state-of-the-art results on the recall@k task that is ordinarily used to evaluate embeddings, we turn to analyzing how well the algorithm disentangles the factors that contribute to class identity. To perform a rigorous evaluation, we put forth formal, quantifiable criteria for disentanglement, and we show that our algorithm outperforms other state-of-the-art deep-embedding methods and disentanglement methods in achieving these criteria.

Summary  This work seeks to combine methods from deep embedding literature with the aims of disentanglement to create disentangled embeddings. The authors use the F statistic: a well known statistical measure of inter vs intra class variance, in order to introduce separation between classes.   To produce the embedding, convolutional features are used to train a fully connected embedding network with a loss designed around the F-statistic. The embeddings were evaluated against other state of the art deep metric losses such as triplet loss, structured softmax loss, and histogram loss. The F-statistic loss did not significantly outperform these other methods.  The authors then evaluated the degree of disentanglement between the produced embeddings using two main metrics: modularity and explicitness. Similar to InfoGAN and Beta-VAE, dimensions of a disentangled representations are required to have mutual information with at most one factor of variation - this is defined as modularity. Explicitness is defined as the existence of a linear transformation between a dimension of the embedding, and the value of the factor of variation it represents. Here this is calculated with a one-vs-rest linear classifier, where explicitness is predicated on the ability of the classifier to predict the correct class from a single dimension of the embedding.     The disentangled representations are trained in a weakly-supervised fashion, where an oracle conveys information about the similarities between batch samples, but not the name of the factor upon which they differ. The embeddings were compared against histogram loss (Ustinova & Lempitsky, 2016), triplet (Schroff et al., 2015), as well as the unsupervised Beta-VAE. Under the metrics of modularity and explicitness, the F-statistic based embeddings performed better than other methods.  Review  F-tests are often used to decompose the variance given in a modeled distribution. Using the F-statistic to encourage disentangled representations seems natural to me. Simply creating well structured embeddings was clearly not the point of this work, as the embeddings did not perform significantly better than other methods. However, under the two computed statistics of modularity and explicitness, the F-statistic based loss performed better than both fully supervised and unsupervised methods (even if comparing to unsupervised methods is unfair).   That being said, the paper was unclear for me in a number of places: 1) I didn’t understand the training mechanism for the disentangled representations. How was 5 fold validation used here, and why was it important? A graphic showing an example curated mini-batch would help here.  2) It would be helpful to explicitly state how this measure of modularity differs from the mutual information terms which InfoGAN use in their losses to encourage disentanglement.  3) Why not directly evaluate against a method which explicitly encourages disentanglement like the F-statistic loss, such as (Chen et al. 2016) which minimizes total correlation between dimensions.  4)  This could also be done in a fully supervised way, this is a stronger argument for the F-statistic in general if it is shown to beat state of the art as both a supervised and weakly-supervised method.  The main idea is simple if not appreciated: using the F-statistic to encourage separation between classes. I have not seen this before in disentanglement research, but I expect to see it again in the future. I think there could have been a stronger evaluation done by comparing against other methods. If already comparing to a unsupervised method like Beta-VAE, why not compare to a method like InfoGAN which explicitly maximizes mutual information between a small subset of the code, and the observation,  I believe this work was well thought-out, but it needs either a stronger evaluation, or a stronger argument for its current evaluation criterion.  ** After rebuttal:  I still don't like that they don't want to compare to InfoGAN. They used the metrics in the following paper (which delineates requirements for disentanglement)  https://openreview.net/pdf?id=By-7dz-AZ   And this paper scores InfoGAN higher than Beta-VAE on almost all metrics e.g. "InfoGAN achieves the highest average disentanglement",  "InfoGAN also achieves the highest average completeness",  "InfoGAN contains the most easily-extractable / explicit information about [embedding] z.  So I don't buy their assertion that Beta-VAE is better than InfoGAN thus comparing to Beta-VAE is all that matters. I still believe the evaluation is insufficient because of this.


Pelee: A Real-Time Object Detection System on Mobile Devices
An increasing need of running Convolutional Neural Network (CNN) models on mobile devices with limited computing power and memory resource encourages studies on efficient model design. A number of efficient architectures have been proposed in recent years, for example, MobileNet, ShuffleNet, and MobileNetV2. However, all these models are heavily dependent on depthwise separable convolution which lacks efficient implementation in most deep learning frameworks. In this study, we propose an efficient architecture named PeleeNet, which is built with conventional convolution instead. On ImageNet ILSVRC 2012 dataset, our proposed PeleeNet achieves a higher accuracy and over 1.8 times faster speed than MobileNet and MobileNetV2 on NVIDIA TX2. Meanwhile, PeleeNet is only 66% of the model size of MobileNet. We then propose a real-time object detection system by combining PeleeNet with Single Shot MultiBox Detector (SSD) method and optimizing the architecture for fast speed. Our proposed detection system 2 , named Pelee, achieves 76.4% mAP (mean average precision) on PASCAL VOC2007 and 22.4 mAP on MS COCO dataset at the speed of 23.6 FPS on iPhone 8 and 125 FPS on NVIDIA TX2. The result on COCO outperforms YOLOv2 in consideration of a higher precision, 13.6 times lower computational cost and 11.3 times smaller model size.

The authors proposes a new network backbone and applied it to improve object detection model speed and accuracy under constrained computation budget. The backbone is modified from densenet and detection architecture modified from SSD. The authors compare their backbone model in their own dataset and ImageNet, and detection model in PASCAL VOC and COCO.  Strengths: - The proposed method is described in detail and evaluated at component level and end-to-end.  - Proposed detection model improves AP by ~4% compared with mobilenetSSD, which is significant.  - Authors also compared the runtime in actual device, which is more convincing than just counting FLOPs.  Weaknesses:  - Although authors have proposed a set of modifications to their baseline DenseNet + SSD model, the novelty on each modification is limited. - Section 2.2.3. The experiment comparing PeleeNet with other network architecture is flawed. The author says "we can evaluate a model pre-trained on ILSVRC 2012 on this dataset", but the whole ImageNet has 1000 object classes but Stanford Dogs dataset only has 120 dog classes. Although some features can be shared in lower layers, comparing a model trained for a variety of categories with a model for dogs only is not fair. Also Table 3, PeleeNet shows big improvement on stanford dogs (~7% top-1 gain over mobilenet) but only 0.6% on the full Imagenet also suggests this experiment is not solid.  Some additional comments:  - It makes a lot of sense separable conv has an advantage in FLOP counts but doesn't run as fast in actual devices, due to bad memory access pattern. However the final Pelee model (which uses conventional conv) has similar MACs with mobilenet SSD and similar runtime too. Additional analysis / profiling could make this paper stronger.  -  Section 1, paragraph 3: "the layer uses a small kernel size (3x3), which is good enough to capture small-size objects": 3x3 is a standard practice for most modern NNs. It is arguable 3x3 is "good enough". Also smaller kernel is not very relevant  with small objects? What matters more is better spatial resolution in conv feature maps, which is usually decided by the stride of conv kernels rather than its size.  - Section 1, paragraph "Composite Function": "post-activation" is referring to BN before ReLU, this name is a bit confusing... better call it "BN before activation" / "BN after activation"?   - Same paragraph: experiment (Table 2) does show BN before ReLU hurt performance. The authors argue BN can be folded into conv this way, but BN after ReLU can be folded into next Conv op, because all activation in the same conv feature map shares the same mean / var / scale / offset in BN implementation?  - The proposed architecture drops 38x38 feature map which would hurt on small objects? Usually small objects would cost more budget to capture and they don't give good improvement on overall mAP. It would be more convincing to analyze how the proposed model works for small / medium / large size objects.  - [Minor] Section 2.1. The proposed model uses avg pool with stride 2. Conv with stride 2 could potentially be a good alternative?  - [Minor] Section 3.1: SSD feature map used in this paper is very similar to [5]. Dropping 2x2 map has limited impact on speed / accuracy.   Meanwhile, Mobilenet V1 / V2 + SSD lite (https://arxiv.org/pdf/1801.04381.pdf) presents similar performance but with smaller size / FLOPs. However it is reasonable to assume this paper to be concurrent work.


Graph Oracle Models, Lower Bounds, and Gaps for Parallel Stochastic Optimization
We suggest a general oracle-based framework that captures different parallel stochastic optimization settings described by a dependency graph, and derive generic lower bounds in terms of this graph. We then use the framework and derive lower bounds for several specific parallel optimization settings, including delayed updates and parallel processing with intermittent communication. We highlight gaps between lower and upper bounds on the oracle complexity, and cases where the "natural" algorithms are not known to be optimal.

The paper presents lower bounds for parallel computation of population risk minimization over a ball using a DAG computation model, aiming to capture both computation and communication over parallel processors. Along the line of a series of papers [3,8,9,15,26,27] trying to provide lower bounds inspired lower bound of Nemirofski [19] on the serial setting, the present paper assumes queries are submitted to either a gradient or a prox oracle. Such queries constitute nodes in a graph, and the decision of which query to submit, as well as what value to "output" at a given node as a solution to the problem, depend only queries (and responses) observed *on the predecessors" of the nodes in the DAG. The authors provide lower bounds for the two oracles in this setting, depending on the Lipschitz constant and smoothness of the objective, as well as on the depth and size of the DAG and the radius of the constraint set. Going beyond these bounds, the authors discuss corresponding upper bounds for different topologies corresponding to different communication patterns, linking them to either existing algorithm and already derived upper bounds, or improving/tightening the gap with new proposed algorithms when the latter is not possible (though a few cases remain open).  The paper is timely, rigorous, and extremely well-written, and the results are of significant consequence. Though the analysis builds upon earlier work (mostly [3,9,26]), the authors amend several assumptions made in the prior art regarding the type of communication allowed. The assumptions considered here (that communication is restricted to sharing computational outcomes across the DAG) are both natural and more general than prior art.  The authors also identify  and elucidate aspects of their proof and function construction that differ from prior work in the end of Section 3 (thank you!).  There is no related work section, but the paper is well positioned with respect to related work spread out throughout the text. Section 4 in particular does a very good job linking lower bounds to a menagerie of existing/proposed algorithms, their upper bounds, and respective gaps (if they exist).  Though this section is lucid and readable, it would be good to be more precise and formal about how A-MB-SGD in [10,14], the algorithm in [11], or SVRG [13,15] indeed fall under the corresponding DAG models (and indeed resort to respective oracle queries) as the ones used in the paper. To deal with space constraints and maintain the readability of Section 4, this formal derivation can be added in the supplement.   Nits: Line 194 on Page 6, 11→ (11)  Line 203 on Page 6, j=1…\tau → j=1,...,\tau (missing commas) Line 241 on Page 7, approach Instead → approach. Instead Line 259 on Page 7, posses → possesses  Line 321 on Page 8, first order → first-order


Adaptive Path-Integral Autoencoder: Representation Learning and Planning for Dynamical Systems
We present a representation learning algorithm that learns a low-dimensional latent dynamical system from high-dimensional sequential raw data, e.g., video. The framework builds upon recent advances in amortized inference methods that use both an inference network and a refinement procedure to output samples from a variational distribution given an observation sequence, and takes advantage of the duality between control and inference to approximately solve the intractable inference problem using the path integral control approach. The learned dynamical model can be used to predict and plan the future states; we also present the efficient planning method that exploits the learned low-dimensional latent dynamics. Numerical experiments show that the proposed path-integral control based variational inference method leads to tighter lower bounds in statistical model learning of sequential data. The supplementary video 1 and the implementation code 2 are available online.

Paper : 5353 This paper discusses representation learning problem, and designs an algorithm to learn variational parameters of approximate inference. The paper has adopted a semi-amortized technique for learning variational parameters of dynamic system for temporal data. In this paper, dynamic system is generated by state space model. In order to express the state space model, just initial state and control input at every time steps needs to be inferred. Paper has utilized adaptive path-integral technique for variational parameter refinement of dynamic system to mitigate the amortization gap( induced by limitation of inference network parameters compared to optimal variational distribution ).  Expressing the problem(finding variational distribution parameters ) in the format that is addressed by stochastic optimal control and adaptation of adaptive  path-integral  technique to dynamic system can be considered main contributions of this paper . Experiments has been done on inverted pendulum and  3D human motion data, results show some small  relative advantage for current method than baselines. For one of datasets (inverted pendulum), the model has shown good predictive capability although not compared with other baseline for quantitative prediction power.   Quality: claims and supporting material both in main paper and supplementary have been verified and raised a few questions that.   One issue found is in equation 10 appeared on page 4, as we know ELBO = E_{q_u(Z)}[log p(X|Z)] -KL(q_u(Z|X)||p(Z)), in supplementary materials shown that KL(q_u(Z|X)||p(Z)) = E_{q_u}[\integ{V(Z)+1/2 ||u(t)||^2}] +log \zetta where log\zetta does not contain the variational parameters, then ELBO will turn out to be  ELBO = E_{q_u(Z)}[log p(X|Z) +\integ{-V(Z)-1/2 ||u(t)||^2}] , although what we have in  10 does not match with what I found in ELBO above, can you please clarify? Also I am not sure if V(z) in line 129 is same as V(z) in line 370 or just used same notation for 2 different expression.  In experimental results of this paper,in inverted pendulum example, the prediction has been good and shows that parameters of variational distribution describes the dynamic system well. It will be informative if prediction for 3D human motion data is presented as well.  Clarity: Although paper has been motivated well and general concepts and background have been explained well, there are some points which need clarification.  On lines 137 and 138 has  mentioned that feed forward/feedback structure can be implemented using RNN, but RNN has not  been used in this paper for inferring parameters, please be more specific on that. Having schematic of network and specifying  inference vs generative network and their parameters help the reader to understand your proposed solutions better.  In experiment section baseline algorithms specification and setups are not mentioned    Originality: As mentioned in summary the main contribution of this paper could be summarized as bellow  - Expressing the problem(finding variational distribution parameters for modeling sequential data ) in the format that is addressed by stochastic optimal control  (SOC)  -  adaptation of adaptive path-integral technique for variational parameter refinement of dynamic system to mitigate the amortization gap  authors have cited many of recent works, and seems have done an in depth literature review also could distinguish their contribution to what has been done so far clearly.    Significance:  As discussed, in summary section, the results show some small relative improvement with baseline algorithms. But as suggested, it will be very useful to have a quantitative prediction comparison  between the proposed algorithm and baselines.