UNSUPERVISED SPEECH RECOGNITION VIA SEGMEN- TAL EMPIRICAL OUTPUT DISTRIBUTION MATCHING
We consider the problem of training speech recognition systems without using any labeled data, under the assumption that the learner can only access to the input utterances and a phoneme language model estimated from a non-overlapping corpus. We propose a fully unsupervised learning algorithm that alternates between solving two sub-problems: (i) learn a phoneme classifier for a given set of phoneme segmentation boundaries, and (ii) refining the phoneme boundaries based on a given classifier. To solve the first sub-problem, we introduce a novel unsupervised cost function named Segmental Empirical Output Distribution Matching, which generalizes the work in (Liu et al., 2017) to segmental structures. For the second sub-problem, we develop an approximate MAP approach to refining the boundaries obtained from Wang et al. (2017). Experimental results on TIMIT dataset demonstrate the success of this fully unsupervised phoneme recognition system, which achieves a phone error rate (PER) of 41.6%. Although it is still far away from the state-of-the-art supervised systems, we show that with oracle boundaries and matching language model, the PER could be improved to 32.5%. This performance approaches the supervised system of the same model architecture, demonstrating the great potential of the proposed method.

Overview:
This paper proposes a new approach to do unsupervised phoneme recognition by learning from unlabelled speech in combination with a trained phoneme language model. The proposed loss function is a combination of a term encouraging the language model of predicted phonemes to match the given language model distribution, and a term to encourage adjacent speech frames to be assigned to the same phoneme class. Phoneme boundaries are iteratively refined using a separate model. Experiments where a hidden Markov model is applied on top of the predicted phonemes are also performed.
Main strengths:
The paper is clear and addresses a very important research problem. The approach and losses proposed in Section 2 have also not been proposed before, and given that an external language model is available, are very natural choices.
Main weaknesses:
The main weakness of this paper is that it does not situate itself within the rich body of literature on this problem.  I give several references below, but I think the authors can include even more studies -- there are several studies around "zero-resource" speech processing, and I would encourage the authors to work through the review papers [1, 6].
Concretely, I do not think the authors can claim that "this is the first fully unsupervised speech recognition method that does not use any oracle segmentation or labels." I think it could be argued that the system of [3] is doing this, and there are even earlier studies. I also don't think this claim is actually necessary since the paper has enough merit to stand on its own, as long as the related work is discussed properly.
For instance, the proposed approach shares commonalities with several other approaches: [2] also used two separate steps for acoustic modelling and boundary segmentation; [4, 7, 8] builds towards the setting where non-matching text data is available (for language model training) together with untranscribed speech for model development; the approach of [5] uses a very similar refinement step to the one described in Section 3, where an HMM model is initialised and retrained on noisy predicted labels.
In the experiments (Section 4), it would also be useful to report more fine-grained metrics. [6] gives an overview of several of the standard metrics used in this area, but at a minimum phoneme boundary recall, precision and F-scores should be reported in order to allow comparisons to other studies.
Overall feedback:
Given that this paper is situated within the broader context of this research area, which already has a small community around it, I think the novelty in the approach is strong enough to warrant publication given that the additional metrics are reported in the experiments.
Papers/links that should be reviewed and cited:
1. E. Dunbar et al., "The Zero Resource Speech Challenge 2017," in Proc. ASRU, 2017.
2. H. Kamper, K. Livescu, and S. Goldwater. An embedded segmental k-means model for unsupervised segmentation and clustering of speech. in Proc. ASRU, 2017.
3. Lee, C.-y. and Glass, J. R. A nonparametric Bayesian approach to acoustic model discovery. ACL, 2012.
4. Ondel, Lucas, Lukaš Burget, Jan Černocký, and Santosh Kesiraju. "Bayesian phonotactic language model for acoustic unit discovery." In Acoustics, Speech and Signal Processing (ICASSP), 2017 IEEE International Conference on, pp. 5750-5754. IEEE, 2017.
5. Walter, O., Korthals, T., Haeb-Umbach, R., and Raj, B. (2013). A hierarchical system for word discovery exploiting DTW-based initialization. ASRU, 2013.
6. M. Versteegh, X. Anguera, A. Jansen, and E. Dupoux, "The Zero Resource Speech Challenge 2015: Proposed approaches and results,” in Proc. SLTU, 2016.
7. https://www.clsp.jhu.edu/wp-content/uploads/sites/75/2018/05/jsalt2016-burget-building-speech-recognition.pdf
8. https://www.clsp.jhu.edu/workshops/16-workshop/building-speech-recognition-system-from-untranscribed-data/


VARIATIONAL DISCRIMINATOR BOTTLENECK: IMPROVING IMITATION LEARNING, INVERSE RL, AND GANS BY CONSTRAINING INFORMATION FLOW
Adversarial learning methods have been proposed for a wide range of applications, but the training of adversarial models can be notoriously unstable. Effectively balancing the performance of the generator and discriminator is critical, since a discriminator that achieves very high accuracy will produce relatively uninformative gradients. In this work, we propose a simple and general technique to constrain information flow in the discriminator by means of an information bottleneck. By enforcing a constraint on the mutual information between the observations and the discriminator's internal representation, we can effectively modulate the discriminator's accuracy and maintain useful and informative gradients. We demonstrate that our proposed variational discriminator bottleneck (VDB) leads to significant improvements across three distinct application areas for adversarial learning algorithms. Our primary evaluation studies the applicability of the VDB to imitation learning of dynamic continuous control skills, such as running. We show that our method can learn such skills directly from raw video demonstrations, substantially outperforming prior adversarial imitation learning methods. The VDB can also be combined with adversarial inverse reinforcement learning to learn parsimonious reward functions that can be transferred and re-optimized in new settings. Finally, we demonstrate that VDB can train GANs more effectively for image generation, improving upon a number of prior stabilization methods. (Video  1 )

Summary:
The authors propose to apply the Deep Variational Information Bottleneck (VIB) method of [1] on discriminator networks in various adversarial-learning-based scenarios. They propose a way to adaptively update the value for the bêta hyper-parameter to respect the constraint on I(X,Z). Their technique is shown to stabilize/allow training when P_g and P_data do not overlap, similarly to WGAN and gradient-penalty based approaches, by essentially pushing their representation distributions (p_z) to overlap with the mutual information bottleneck. It can also be considered as an adaptive version of instance noise, which serves the same goal. The method is evaluated on different adversarial learning setup (imitation learning, inverse reinforcement learning and GANs), where it compares positively to most related methods. Best results for ‘classical’ adversarial learning for image generation are however obtained when combining the proposed VIB with gradient penalty (which outperforms by itself the VGAN in this case).
Pros :
- This paper brings a good amount of evidence of the benefits to use the VIB formulation to adversarial learning by first showing the effect of such approach on a toy example, and then applying it to more complex scenarios, where it also boosts performance. The numerous experiments and analyses have great value and are a necessity as this paper mostly applies the VIB to new learning challenges.
- The proposition of a principled way of adaptively varying the value of Bêta to actually respect more closely the constraint I(X,Z) < I_c, which to my knowledge [1] does not perform, is definitely appealing and seems to work better than fixed Bêtas and does also bring the KL divergence to the desired I_c.
- The technique is fairly simple to implement and can be combined with other stabilization techniques such as gradient penalties on the discriminator.
Cons:
- In my view, the novelty of the approach is somewhat limited, as it seems like a straightforward application of the VIB from [1] for discriminators in adversarial learning, with the difference of using an adaptive Bêta.
- I think the Bêta-VAE [2] paper is definitely related to this paper and to the paper on which it is based [1] and should thus be cited as the authors use a similar regularization technique, albeit from a different perspective, that restricts I(X,Z) in an auto-encoding task.
- I think the content of batches used to regularize E(z|x) w.r.t. to the KL divergence should be clarified, as the description of p^tilde “being a mixture of the target distribution and the generator” (Section 4) can let the implementation details be ambiguous. I think batches containing samples from both distributions can cause problems as the expectation of the KL divergence on a batch can be low even if the samples from both distributions are projected into different parts of the manifold. This makes me think batches are separated? Either way, this should be more clearly stated in the text.
- The last results for  the ‘traditional’ GAN+VIB show that in this case, gradient penalty (GP) alone outperforms the proposed VGAN, and that both can be combined for best results. I thus wonder if the results in all other experiments could show similar trends if GP had been tested in these cases as well. In the imitation learning task, authors compare with instance noise, but not with GP, which for me are both related to VIB in what they try to accomplish. Was GP tested in Imitation Learning/Inverse RL ? Was it better? Could it still be combined with VIB for better results?
- In the saliency map of Figure 5, I’m unclear as to what the colors represent (especially on the GAIL side). I doubt that this is simply due to the colormap used, but this colormap should be presented.
Overall, I think this is an interesting and relevant paper that I am very likely to suggest to peers working on adversarial learning, and should therefore be presented. I think the limited novelty is counterbalanced by the quality of empirical analysis. Some clarity issues and missing citations should be easy to correct. I appreciate the comparison and combination with a competitive method (Gradient Penalty) in Section 5.3, but I wish similar results were present in the other experiments, in order to inform readers if, in these cases as well, combining VIB with GP leads to the best performance.
[1] Deep Variational Information Bottleneck, (Alemi et al. 2017)
[2] beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework (Higgins et al. 2017)


The non-convex nature of the loss landscape of deep neural networks (DNN) lends them the intuition that over the course of training, stochastic optimization algorithms explore different regions of the loss surface by entering and escaping many local minima due to the noise induced by mini-batches. But is this really the case? This question couples the geometry of the DNN loss landscape with how stochastic optimization algorithms like SGD interact with it during training. Answering this question may help us qualitatively understand the dynamics of deep neural network optimization. We show evidence through qualitative and quantitative experiments that mini-batch SGD rarely crosses barriers during DNN optimization. As we show, the mini-batch induced noise helps SGD explore different regions of the loss surface using a seemingly different mechanism. To complement this finding, we also investigate the qualitative reason behind the slowing down of this exploration when using larger batch-sizes. We show this happens because gradients from larger batch-sizes align more with the top eigenvectors of the Hessian, which makes SGD oscillate in the proximity of the parameter initialization, thus preventing exploration.

The subject of how a given algorithm explores the landscape is still a poorly understood area in training neural networks. There is a large body of recent work that attempts to shed light on this puzzle, and each one tries to claim their share in the furthering of the understanding of the relationship between the geometry of the landscape and the dynamics that one chooses in optimization. The present paper is a fine addition to the literature with interesting observations and novel questions, however, it falls short in many core areas: An apparent work in progress that has a great potential.
It is safe to say that "A walk with SGD" has an important single focus in mind: Does the SGD cross over barriers in the weight space of the underlying neural network? This question, at its heart, is intimately linked with the many properties that are attributed to the modern algorithm of the choice and the way it navigates a given non-convex landscape. The paper claims to provide an almost negative answer to the question and thereby busting several myths that are attributed to the "trick" part of SGD algorithm. As good as it sounds, unfortunately, the paper falls short of providing a convincing evidence (be it theoretical or empirical), and the way it tries to frame itself unique and different in relation to related works only indicate a lack of deep understanding of the existing literature. Therefore, I think there are several ways the paper should be improved before it is ready.
A major question (that I hope will easily be addressed) is on the definition of the barrier itself. According to the text, a barrier is defined judging by the minima of two 1-dimensional segments that connect weights connecting three consecutive steps: if the minimum of the line segment defined by the latter step is larger than the former, then it declared that a barrier is crossed. In a low dimensional world, this makes total sense, however, I fail to understand what kind of barrier it implies on the geometry of the landscape: Can the 1-dimensional lines be on the sides of a valley? Can one find *another* 1-dimensional projection for which the inequality is broken? How do such dependencies change the understanding of the problem? And if one is indeed only interested in the flat line segments (since SGD is making discrete steps), then one can, in principle, observe barrier crossing in a convex problem, as well? Is there an argument for otherwise? Or if it is a notion that applies equally well in a convex case then how should we really think about the barrier crossing? On the opposite point of view, can one not imagine a barrier crossing that doesn't appear in this triangular inequality above?
The paper is full of empirical evidence that is guided by a simple observable that is very intuitive, however, it lacks a comprehensive discussion on the new quantity they propose that I consider a major flaw, but that I think (hope) that the authors can fix very easily. Some minor points that would improve the readability and clarity for the reader:
- The figures are not very reader-friendly, this can be improved by better using the whitespaces in the paper but it can also be improved by finding further observables that would summarize the observations instead of showing individual consecutive line interpolations.
- What are the values of the y-axis in Figure 5 and 6? Are they the top eigenvalues of the Hessian?
- In the models that are compared in Figure 7, what are their generalization properties (early stopping and otherwise)?
- The interpretation at the end of p. 6 may be a good motivation for the reader if it had been introduced earlier for that section.
Finally, Section 5 reads very strangely, I have hard times understanding why certain phrases exist the way they are in this part. Here are further notes on Section 5:
- Why the whole first paragraph focused on hot the current paper is different than Goodfellow et al (2014) (it is obvious that it is for a different purpose), or why do we read the sentence "Li et al (2017b) also visualize...." What way do they visualize, is it the only paper that does visualization, what's the relation with the current paper and barrier crossing?
- For the second paragraph, I can suggest another paper, https://arxiv.org/abs/1803.06969, that was at ICML which also looks at the diffusion process through the parameter distance at different times which is similar to Hoffer et al. which also claims no barrier crossing similar to the present paper.
- However, my main issue is the exact connection between diffusion and no barrier crossing and it's connection to SGD preferring wide local minima instead of a narrow one. The second paragraph of the conclusion touches upon this subject. But it is not entirely clear how they are linked (except for the brittle SDE approximation at Li et al (see https://arxiv.org/abs/1810.00004)). Overall, the paper would benefit a lot from the discussion on why it is preferable to have SGD choose one basin over another in the beginning, as it is, it looks like the paper has another agenda behind the scenes.
- In the fourth paragraph of the conclusion, the paper refers to three papers that link DNN to spin glasses, in two of the (older) references the networks are far from what we have today, and the third one is far from "showing" anything between DNN and spin glass. In any case, what's the link between the aspects studied there with the present paper?
- Finally, the paper claims at the last few sentences that the works referred a little bit earlier look at the loss surface "in isolation from the optimization dynamics", however, many of those works cited have their empirical observations much like the current paper, and clearly they all "study the DNN loss surface along the trajectory of SGD" necessarily as it is the way to find local minima, saddle points, paths, curvature etc... The present paper is already very interesting and full of novel insight, I fail to see the value of struggling to stand out like this.
Overall, I think the paper is a very interesting step forward in understanding SGD dynamics on the DNN landscape. And, even though it has many shortcomings as it currently stands, I think it has a lot of room to improve.


Derivative-free optimization (DFO) using trust region methods is frequently used for machine learning applications, such as (hyper-)parameter optimization without the derivatives of objective functions known. Inspired by the recent work in continuous-time minimizers, our work models the common trust region methods with the exploration-exploitation using a dynamical system coupling a pair of dynamical processes. While the first exploration process searches the minimum of the blackbox function through minimizing a time-evolving surrogation function, another exploitation process updates the surrogation function time-to-time using the points traversed by the exploration process. The efficiency of derivativefree optimization thus depends on how the two processes couple. In this paper, we propose a novel dynamical system, namely SHE 2 -Stochastic Hamiltonian Exploration and Exploitation, that surrogates the subregions of blackbox function using a time-evolving quadratic function, then explores and tracks the minimum of the quadratic functions using a fast-converging Hamiltonian system. The SHE 2 algorithm is later provided as a discrete-time numerical approximation to the system. To further accelerate optimization, we present P-SHE 2 that parallelizes multiple SHE 2 threads for concurrent exploration and exploitation. Experiment results based on a wide range of machine learning applications show that P-SHE 2 outperform a boarder range of derivative-free optimization algorithms with faster convergence speed under the same settings.

This paper suggests a continuous-time framework consisting of two coupled processes in order to perform derivative-free optimization. The first process optimizes a surrogate function, while the second process updates the surrogate function. This continuous-time process is then discretized in order to be run on various machine learning datasets. Overall, I think this is an interesting idea as competing methods do have high computational complexity costs. However, I’m not satisfied with the current state of the paper that does not properly discuss notions of complexity of their own method compared to existing methods.
1) “The computational and storage complexity for (convex) surrogates is extremely high.” The discussion in this paragraph is too superficial and not precise enough.
a) First of all, the authors only discuss quadratic models but one can of course use linear models as well, see two references below (including work by Powell referenced there):
Chapter 9 in Nocedal, J., & Wright, S. J. (2006). Numerical optimization 2nd.
Conn, A. R., Scheinberg, K., & Vicente, L. N. (2009). Global convergence of general derivative-free trust-region algorithms to first-and second-order critical points. SIAM Journal on Optimization, 20(1), 387-415.
I think this discussion should also be more precise, the authors claim the cost is extremely high but I would really expect a discussion comparing the complexity of this method with the complexity of their own approach. As discussed in Nocedal (reference above) the cost of each iteration with a linear model is O(n^3) instead of O(n^4) where n is the number of interpolation points. Perhaps this can also be improved with more recent developments, the authors should do a more thorough literature review.
b) What is the complexity of the methods cited in the paper that rely on Gaussian processes?
(including (Wu et al., 2017) and mini-batch (Lyu et al., 2018)).
2) “The convergence of trust region methods cannot be guaranteed for high-dimensional nonconvex DFO”
Two remarks: a) This statement is incorrect as there are global convergence guarantees for derivative-free trust-region algorithms, see e.g.
Conn, A. R., Scheinberg, K., & Vicente, L. N. (2009). Global convergence of general derivative-free trust-region algorithms to first-and second-order critical points. SIAM Journal on Optimization, 20(1), 387-415.
In chapter 10, you will find global convergence guarantees for both first-order and second-order critical points.
b) The authors seem to emphasize high-dimensional problems although the convergence guarantees above still apply. For high-order models, the dimension does have an effect, please elaborate on what specific comment you would like to make. Finally, can you comment on whether the lower bounds derived by Jamieson mentioned depend on the dimension.
3) Quadratic loss function
The method developed by the authors rely on the use of a quadratic loss function. Can you comment on generalizing the results derived in the paper to more general loss functions? It seems that the computational complexity wouldn’t increase as much as existing DFO methods. Again, I think it would be interesting to give a more in-depth discussion of the complexity of your approach.
4) Convergence rate
The authors used a perturbed variant of the second-order ODE defined in Su et al. 2014. The noise added to the ODE implies that the analysis derived in Su et al. 2014 does not apply as is. In order to deal with the noise the authors show that unbiased noise does not affect the asymptotic convergence. I think the authors could get strong non-asymptotic convergence results. In a nutshell, one could use tools from Ito calculus in order to bound the effect of the noise in the derivative of the Hamiltonian used in Lemma 1. See following references:
Li, Q., Tai, C., et al. (2015). Stochastic modified equations and adaptive stochastic gradient algorithms. arXiv preprint arXiv:1511.06251.
Krichene, W., Bayen, A., and Bartlett, P. L. (2015). Accelerated mirror descent in continuous
and discrete time. In Advances in neural information processing systems, pages 2845–2853.
Of course, the above works rely on the use of derivatives but as mentioned earlier, one should be able to rely on existing DFO results to prove convergence. If you check Chapter 2 in the book of Conn et al. (see reference above), you will see that linear interpolation schemes already offer some simple bounds on the distance between the true gradient of the gradient of the model (assuming Lipschitz continuity and differentiability).
5) Noise
“The noise would help the system escape from an unstable stationary point in even shorter time”
Please add a relevant citation. For isotropic noise, see
Ge, R., Huang, F., Jin, C., and Yuan, Y. Escaping from saddle points-online stochastic gradient for tensor decomposition.
Jin, C., Netrapalli, P., and Jordan, M. I. Accelerated gradient descent escapes saddle points faster than gradient descent. arXiv preprint arXiv:1711.10456,
6) Figure 2
Instead of having 2 separate plots for iteration numbers and time per iteration, why don’t you combine them to show the loss vs time. This would make it easier for the reader to see the combined effect.
7) Empirical evaluation
a) There are not enough details provided to be able to reproduce the experiments. Reporting the range of the hyperparameters (Table 2 in the appendix) is not enough. How did you select the hyperparameters for each method? Especially step-size and batch-size which are critical for the performance of most algorithms.
b) I have to admit that I am not extremely familiar with common experimental evaluations used for derivative-free methods but the datasets used in the paper seem to be rather small. Can you please justify the choice of these datasets, perhaps citing other recent papers that use similar datasets?
8) Connection to existing solutions
The text is quite unclear but the authors seem to claim they establish a rigorous connection between their approach and particle swarm (“In terms of contribution, our research made as yet an rigorous analysis for Particle Swarm”). This however is not **rigorously** established and needs further explanation. The reference cited in the text (Kennedy 2011) does not appear to make any connection between particle swarm and accelerated gradient descent. Please elaborate.
9) SGD results
Why are the results for SGD only reported in Table 1 and not in the figure? Some results for SGD are better than for P-SHE2 so why are you bolding the numbers for P-SHE2?
It also seem surprising that SGD would achieve better results than the accelerated SGD method. What are the possible explanations?
10) Minor comments
- Corollaries 1 and 2 should probably be named as theorems. They are not derived from any other theorem in the paper. They are also not Corollaries in Su et al. 2014.
- Corollary 2 uses both X and Z.
- Equation 5, the last equation with \dot{V}(t): there is a dot missing on top of the first X(t)
“SHE2 should enjoy the same convergence rate Ω(1/T) without addressing any further assumptions” => What do you mean by “should”?
- There are **many** typos in the text!! e.g. “the the”, “is to used”, “convergeable”,... please have someone else proofread your submission.


SGD CONVERGES TO GLOBAL MINIMUM IN DEEP LEARNING VIA STAR-CONVEX PATH
Stochastic gradient descent (SGD) has been found to be surprisingly effective in training a variety of deep neural networks. However, there is still a lack of understanding on how and why SGD can train these complex networks towards a global minimum. In this study, we establish the convergence of SGD to a global minimum for nonconvex optimization problems that are commonly encountered in neural network training. Our argument exploits the following two important properties: 1) the training loss can achieve zero value (approximately), which has been widely observed in deep learning; 2) SGD follows a star-convex path, which is verified by various experiments in this paper. In such a context, our analysis shows that SGD, although has long been considered as a randomized algorithm, converges in an intrinsically deterministic manner to a global minimum.

This paper attempts to account for the success of SGD on training deep neural networks. Starting from two empirical observations: (1) deep neural networks can almost achieve zero training loss; (2) the path of iterates generated by SGD on these models follow approximately the “star convex path”, under the assumptions that individual functions share a global minima with respect to which the path of iterates generated by SGD satisfies the star convexity properties, the papers shows that the iterates converges to the global minima.
In terms of clarity, I think the paper can definitely benefit if the observations/assumptions/definitions/theorems are stated in a more formal and mathematically rigorous manner. For example:
- On page 3, “fact 1”: I don’t think “fact” is the right word here. “Fact” refers to what has been rigorously proved or verified, which is not the case for what is in the paper here. I believe “observation” is more appropriate. Also the assumption that l_i is non-negative should be formally added.
- On page 3, section 3.1: the x^* here is the last iteration produced by SGD. Then how can it be called the “global minima”? The caption of Figure 1 on page 4 is simply misleading.
- On page 4, the statement in definition 1 is more like a theorem than a definition. It is giving readers the impression that any path generated by SGD satisfies the star-convex condition, which is not the case here. A definition should look like “we call a path generated by SGD a star-convex path if it satisfies …”. Definition 2 on page 6 has the similar issue.
In terms of quality, while I believe the paper is technically correct,  I have one minor question here:
Page 3, Fact 1: How can you conclude that the set of common global minimizers are bounded? In fact I don’t believe this is true at all in general. If you have a ReLu network, you can scale the parameters as described in [1], then the model is invariant. Therefore, the set of common minimizer is definitely NOT bounded.
In terms of significance, I think this paper is very interesting as it attempts to draw the connection between the aforementioned observations and the convergence properties of SGD. Unfortunately I think that this paper is less significant than it has appeared to be, although the analysis appears to be correct.
First of all, all the analysis of this paper is based on one very important and very strong assumption, namely, all individual functions $l_i$ share at least one common global minimizer. The authors have attempted to justify this assumption by empirical evidences (figure 1). However, achieving near-zero loss is completely different from achieving exact zero because only when the model achieves exact zero can you argue that a common global minimizer exists.
Secondly, the claim that the iterate converges to the global minima is based on the assumption that the path follows an “epoch-wise star-convex” property. From this property, it only takes simple convex analysis to reach the conclusion of theorem 1 and 2. Meanwhile, the assumption that the path does follow the “epoch-wise start-convex” properties is not at all informative. It is not clear why or when the path would follow such a path. Therefore theorem 1 and 2 are not more informative than simply assuming the sequence converges to a global minimizer.
In fact, it is well-known that SGD with constant stepsize converges to the unique minimizer if one assumes the loss function F is strongly convex and the variance of the stochastic gradient g_k is bounded by a multiple of the norm-square of the true gradient:
Var(g_k) <= M ||∇F(x_k)||^2
Which is naturally satisfied if all individual functions share a common minimizer. Therefore, I don’t think the results shown in the paper is that surprising or novel.
With respect to the empirical evidence, the loss function l_i is assumed to be continuously differentiable with Lipschitz continuous gradients, which is not true for networks using ReLU-like activations. Then how can the paper use models like Alexnet to justify the theory? Also, if what the authors claim is true, then the stochastic gradient would have vanishing variance as it approaches x^*. Can the authors show this empirically?
In summary, I think this paper is definitely interesting, but the significance is not as much as it would appear.
Ref:
[1] Dinh, L., Pascanu, R., Bengio, S., & Bengio, Y. (2017). Sharp minima can generalize for deep nets. arXiv preprint arXiv:1703.04933.


We formulate stochastic gradient descent (SGD) as a novel factorised Bayesian filtering problem, in which each parameter is inferred separately, conditioned on the corresponding backpropagated gradient. Inference in this setting naturally gives rise to BRMSprop and BAdam: Bayesian variants of RMSprop and Adam. Remarkably, the Bayesian approach recovers many features of state-of-the-art adaptive SGD methods, including amongst others root-mean-square normalization, Nesterov acceleration and AdamW. As such, the Bayesian approach provides one explanation for the empirical effectiveness of state-of-the-art adaptive SGD algorithms. Empirically comparing BRMSprop and BAdam with naive RMSprop and Adam on MNIST, we find that Bayesian methods have the potential to considerably reduce test loss and classification error.

In this work, the authors attempt to unify existing adaptive gradient methods under the Bayesian filtering framework with the dynamical prior.  In Ollivier, 2017, a framework is proposed to connect Bayesian filtering and natural gradient.  On the other hand,  in Khan et al., 2018. an approach is proposed to connect natural gradient and adaptive gradient methods.  The main contributions of this work are (1)  introducing a dynamical prior and (2) recovering RMSProp and Adam as special cases.
However, the proposed dynamical prior is very similar to the fading memory technique used in Ollivier, 2017. (see Proposition 3 of Ollivier, 2017)
Furthermore, the authors argue that this work recovers a root-mean-square form while Khan et al., 2018 recovers a different sum-square form. Unfortunately, the authors have to use a series of unnatural approximations to recover the root-mean-square form. In fact, as mentioned in Khan, 2017b  this proposed method without these approximations is also a mean-square form. (also see Eq (2.28-2.29) of Ollivier, 2017)
Since the authors mainly follow Ollivier, 2017 and make unnatural approximations,  the work has a limited impact.  To get a higher rating, the authors should clearly give justifications and insights of these approximations.
Detailed comments:
(1) On Page 1,  "The typical approach to Bayesian filtering, where we infer a distribution, ... jointly, forces us to use extremely strong, factorised approximations, and it is legitimate to worry that these strong approximations might meaningfully disrupt the ability of Bayesian filtering to give close-to-optimal updates.   ... we instead consider ... that incorporates factorisation into the problem setting, and therefore requires fewer approximations downstream. "
The proposed method is equivalent to jointly perform Kalman filtering with full-covariance with an additional diagonal-approximation step. This additional step might also meaningfully disrupt the ability of Bayesian filtering. Furthermore, such approximation ignores the off-diagonal terms in the low-rank approximation at Eq (8).
Minor: You should use \approx at Eq (8) since a rank-1 approximation is used.
(2) On page 2, "It has been noted that under specific circumstances, natural gradient is approximate Bayesian filtering (Ollivier, 2017), allowing us to link Bayesian filtering to the rich literature on natural gradients.  However, this only occurs when the dynamical prior in the Bayesian filtering problem has a specific form: the parameters being fixed over time (i.e.  arguably an online data, rather than a true Bayesian filtering setting)."
The authors should comment the difference between the dynamical prior and the fading memory technique (see Proposition 3 of Ollivier, 2017) where at page 14 of Ollivier, 2017, Ollivier mentions that "this is equivalent ... or to the addition of an artificial process noise ... in the model".  I think Ollivier's idea is very similar to the dynamical prior used at Eq (1) of this submission.  Furthermore, the second-order Taylor expansion with a Fisher information-based estimation of Hessian (see the equation below Eq(1) of this submission) is exactly the same as Ollivier's Extended Kalman filter (see  Eq 2.25 at Lemma 9  and Lemma 10 of Ollivier, 2017).  The authors should cite Ollivier, 2017.
Minor: Eq (6) should be E_p [ - \nabla_z^2 \log p(d|z) ] = E_p  [ e e^T ], where "-", the negative sign is missing. Please see the definition of the Fisher information matrix.

(3) On page 2, "While there have been attempts to use natural gradients to recover the Adam or RMSprop root-mean-square form for the gradient normalizer, in practice a different sum-square form emerges (Khan & Lin, 2017; Khan et al., 2018). In contrast, we show that to recover the Adam or RMSprop form for the gradient normalizer."
Khan et al., 2018 is a mean-square form for variational inference due to the entropy term of the variational distribution. (see Sec 3 and 5 of  Khan et al., 2018 and Khan, 2017b )
Unfortunately, the "root-mean-square form" does not appear naturally in this submission. In practice, the proposed update is also a mean-square form  (see Eq (2.28-2.29) of Ollivier, 2017 and Khan, 2017b) without a series of unnatural approximations used in this submission.
To justify these assumptions, the authors should explain when "the steady state posterior variance" (see sec 2.21) and  "a self-consistent solution" (see sec 7.1) achieve.  As far as I know, \sigma^2_t = \sigma^2_{t+1} in sec 2.2.1 only holds in the limit case when t-> \inifity.  Why does the equality hold at each time step t? The authors should give a justification or an intuition about these approximations since this paper is a theory paper. Please also see my next point.
(4) Section 7.1 is also confusing.
In sec 7.1, the authors assume that A \in O(\eta). However, A=\eta^2/(2\sigma^2) in sec 2.2 and A_{1,1} =  ( \eta_w^2+\eta^2 )/ (2\sigma^2) at Eq (14). In both cases, A can be \in O(\eta^2). This is very *critical* since the authors argue that O(\eta^3) can be neglected in sec 7.1.  The authors use this point to show that Adam is a special case.
If A \in O(\eta^2), we know that "A \Sigma_{post}" \in O(\eta^3) should be neglected. At the last equation on page 10,  the authors do not neglect "A \Sigma_{post}". Why?  The authors should clarify this point to avoid doing *selective* neglection.  Again, the impact of this paper should be inspiring new adaptive methods.
The authors also mention that the second-order term in A is neglected in sec 7.2. Any justification?
References
[1] Ollivier, Yann. "Online Natural Gradient as a Kalman Filter." arXiv preprint arXiv:1703.00209 (2017).
[2] Khan, Mohammad Emtiyaz, and Wu Lin. "Conjugate-computation variational inference: Converting variational inference in non-conjugate models to inferences in conjugate models." arXiv preprint arXiv:1703.04265 (2017).
[3] Khan, Mohammad Emtiyaz, et al. "Vprop: Variational Inference using RMSprop." arXiv preprint arXiv:1712.01038 (2017b).
[4] Khan, Mohammad Emtiyaz, et al. "Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam" (2018)


LEARNING TO SIMULATE
Simulation is a useful tool in situations where training data for machine learning models is costly to annotate or even hard to acquire. In this work, we propose a reinforcement learning-based method for automatically adjusting the parameters of any (non-differentiable) simulator, thereby controlling the distribution of synthesized data in order to maximize the accuracy of a model trained on that data. In contrast to prior art that hand-crafts these simulation parameters or adjusts only parts of the available parameters, our approach fully controls the simulator with the actual underlying goal of maximizing accuracy, rather than mimicking the real data distribution or randomly generating a large volume of data. We find that our approach (i) quickly converges to the optimal simulation parameters in controlled experiments and (ii) can indeed discover good sets of parameters for an image rendering simulator in actual computer vision applications.

Pros:
* Using RL to choose the simulator parameters is a good idea. It does not sound too novel, but at the same time I am not personally aware of this having been explored in the past (Note that my confidence is 4, so maybe other reviewers might be able to chime in on this point)
* In theory, you don't need domain adaptation or other sim2real techniques if you manage to get the optimal parameters of the simulator with this method.
* Certain attributes of the method were evaluated sufficiently: eg the number of training epochs for each policy iteration, the dataset size generated in each iteration, and whether initialization was random or not in each iteration.
Cons:
* Experiments were underwhelming, and the choice of problems/parameters to tune was not the right one for the problem.
* Parts of the paper could be clearer
QUALITY:
* I believe that although the idea is great, but the quality of the experiments could have been higher. Firstly, better problems could have been selected to showcase the method. I was excited to see experiments with CARLA, but was underwhelmed when I realized that the only parameter of the simulator that the method controlled was the number and the type of cars in the scene, and the task of interest was a car counting task (for which not much detail was provided). This would have been much more interesting and useful to the community if more parameters, including rendering parameters (like lighting, shading, textures, etc) were part of the search space. Similarly, the semantic segmentation task could have used more than one category. But even for the one category, there were no previous methods considered, and the only comparison was between random parameters and the learned ones, where we only see marginal improvement, and what I perceive to be particularly low IoU for the car (although it'd help to know what's the SOTA there for comparison) For both vision applications I could help but wonder why the authors did not try to simply train on the  validation set to give us another datapoint to evaluate the performance of the method: this is data that *is* used for training the outer loop, so it does beg the question of what is the advantage of having hte inner loop.
CLARITY:
* The writing of the paper was clear for the most part, however the experimental section could have been clearer. I was wondering how model/hyperparameter selection was performed? Was there another validation set (other than the one used to train the outer loop)
* The proposed policy is dubbed "its". What does it mean?
* It's not clear what is a "deliberately adversarial" initialization. Could you elaborate?
* The letter R is used to mean "reward" and "rendering". This is confusing. Similarly some symbols are not explicitly explained (eg S) Generally Section 2.3 is particularly unclear and confusing until one gets to the experimental section.
* Section 3 discusses the technique and states that "we can thus generate or oversample unusual situations that would otherwise not be part of the training data" I believe it is important to state that, as the method is presented, this is only true if the "validation" data is varied enough and includes such situations. I believe this would be more applicable if eg rendering parameters were varied and matched the optimal ones.
* Also the method is presented as orthogonal to domain adaptation and other sim-to-real techniques. However, I do not necessarily believe that this paper should be discussed outside the context of such techniques like domain randomization, Cycada, PixelDA etc. Even though these (esp. the latter ones) focus on vision, I do think it sets the right context.
ORIGINALITY:
* As far as I'm aware noone has tried something similar yet. However, I'm not confident on this.
SIGNIFICANCE:
* Although the idea is good, I don't think that the approach to select the simulation parameters presented in the experiments in such a way is significant. I think that eg doing so for rendering parameters would be a lot more powerful and useful (and probably a lot more challenging). Also, I think that a single set of parameters (which seems to be what the goal is in this work) is not what one wants to achieve; rather one wants to find a good range of parameters that can help in the downstream task.


Positive-unlabeled (PU) learning addresses the problem of learning a binary classifier from positive (P) and unlabeled (U) data. It is often applied to situations where negative (N) data are difficult to be fully labeled. However, collecting a non-representative N set that contains only a small portion of all possible N data can be much easier in many practical situations. This paper studies a novel classification framework which incorporates such biased N (bN) data in PU learning. The fact that the training N data are biased also makes our work very different from those of standard semi-supervised learning. We provide an empirical risk minimization-based method to address this PUbN classification problem. Our approach can be regarded as a variant of traditional example-reweighting algorithms, with the weight of each example computed through a preliminary step that draws inspiration from PU learning. We also derive an estimation error bound for the proposed method. Experimental results demonstrate the effectiveness of our algorithm in not only PUbN learning scenarios but also ordinary PU leaning scenarios on several benchmark datasets.

The authors rst present standard binary (positive negative or PN) classica-
tion, followed by positive unlabeled (PU) classication, that they motivate with
examples, such as one-class remote sensing classication. The new setting that
they introduce and study is called positive unlabeled biaised negative (PUbN
classication) and adds a biaised negative sample to PU learning. They give
motivating examples and compare this setting to the existing literature. A con-
vincing case is made regarding the dierence between the PUbN problem and
the known problems of semi-supervised learning and dataset shift.
They start by recalling the notations and nature of standard binary classi-
cation, PU classication and the nnPU (non-negative PU) strategy, as in the
previous PU learning papers. Then, they present the semi-supervised setting
under the name PNU learning, which simply studies the minimization of a con-
vex combination of the PN risk and the PU risk. As in PU learning, a correction
exists to avoid considering the estimate of the negative risk to be negative, re-
ferred to as nnPNU.
Finally, the authors introduce PUbN learning as the problem in which we
only have access to negatives that follow the law p(x\mid y = -1; s = +1), where s
is a latent variable that formalizes the bias.
As in PU learning, the authors derive an unbiased estimator of the risk that
involves only distributions for which data is available. However, they need
to reweight the P and bN distribution by the unknown posterior probability
sigma(x) = p(s = +1\mid x) of s. Considering s as the label, the problem of learning
a probabilistic classier separating the elements for which s = +1 and s = 􀀀1
can be seen as a PU learning problem, which gives an estimator ^ of sigma,
and makes the method practical.
They derive estimation error bounds, that depend on the mean squared
difference between sigma and sigma^ and a term of order n^-1/2 where n is the cardinal
of the smallest sample. They considered the function ^ as a xed function in
their bounds, which implies that the bounds are only true if some of the data is
kept for the estimation of sigma^. Finally, they present a variant of their algorithm
for PU learning, named PUbNnN where unlabeled instances are not all given
the same weight, but weighted according to sigma hat. The experiments use neural networks with stochastic optimization, on the classic datasets MNIST, CIFAR-10 and 20 Newsgroup. They report better per-
formance using their technique on all datasets. The authors documented their
experiences thoroughly in the appendix. However, I did not find information
about the nature of the estimator of the posterior probability sigma^, which is im-
portant for reproducibility. Furthermore, in appendix B, choosing sigma^ = 0 will
minimize the criterion . Finally, they proceed to justify the dominance of
the variant of their method over usual nnPU learning.


Artificial neural networks are built on the basic operation of linear combination and non-linear activation function. Theoretically this structure can approximate any continuous function with three layer architecture. But in practice learning the parameters of such network can be hard. Also the choice of activation function can greatly impact the performance of the network. In this paper we are proposing to replace the basic linear combination operation with non-linear operations that do away with the need of additional non-linear activation function. To this end we are proposing the use of elementary morphological operations (dilation and erosion) as the basic operation in neurons. We show that these networks (Denoted as DenMo-Net) with morphological operations can approximate any smooth function requiring less number of parameters than what is necessary for normal neural networks. The results show that our network perform favorably when compared with similar structured network.

The authors introduce Morph-Net, a single layer neural network where
the mapping is performed using morphological dilation and erosion.
I was expecting something applied to convolutional networks as such operators
are very popular in image processing, so the naming is a bit misleading.
It is shown that the proposed network can approximate any smooth function,
assuming a sufficiently large number of hidden neurons, that is a nice result.
Clarity should be improved, for example it is mentioned that the structuring
element is learned but never clearly explained how and what difficulties it poses.
In the main text it is written that alpha is {-1, 1}, which would result in a
combinatorial search, but never explained how it is learned in practice.
This is shown only in the appendix but it is not clear to me that using a binarization
with the weights is not prone to degenerate solutions and/or to learn at all
if proper initialization is not used.
Did the authors experiment with smooth versions or other form of binarization with
straight-through estimator or sampling?
In the proof for theorem 1 it is not clear if the convergence of the proposed
network is faster or slower than that of a classic single layer network.
The main result of the paper is that the structuring element can be learned,
but there is no discussion on what it is learned. Also, there is no comparison
on related approaches that try to learn the structuring element in an end-to-end
fashion such as [1].
Experiments lack a more thorough comparison with state-of-the-art and at least
an ablation study to show that the proposed approach is effective and has merit.
For example, what is the relative contribution of using dilation and erosion
jointly versus either one of them.
What is the comparison with a winner-take-all unit over groups of neurons
such as max-pooling?
It seems that extending the work to multiple layers should be trivial but it is
not reported and is left to future investigations. This hints at issues with
the optimization and should be discussed, is it related to the binarization
mentioned above?
Overall the idea is interesting but the way the structuring element is learned
should be discussed in more details and exemplified visually. Experiments need
to be improved and overall applicability is uncertain at this stage.
=======
[1] Masci et al., A Learning Framework for Morphological Operators Using Counter--Harmonic Mean.


Current deep learning based text classification methods are limited by their ability to achieve fast learning and generalization when the data is scarce. We address this problem by integrating a meta-learning procedure that uses the knowledge learned across many tasks as an inductive bias towards better natural language understanding. Inspired by the Model-Agnostic Meta-Learning framework (MAML), we introduce the Attentive Task-Agnostic Meta-Learning (ATAML) algorithm for text classification. The proposed ATAML is designed to encourage task-agnostic representation learning by way of task-agnostic parameterization and facilitate task-specific adaptation via attention mechanisms. We provide evidence to show that the attention mechanism in ATAML has a synergistic effect on learning performance. Our experimental results reveal that, for few-shot text classification tasks, gradient-based meta-learning approaches outperform popular transfer learning methods. In comparisons with models trained from random initialization, pretrained models and meta trained MAML, our proposed ATAML method generalizes better on single-label and multi-label classification tasks in miniRCV1 and miniReuters-21578 datasets.

Summary of paper: For the few shot text classification task, train a model with MAML where only a subset of parameters (attention parameters in this case) are updated in the inner loop of MAML. The empirical results suggest that this improves over the MAML baseline.
I found this paper confusingly written. The authors hop between a focus on meta-learning to a focus on attention, and it remains unclear to me how these are connected. The description of models is poor -- for example, the ablation mentioned in 4.5.3 is still confusing to me (if the attention parameters are not updated in the inner loop of MAML, then what is?). Furthermore, even basic choices of notation, like A with a bar underneath in a crowded table, seem poorly thought out.
I find the focus on attention a bit bizarre. It's unclear to me how any experiments in the paper suggest that attention is a critical aspect of meta-learning in this model. The TAML baseline (without attention) underperforms the ATAML model (with attention), but all that means is that attention improves representational power, which is not surprising. Why is attention considered an important aspect of meta learning?
To me, the most interesting aspect of this work is the idea of not updating every parameter in the MAML inner loop. So far, I've seen all MAML works update all parameters. The experiments suggest that updating a small subset of parameters can improve results significantly in the 1-shot regime, but the gap between normal MAML and the subset MAML is much smaller in the 5-shot regime. This result suggests updating a subset of parameters can serve as a method to combat overfitting, as the 1-shot regime is much more data constrained than the 5-shot regime.
It's unfortunate that the authors do not dig further down this line of reasoning. When does the gap between MAML on all parameters and only on a subset of parameters become near-zero? Does the choice of the subset of parameters matter? For example, instead of updating the attention weights, what happens if the bottommost weights are updated? How would using pretrained parameters (e.g., language modeling pretraining) in meta-learning affect these results? In general, what can be learned about overfitting in MAML?
To conclude, the paper is not written well and has a distracting focus on attention. While it raises an interesting question about MAML and overfitting, it does not have the experiments needed to explore this topic well.


In many reinforcement learning applications, it is crucial that the agent interacts with the environment only through safe policies, i.e., policies that do not take the agent to certain undesirable situations. These problems are often formulated as a constrained Markov decision process (CMDP) in which the agent's goal is to optimize its main objective while not violating a number of safety constraints. In this paper, we propose safe policy optimization algorithms that are based on the Lyapunov approach to CMDPs, an approach that has well-established theoretical guarantees in control engineering. We first show how to generate a set of state-dependent Lyapunov constraints from the original CMDP safety constraints. We then propose safe policy gradient (PG) algorithms that train a neural network policy using deep deterministic policy gradient (DDPG) or proximal policy optimization (PPO), while guaranteeing near-constraint satisfaction at every policy update by projecting either the policy parameter or the action onto the set of feasible solutions induced by the linearized Lyapunov constraints. Unlike the existing (safe) constrained policy gradient algorithms, ours are more data efficient as they are able to utilize both on-policy and off-policy data. Furthermore, the action-projection version of our algorithms often leads to less conservative policy updates and allows for natural integration into an end-to-end PG training pipeline. We evaluate our algorithms and compare them with constrained policy optimization (CPO) and the Lagrangian method on several high-dimensional continuous state and action simulated robot locomotion tasks, in which the agent must satisfy certain safety constraints while minimizing its expected cumulative cost.

￼
To my understanding, this paper builds on prior work from Chow et al. to apply Lyapunov-based safe optimization to the policy-gradient setting. This seems is similar to work by Achiam 2017. While this work seems like an interesting framework for encompassing several classes of constrained policy optimization settings in the Lyapunov-based setting, I have some concerns about the evaluation methodology.
It is claimed that the paper compares against “two baselines, CPO and the Lagrangian method, on several robot locomotion tasks, in which the agent must satisfy certain safety constraints while minimizing its expected cumulative cost.” Then it is stated in the experimental section “However since backtracking line-search in TRPO can be computationally expensive, and it may lead to conservative policy updates, without loss of generality we adopt the original construction of CPO to create a PPO counterpart of CPO (which coincides with SPPO) and use that as our baseline.” This seems directly to contrast to the earlier statement which states that it is unclear how to modify the CPO methodology to other RL algorithms. Moreover, is this really a fair comparison? The original method has been modified to form a new baseline and I’m not sure that it is “without loss of generality”.
Also, it is unclear whether the results can be accepted at face value. Are these averaged across several random seeds and trials? Will performance hold across them? What would be the variance? Recent work has shown that taking 1 run especially in MuJoCo environments doesn’t necessarily provide statistically significant values. In fact the original CPO paper shows the standard deviations across several random seeds and compares directly against an earlier work in this way (PDO). Moreover, it is unclear why CPO was not directly compared against and neither was the "equivalent" baseline not compared on similar environments as in the original CPO paper.
Comments:
Figure 3 is difficult to parse, the ends of the graphs are cut off. Maybe putting the y axis into log format would help with readability here or having the metrics be in a table.


COMPLEMENT OBJECTIVE TRAINING
Learning with a primary objective, such as softmax cross entropy for classification and sequence generation, has been the norm for training deep neural networks for years. Although being a widely-adopted approach, using cross entropy as the primary objective exploits mostly the information from the ground-truth class for maximizing data likelihood, and largely ignores information from the complement (incorrect) classes. We argue that, in addition to the primary objective, training also using a complement objective that leverages information from the complement classes can be effective in improving model performance. This motivates us to study a new training paradigm that maximizes the likelihood of the groundtruth class while neutralizing the probabilities of the complement classes. We conduct extensive experiments on multiple tasks ranging from computer vision to natural language understanding. The experimental results confirm that, compared to the conventional training with just one primary objective, training also with the complement objective further improves the performance of the state-of-the-art models across all tasks. In addition to the accuracy improvement, we also show that models trained with both primary and complement objectives are more robust to single-step adversarial attacks.

========
Summary
========
The paper deals with the training of neural networks for classification or sequence generation tasks, using a cross-entropy loss. Minimizing the cross-entropy means maximizing the predicted probabilities of the ground-truth classes (averaged over the samples). The authors introduce a "complementary entropy" loss with the goal of minimizing the predicted probabilities of the complementary (incorrect) classes. To do that, they use the average of sample-wise entropy over the complement classes. By maximizing this entropy, the predicted complementary probabilities are encouraged to be equal and therefore, the authors claim that it neutralizes them as the number of classes grows large. The proposed training procedure, named COT, consists of alternating between the optimization of the two losses.
The procedure is tested on image classification tasks with different datasets (CIFAR-10, CIFAR-100, Street View House Numbers, Tiny ImageNet and ImageNet), machine translation (training using IWSLT dataset, validation and test using TED tst2012/2013 datasets), and speech recognition (Gooogle Commands dataset). In the experiments, COT outperforms state-of-the-art models for each task/dataset.
Adversarial attacks are also considered for the classification of images of CIFAR-10: using the Fast Gradient Sign and Basic Iterative Fast Gradient Sign methods on different models, adversarial examples specifically designed for each model, are generated. Then results of these models are compared to COT on these examples. The authors admit
that the results are biased since the adversarial attacks only target part of the COT objective, hence more accurate comparisons should be done in future work.
===========================
 Main comments and questions
===========================
End of page 1: "the model behavior for classes other than the ground  truth stays unharnessed and not well-defined". The probabilities  should still sum up to 1, so if the ground truth one is maximized,  the others are actually implicitly minimized. No?
Page 3, sec 2.1: "optimizing on the complement entropy drives ŷ_ij to 1/(K − 1)". I believe that it drives each term ŷ_ij /(1 − ŷ_ig ) to be equal to 1/(K-1). Therefore, it drives ŷ_ij to (1 − ŷ_ig)/(K-1) for j!=g.
This indeed flattens the ŷ_ij for j!=g, but the effect on ŷ_ig is not controlled. In particular this latter can decrease. Then in the next step of the algorithm, ŷ_ig will be maximized, but with no explicit control over the complementary probabilities. There are two objectives that are optimized over the same variable theta. So the question is, are we sure that this procedure will converge? What prevents situations where the probabilities will alternate between two values?
For example, with 4 classes, we look at the predicted probabilities of a given sample of class 1:
Suppose after step 1 of Algo 1, the predicted probabilities are:  0.5 0.3 0.1 0.1
After step 2:  0.1 0.3 0.3 0.3
Then step 1: 0.5 0.3 0.1 0.1
Then step 2: 0.1 0.3 0.3 0.3
And so on... Can this happen? Or why not? Did the algorithm have trouble converging in any of the experiments?
Sec 3.1:
"additional efforts for tuning hyper-parameters might be required for optimizers to achieve the best performance": Which hyper-parameters are considered here? If it is the learning rate, why not use a different one, tuned for each objective?
Sec 3.2:
The additional optimization makes each training iteration more costly. How much more? How do the total running times of COT compare to the ones of the baselines? I think this should be mentioned in the paper.
Sec 3.4:
As the authors mention, the results are biased and so the comparison is not fair here. Therefore I wonder about the  relevance of this section. Isn't there an easy way to adapt the attacks to the two objectives to be able to illustrate the conjectured robustness of COT? For example, naively having a two steps perturbation of the input: one based on the gradient of the primary objective and then perturb the result using the gradient of the complementary objective?
===========================
Secondary comments and typos
===========================
Page 3, sec 2.1: "...the proposed COT also optimizes the complement objective for neutralizing the predicted probabilities...", using maximizes instead of optimizes would be clearer.
In the definition of the complement entropy, equation (2), C takes as parameter only y^hat_Cbar but then in the formula, ŷ_ig appears. Shouldn't C take all \hat_y as an argument in this case?
Algorithm 1 page 4: I find it confusing that the (artificial) variable that appears in the argmin (resp. argmax) is theta_{t-1}
(resp. theta'_t) which is the previous parameter. Is there a reason for this choice?
Sec 3:
"We perform extensive experiments to evaluate COT on the tasks" --> COT on tasks
"compare it with the baseline algorithms that achieve state-of-the-art in the respective domain." --> domainS
"to evaluate the model’s robustness trained by COT when attacked" needs reformulation.
"we select a state- of-the-art model that has the open-source implementation" --> an open-source implementation
Sec 3.2:
Figure 4: why is the median reported and not the mean (as in Figure 3, Tables 2 and 3)?
Table 3 and 4: why is it the validation error that is reported and not the test error?
Sec 3.3:
"Neural machine translation (NMT) has populated the use of neural sequence models": populated has not the intended meaning.
"We apply the same pre-processing steps as shown in the model" --> in the paper?
Sec 3.4:
"We believe that the models trained using COT are generalized better" --> "..using COT generalize better"
"using both FGSM and I-FGSM method" --> methodS
"The baseline models are the same as Section 3.2." --> as in Section 3.2.
"the number of iteration is set at 10." --> to 10
"using complement objective may help defend adversarial attacks." --> defend against
"Studying on COT and adversarial attacks.." --> could be better formulated
References: there are some inconsistencies (e.g.: initials versus first name)
Pros
====
- Paper is clear and well-written
- It seems to me that it is a new original idea
- Wide applicability
- Extensive convincing experimental results
Cons
====
- No theoretical guarantee that the procedure should converge
- The training time may be twice longer (to clarify)
- The adversarial section, as it is,  does not seem relevant for me


Currently available survival analysis methods are limited in their ability to deal with complex, heterogeneous, and longitudinal data such as that available in primary care records, or in their ability to deal with multiple competing risks. This paper develops a novel deep learning architecture that flexibly incorporates the available longitudinal data comprising various repeated measurements (rather than only the last available measurements) in order to issue dynamically updated survival predictions for one or multiple competing risk(s). Unlike existing works in the survival analysis on the basis of longitudinal data, the proposed method learns the time-to-event distributions without specifying underlying stochastic assumptions of the longitudinal or the time-to-event processes. Thus, our method is able to learn associations between the longitudinal data and the various associated risks in a fully data-driven fashion. We demonstrate the power of our method by applying it to real-world longitudinal datasets and show a drastic improvement over state-of-the-art methods in discriminative performance. Furthermore, our analysis of the variable importance and dynamic survival predictions will yield a better understanding of the predicted risks which will result in more effective health care.

The paper proposes a deep architecture that conducts survival analysis from longitudinal data where multiple competing risks are present. Experimental results demonstrate the effectiveness of the proposed method.  Specific comments follow:
1. A primary concern in the reviewer's opinion is the scalability of the architecture. While the reviewer appreciates the discussion of the scalability issue in terms of the output layer in the paper, the architecture might also not scalable if the number of competing risks is large because of the increase of the cause-specific subnetworks in the architecture.  Overall, the reviewer finds the lack of a principled approach to deal with competing risks and long time horizon presented in the paper.  Since dealing with competing risks in survival analysis is the goal of the paper, the reviewer finds the method presented insufficient for acceptance. As a remedy, for example, for the output layer, can the author consider the use of a neural net to model o_k at a particular time using time and f_{c_k}() as input?
Other issues:
2. it will be nice to explain (1) and (2) a little after presenting the formula.
3. page 5, $\mathbf{y}_j$ should also be explained because the next place where $\mathbf{y}_j$ is present is (4), which is one page later.
4. the term "dynamic survival analysis" is also obscure. What exactly does "dynamic" mean? To the reviewer's understanding, compared to standard survival analysis, the architecture models directly from raw longitudinal data of repeated measurements, and hence is called "dynamic".
5. even the "dynamic" part of the dynamic survival analysis is not very novel, see, for example,
Recurrent Marked Temporal Point Processes: Embedding Event History to Vector
and the follow-up works in the use of deep learning for point process modeling.
===============After Reading Authors' Response ================
The reviewer would like to thank the authors for their detailed response and careful revision of the paper to address the reviewer's concern. However, the reviewer is not persuaded by the authors' response. Specifically,
1. the reviewer is not satisfied with the explanation and modification to address the scalability issue stemming from both the cause-specific subnetworks and the output layer. Simplifying the structure and parameterization of cause-specific subnetworks when many are present seems like a comprise rather than a principled approach to address the issue. The same is true for the exponentially distributed parameterization of the output layer.
2. It is the reviewer's impression that for point process neural networks, it is possible to use the covariate information for prediction, as opposed to the claim given by the author.


STOCHASTIC GRADIENT/MIRROR DESCENT: MINI- MAX OPTIMALITY AND IMPLICIT REGULARIZATION
Stochastic descent methods (of the gradient and mirror varieties) have become increasingly popular in optimization. In fact, it is now widely recognized that the success of deep learning is not only due to the special deep architecture of the models, but also due to the behavior of the stochastic descent methods used, which play a key role in reaching "good" solutions that generalize well to unseen data. In an attempt to shed some light on why this is the case, we revisit some minimax properties of stochastic gradient descent (SGD) for the square loss of linear models-originally developed in the 1990's-and extend them to general stochastic mirror descent (SMD) algorithms for general loss functions and nonlinear models. In particular, we show that there is a fundamental identity which holds for SMD (and SGD) under very general conditions, and which implies the minimax optimality of SMD (and SGD) for sufficiently small step size, and for a general class of loss functions and general nonlinear models. We further show that this identity can be used to naturally establish other properties of SMD (and SGD), namely convergence and implicit regularization for over-parameterized linear models (in what is now being called the "interpolating regime"), some of which have been shown in certain cases in prior literature. We also argue how this identity can be used in the so-called "highly over-parameterized" nonlinear setting (where the number of parameters far exceeds the number of data points) to provide insights into why SMD (and SGD) may have similar convergence and implicit regularization properties for deep learning.

This is a very interesting paper and it suggests a novel way to think of "implicit regularization". The power of this paper lies in its simplicity and its inspiring that such almost-easy arguments could be made to get so much insight. It suggests that minimizers of the Bregrman divergence are an alternative characterization of the asymptotic end-points of  "Stochastic Mirror Descent" (SMD) when it converges. So choice of the strongly convex potential function in SMD is itself a regularizer!
Its a very timely paper given the increasing consensus that "implicit regularization" is what drives a lot of deep-learning heuristics. This paper at its technical core suggests a modified notion of Bregman-like divergence (equation 15) which on its own does not need a strongly convex potential. Then the paper goes on to show that there is an invariant of the iterations of SMD along its iterations which involves a certain relationship (equation 18) between the usual Bregman divergence and their modified divergence. I am eager to see if such relationships can be shown to hold for more complicated iterative algorithms!
But there are a few points in the paper which are not clear and probably need more explanation and let me list them here. ( and these are the issues that prevent me from giving this paper a very high rating despite my initial enthusiasm )
1.
Can the authors explain how is the minimax optimality result of Theorem 6 (and Corollary 7) related to the main result of the paper which is probably Proposition 8 and and 9? Is that minimax optimiality a different insight separate from the main line of the arguments (which I believe is Proposition 8 and 9)?
2.
Is the gain in Proposition 9 over Proposition 8 is all about using loss convexity to ensure that the SMD converges and w_\infty exists?
3.
The paper has highly insufficient comparisons to many recent other papers on the idea of "implicit bias" like, https://arxiv.org/abs/1802.08246, https://arxiv.org/abs/1806.00468 and https://arxiv.org/abs/1710.10345. It seems pretty necessary that there be a section making a detailed comparison with these recent papers on similar themes.


MUSIC TRANSFORMER: GENERATING MUSIC WITH LONG-TERM STRUCTURE
Music relies heavily on repetition to build structure and meaning. Self-reference occurs on multiple timescales, from motifs to phrases to reusing of entire sections of music, such as in pieces with ABA structure. The Transformer (Vaswani et al., 2017), a sequence model based on self-attention, has achieved compelling results in many generation tasks that require maintaining long-range coherence. This suggests that self-attention might also be well-suited to modeling music. In musical composition and performance, however, relative timing is critically important. Existing approaches for representing relative positional information in the Transformer modulate attention based on pairwise distance (Shaw et al., 2018). This is impractical for long sequences such as musical compositions because their memory complexity for intermediate relative information is quadratic in the sequence length. We propose an algorithm that reduces their intermediate memory requirement to linear in the sequence length. This enables us to demonstrate that a Transformer with our modified relative attention mechanism can generate minute-long compositions (thousands of steps) with compelling structure, generate continuations that coherently elaborate on a given motif, and in a seq2seq setup generate accompaniments conditioned on melodies 1 . We evaluate the Transformer with our relative attention mechanism on two datasets, JSB Chorales and Maestro, and obtain state-of-the-art results on the latter.

The authors address the problem raised by applying a fully attentional network (FAN) to model music.
They argue clearly for the need of relational positional embedding in that problem (instead of absolute positional as in vanilla FAN), and highlight the quadratic memory footprint of the current solution (Shaw et al. 2018).
The main contribution of the paper is a solution to this, consisting in a smart idea (sect 3.4.1 and 3.4.2) which allows them to compute relative embeddings without quadratic overhead.
The model performs indeed better than Shaw et al.'s on the single data-set they compared both. On the other one, the argument is that Shaw et al. 2018 cannot be applied because the sequences are too long.
I have two concerns with the paper:
	1/ it is very hard to read at times. In particular, the main contribution took me several passes the understand. I list below a few recommendations for improvement
	2/ the main argument is that the model requires less memory and is faster. However, the only empirical evidence in that direction is given in the introduction (Sect 1.1., second paragraph).
		The following points remain unclear to me:
			a) why can't the Relative Transformer be applied to Piano-e composition. What is the maximal length that is possible?
			b) how much faster / less memory is the relative music transformers? The only data-point is in Sect 1.1., which seems indeed impressive (but then one wonders why this is not exploited further). A deeper analysis of the comparative memory footprint would greatly strengthen the paper in my opinion.

Why "music" relative transformers? Nothing in the model restrict it to that use case. The use of FAN over audio has been explored with limited success, one of the reasons being that - similarly to this use-case here - audio sequences tend to be longer than text.

minor comments:
	- abstract, ln9: there seems to be a verb missing
	- p1,ln-2: "dramatic" improvements seems to be exaggerated
	- p2,ln11: "too long". too long for what?
	- p4,ln15: (Table 1). is one sentence by itself. Also, a clear explanation of that table is missing
	- p5,item 2: an explanation in formula would be helpful for those not familiar with reshaping
	- Fig3: it seems very anecdotical. Similar green bloxes might be placed on the left plot
	- sect4.1.1,ln3. that sentence does not parse
	- Table 2: what is cpsi?
	- $l$ is nicer formatted as $\ell$
	- care should be taken to render the Figures more readable (notably the quality of Fig 4, and labels of Fig 7)
	- footnotes in Figures are not displayed (Table 2 and 4)
	- the description of the human evaluation leaves some open questions. I could not come up with 180 ratings (shouldn't it be 180 * 3 ratings?). Also, at least the values of Relative Transformer vs other 3 models should be shown (or all 6 comparisons). Here you call "relative transformer" your model, previously you used that term to refer to (Shaw et al. 2018).
		when reporting statistical significance, there are some omissions which should be clarified.
	- (Shaw et al. 2018) has been published at NAACL. For such an important citation, you should update the reference from the arxiv version.


EVALUATING ROBUSTNESS OF NEURAL NETWORKS WITH MIXED INTEGER PROGRAMMING
Neural networks trained only to optimize for training accuracy can often be fooled by adversarial examples -slightly perturbed inputs misclassified with high confidence. Verification of networks enables us to gauge their vulnerability to such adversarial examples. We formulate verification of piecewise-linear neural networks as a mixed integer program. On a representative task of finding minimum adversarial distortions, our verifier is two to three orders of magnitude quicker than the state-of-the-art. We achieve this computational speedup via tight formulations for non-linearities, as well as a novel presolve algorithm that makes full use of all information available. The computational speedup allows us to verify properties on convolutional and residual networks with over 100,000 ReLUs -several orders of magnitude more than networks previously verified by any complete verifier. In particular, we determine for the first time the exact adversarial accuracy of an MNIST classifier to perturbations with bounded l ∞ norm = 0.1: for this classifier, we find an adversarial example for 4.38% of samples, and a certificate of robustness to norm-bounded perturbations for the remainder. Across all robust training procedures and network architectures considered, and for both the MNIST and CIFAR-10 datasets, we are able to certify more samples than the state-of-the-art and find more adversarial examples than a strong first-order attack.

The authors perform a careful study of mixed integer linear programming approaches for verifying robustness of neural networks to adversarial perturbations. They propose three enhancements to MILP formulations of neural network verification: Asymmetric bounds, restricted domain and progressive bound tightening, which lead to significantly more scalable verification algorithms vis-a-vis prior work. They study the effectiveness of MILP solvers both in terms of verifying robustness (compared to other complete/incomplete verifiers) and generating adversarial attacks (compared to PGD attacks) and show that their approach compares favorable across a number of architectures on MNIST and CIFAR-10. They perform careful ablation studies to validate the importance of the
Quality: The paper is very well written and organized. The problem is certainly of great interest to the deep learning community, given the difficulty of properly evaluating (and then improving) defenses against adversarial attacks. The experiments are done carefully with convincing ablation studies.
Clarity: The authors explain the relevant concepts carefully and all the experimental results are clearly written and explained.
Originality: The authors propose conceptually simple but practically significant enhancements to MILP formulations of neural network verification. However, the novelty wrt https://arxiv.org/pdf/1711.00455.pdf is not discussed carefully in my view (the  asymmetric bounds were already studied in this paper, as well as a novel branch and bound strategy). The progressive bound tightening is a novel idea as far as I can see - however, the ablation experiments show that this idea is not significant in terms of performance improvement. In terms of experiments, the authors indeed obtain strong results on verified adversarial error rates and generate attacks that PGD is unable to - however, again the results do not outperform latest results (in terms of the  best achievable upper bounds on verified error rates) available well before the ICLR deadline - https://arxiv.org/pdf/1805.12514.pdf . It would be great if the authors addressed these issues in a revised version of the paper.
Significance: The work does establish a strong algorithm for complete verification of neural networks along with several ideas that are critical to obtain strong performance with this approach.
Question:
1. I am unclear on the "restricted domain" contribution claimed in the paper - is this just exploiting the fact that the inputs to the classifier are normalized to a given range, in addition to being no more than eps away from the nominal input?
Cons
1. The authors do not compare their approach to that of https://arxiv.org/pdf/1711.00455.pdf , both in terms of conceptual novelty and in terms of experimental results. In particular, it is not clear to me whether the authors' approach remains superior on domains where tight bounds on the neural networks inputs are not available, like the problems studied in the ACAS system in the ReLuPlex paper.
2. The authors' MILP solution approach relies on having access to the state of the art commercial MILP solver Gurobi. While Gurobi is free for academic research use, for large scale neural network verification applications, this does restrict use of the approach (particularly due to limited licenses being available). It would be interesting to see a comparison that uses a freely available MILP solver (like scip.zib.de) to see how critical the approach's scalability depends on the quality of the MILP solver.
3. The authors do not outperform the latest SOA numbers in terms of verified adversarial error rates on MNIST and CIFAR classifers. It would be good to see a comparison on results from https://arxiv.org/pdf/1711.00455.pdf  (I believe the training code and trained networks are available online).


Sequence generation models such as recurrent networks can be trained with a diverse set of learning algorithms. For example, maximum likelihood learning is simple and efficient, yet suffers from the exposure bias problem. Reinforcement learning like policy gradient addresses the problem but can have prohibitively poor exploration efficiency. A variety of other algorithms such as RAML, SPG, and data noising, have also been developed from different perspectives. This paper establishes a formal connection between these algorithms. We present a generalized entropy regularized policy optimization formulation, and show that the apparently divergent algorithms can all be reformulated as special instances of the framework, with the only difference being the configurations of reward function and a couple of hyperparameters. The unified interpretation offers a systematic view of the varying properties of exploration and learning efficiency. Besides, based on the framework, we present a new algorithm that dynamically interpolates among the existing algorithms for improved learning. Experiments on machine translation and text summarization demonstrate the superiority of the proposed algorithm.

The authors propose a more unified view of disparate methods for training sequence models. Specifically, a multi-term objective L(q,theta) consisting of:
1) The standard reward maximization objective of policy gradient, E_{p_\theta}[R],
2) A weighted (weight alpha) reverse KL divergence of the parametric policy and a non-parameteric policy q,
3) A weighted (weight beta) entropy term on q,
Is proposed for sequence training (see equation (1). L can be iteratively optimized by solving for q given p_\theta, and the \theta given q (see eq. 2).
This framework mathematically generalizes softmax-policy gradient (SPG, alpha=1, beta=0), and reward-augmented maximum likelihood (alpha=0, beta=temperature), and also standard entropy regularized policy gradient (alpha=0), among other algorithms.
The paper is well written, and the approach sensible. However, combining SPG and RAML by introducing their respective regularization terms is a rather straightforward exercise, and so seems quite incremental.
Other major concerns are:
1) the true utility of the model, and
2) the integrity of the experiments.
Wrt:
1), While RAML was a significant contribution at the time, it is now well established that RAML generally doesn't perform well at all in practice due to exposure bias (not conditioning on it's own previous predictions during training). Moreover SPG, as the authors point out, was supposed to address the need for ML pre-training, but required much engineering to work. The fact is that REINFORCE-based policy gradient methods are still more effective than these methods, provided they have a good baseline to reduce varince. Which brings me to point
2) Was MIXER run with a learned baseline and judiciously optimized? Table 1 suggests that MIXER can outpeform ML by only 0.1 Bleu points, and outpeformed by RAML? Something is wrong with your implementation then. Moreover, there are techniques like self-critical sequence training (SCST), which far outpeform MIXER, and we haven't even discussed Actor-Critic baselines...
In summary, the contribution over RAML and SPG in combining them is quite incremental, and the practical importance of combining them is questionable, as is the integrity of the presented experiments, given how poorly MIXER is reported perform, and the omission of stronger baselines like SCST and AC methods. Also, a paper on essentially the same approach was submitted and rejected from ICLR 2018(https://openreview.net/pdf?id=H1Nyf7W0Z), although this paper is better written, and puts the method more fully in context with existing work, I think that several of the concerns with that paper apply here as well.
Look forward to the authors' feedback, and additional/corrected results - will certainly update my score if these concerns are addressed. In particular, if this generalization can significantly outpeform existing methods it generalizes with non-degenerate settings, this would overcome the more incremental contribution of combining SPG and RAML.
Current Ratings:
Evaluation      2/5: Results are not consistent with previous results (e.g. MIXER results). Stronger baselines such as SCST and AC are not considered.
Clarity         5/5: Clear paper, well written.
Significance    3/5: RAML and SPG have not been established as important methods in practice, so combining them is less interesting.
Originality     2/5: RAML and SPG are fairly straightforward to combine for experts interested in these methods.
Rating          4/10 Okay but not good enough, reject.
Confidence      5/5
Pros:
- Generalizes RAML and SPG (and also standard entropy-regularized policy gradient).
- Well written paper, clean generalization.
Cons:
- RAML and SPG have not been established as important methods in practice.
- generalization of RAML and SPG is straightforward,  incremental.
- Existing baselines in the paper (i.e. MIXER) do not perform as expected (i.e barely better than ML, worse than RAML)
- Stronger REINFORCE-based algorthms like SCST, as well as Actor-critic algorithms, have not been compared.
Update after author responses:
--------------------------------------------
Authors, thank you for your feedback.
While it is true that generalizing RAML and SPG into a common framework is not trivial, the presented framework simply augments the dual form of SPG (i.e. REPS [16] in the SPG paper) with a RAML term. Furthermore, the MLE interpretation discussed is contained within the RAML paper, and the reductions to RAML and SPG are straightforward by design, and so do not really provide much new insight. Considering this, I feel that the importance of the paper largely rests on investigating and establishing the utility of the approach experimentally.
Wrt the experiments, I appreciate that the authors took the time to investigate the poor performance of MIXER. However, the unusally poor performance of MIXER remains unexplained, and falls short even of scheduled sampling (SS), which suggests a lingering major issue. REINFORCE techniques rely on 1) strong baselines, verified by the authors, 2) larger batch sizes to reduce variance, and 3) pre-training to reduce variance and facilitate efficient exploration. If the MLE is undertrained or overtrained (the latter the more likely issue given the plots), then MIXER will perform poorly. Actually, it is now standard practice to pre-train with MLE+SS before RL training, and this is really the (also dynamically weighted objective) baseline that should be compared against. The current REINFORCE results (MIXER or otherwise) really need to be updated (or at the least removed, as they are not captured by the framework, but the comparison to PG methods is important!).
More generally, I feel that the experiments are not yet comprehensive enough. While the authors have shown that they can outperform SPG and RAML with a scheduled objective, it is not currently clear how sensitive/robust the results are to the term weight scheduling, or even what most appropriate general weights/scheduling approach actually is.
Overall I feel that the paper is still in need of substantial maturation before publication, although I have revised my score slightly upward.


SIGNSGD VIA ZEROTH-ORDER ORACLE
In this paper, we design and analyze a new zeroth-order (ZO) stochastic optimization algorithm, ZO-signSGD, which enjoys dual advantages of gradient-free operations and signSGD. The latter requires only the sign information of gradient estimates but is able to achieve a comparable or even better convergence speed than SGD-type algorithms. Our study shows that ZO-signSGD requires
On the application side we explore the connection between ZO-signSGD and black-box adversarial attacks in robust deep learning. Our empirical evaluations on image classification datasets MNIST and CIFAR-10 demonstrate the superior performance of ZO-signSGD on the generation of adversarial examples from black-box neural networks.

The authors proposed a zero-order version of the recent signSGD algorithm, by replacing the stochastic gradient with a usual function difference estimate. Similar convergence rates as signSGD were obtained, with an additional sqrt(d) factor which is typical in zero-order methods. Three (typical) gradient estimates based on function values were discussed. Overall, the obtained results are relatively straightforward combination of signSGD with existing zero-order techniques.
Quality: The technical part of this paper seems to be solid. The experiments, on the other hand, are quite ambiguous. First off, why do you choose that peculiar least squares binary classification problem on page 7? Is Assumption A2 satisfied for this problem? Why not use logistic regression? The experimental results are also strange: Why would ZO-signSGD converge faster than ZO-SGD or any other ZO variant? Shouldn't they enjoy similar rates of convergence? Why would taking the sign make the algorithm converge faster? Note that the original motivation for signSGD is not for faster convergence but less communication. For the second set of experiment, how do you apply ZO-SGD to generate adversarial examples? Again, why do we expect ZO-signSGD to perform better than ZO-SGD?
Clarity: This paper is mostly well-written, but the authors at times largely overclaim their contributions or exaggerate the technical challenges.
-- Page 2, 2nd line: the authors claim that "Our analysis removes the impractical assumption of b = O(T)", but in the later examples (page 6, top), they require q = O(T). How is this any different than b = O(T)? Even worse, the former case also require b = n, i.e., there is no stochasity at all...
-- Assumption A2: how crucial is this assumption for obtaining the convergence results? note that not many functions have Lipschitz continuous bounded gradients... (logistic regression is an example)
-- Page 4, top: "ZO-signSGD has no restriction on the mini-batch size b"? The rates at the end of page 5 suggests otherwise if we want the bound to go to 0 (due to the term sqrt(d/b)).
-- Page 4, top: the last two technical challenges do not make sense: once we replace f by f_mu, these difficulties go away immediately, and it is well-known how to relate f_mu with f.
Originality: The originality seems to be limited. Contrary to what the authors claimed, I found the established results to be relatively straightforward combination of signSGD and existing zero-order techniques. Can the authors elaborate on what additional difficulties they need to overcome in order to extend existing zero-order results to the signSGD case?
Significance: The proposed zero-order version of signSGD may potentially be significant in applications where gradient information is not available and yet distributed optimization is needed. This, however, is not demonstrated in the paper as the authors never considered distributed optimization.
##### added after author response #####
I appreciate the authors effort in trying to make their contributions precise and appropriate. The connection between ZO-signSGD and adversarial examples is further elaborated, which I agree is an interesting and potentially fruitful direction. I commend the authors for supplying further experiments to explain the pros and cons of the proposed algorithms. Many of the concerns in my original review were largely alleviate/addressed. As such, I have raised my original evaluation.