Convolutional Neural Networks (CNNs) have achieved tremendous success for many computer vision tasks, which shows a promising perspective of deploying CNNs on mobile platforms. An obstacle to this promising perspective is the tension between intensive resource consumption of CNNs and limited resource budget on mobile platforms. Existing works generally utilize a simpler architecture with lower accuracy for a higher energy-efficiency, i.e., trading accuracy for resource consumption. An emerging opportunity to both increasing accuracy and decreasing resource consumption is class skew, i.e., the strong temporal and spatial locality of the appearance of classes. However, it is challenging to efficiently utilize the class skew due to both the frequent switches and the huge number of class skews. Existing works use transfer learning to adapt the model towards the class skew during runtime, which consumes resource intensively. In this paper, we propose probability layer, an easily-implemented and highly flexible add-on module to adapt the model efficiently during runtime without any fine-tuning and achieving an equivalent or better performance than transfer learning. Further, both increasing accuracy and decreasing resource consumption can be achieved during runtime through the combination of probability layer and pruning methods.

The paper proposes a simple idea to calibrate probabilities outputted by a CNN model to adapt easily to environments where class distributions change with space and time (and are often skewed). The paper shows that such a simple approach is sufficient to get good accuracies without requiring any costly retraining or transfer learning. Thereby proving to give benefits in terms of resource consumption and at the same time giving better results than the state of the art.
However,
A] The proposed calibration doesn't take any CNN specific details into consideration, rather it is a general calibration method which was also proposed in Saerens et. al, 2002 (cited in the paper). It is unclear why the paper specifically talks about CNN.
B] The proposed Class Skew Detector is a simple method. Change-point detection is a well-studied area. The paper lacks a literature review in this area and a reasoning of why the proposed approach is preferred. Also, an independent analysis of how the class skew detector behaves in the face of rapidly changing class skews versus slow changing class skews is warranted here. Particularly, given that the paper proposes to use this approaches in mobile which may work in both rapid and slow changing class skews.
C] The Class Skew Detector is dependent on the base model. Thus, it is also likely that the empirical distribution estimated is biased and yet the final accuracies reported are much higher than the base model accuracies. There is something interesting happening here. An analysis of the robustness of the proposed approach in the face of noisy class skew detection could potentially make this paper a stronger work.
D] The analysis in the paper has largely focused on pre-trained models. However, another analysis that could have been useful here is, varying the quality of the classifier (e.g. classifier trained on skewed training data vs. balanced training data) and measuring how the quality of the classifier correlates with the final performance. Maybe even attempt to answer the question "which classifiers are likely to work with this approach?" In fact, this analysis can be either done in a general context of any classifier or just CNN's and identifying whether certain properties of CNN help in getting better performance.
The paper lacks novelty and at the same time, it is not quite compensating that with a detailed analysis of the work. The problem is interesting and I like the work because the approach is simple and the results look good. I think with a stronger focus on more detailed analysis, this can be a good submission to an applied conference like MobiCom etc.
By the way, the paper is riddled with several spelling errors -
"filed" -> "field", page 1, second paragraph, last line
"complimentary" -> "complementary", page 2, section 2, paragraph 1, last line
"epoches" -> "epochs", page 2, section 2, transfer learning, second paragraph, second last line
"CNNs does not use" -> "CNNs do not use", page 3, section 3, intuition, first paragraph, first line
"formular" -> "formula", page 4, above equation 4
Equation 4 has a typo in the denominator, P_t(i) should be P_t(j), same with Equation 5
"obstained" -> "obtained", page 7, second paragraph, first line
"adaptation" is almost everywhere spelled as "adaption"


We consider new variants of optimization algorithms. Our algorithms are based on the observation that mini-batch of stochastic gradients in consecutive iterations do not change drastically and consequently may be predictable. Inspired by the similar setting in online learning literature called OPTI-MISTIC ONLINE LEARNING, we propose two new optimistic algorithms for AMSGrad and Adam, respectively, by exploiting the predictability of gradients. The new algorithms combine the idea of momentum method, adaptive gradient method, and algorithms in OPTIMISTIC ONLINE LEARNING, which leads to speed up in training deep neural nets in practice.

The paper proposes new online optimization algorithms by adding the idea of optimistic updates to the already popular components of adaptive preconditioning and momentum (as used in AMSGrad and ADAM). Such optimistic schemes attempt to guess the yet-unseen gradients before each update, which can lead to better regret guarantees when the guesses are accurate in a certain sense. This in turn can lead to faster convergence when the resulting algorithm is used in an optimization framework. The specific contribution of the present paper is proving formally that optimistic updates can indeed be combined with advanced methods like ADAM and AMSGrad, also providing a regret analysis of the former algorithm. On the practical front, the authors also propose a method closely resembling Anderson acceleration for guessing the next gradient, and the eventual scheme is shown to work well empirically in training deep neural networks.
The idea of optimistic updates has been popular in recent years within the online-learning literature, and has been used with particularly great success for achieving improved convergence guarantees for learning equilibria in games. More recently, optimistic updates have also appeared in more "practical" settings such as training GANs, where they were shown to improve stability of training. The present paper argues that the idea of optimism can be useful for large-scale optimization as well, if the gradient guesses are chosen appropriately.
I have lukewarm feelings about the paper. On the positive side, the proposed method is a natural and sensible combination of solid technical ideas, and its theoretical analysis appears to be correct. As the authors point out, their algorithm incorporates the idea of optimism in a much more natural way than the related optimistic ADAM algorithm previously proposed by Daskalakis et al. (2018) does. The experiments also indicate some advantage of optimism in the studied optimization problems.
On the other hand, the theoretical contribution is marginal: the algorithm and its analysis is a straightforward combination of previous ideas and the result itself doesn't strike me as surprising at all. Then again, perhaps this is more of a presentation issue, as it may be the case that the authors did not manage to highlight clearly enough the technical challenges they needed to overcome to prove their theoretical results. Furthermore, I find the method for guessing the gradients to be rather arbitrary and poorly explained---at least I'm not sure if anyone unfamiliar with the mentioned gradient extrapolation methods would find this approach to be sensible at all.
I am not fully convinced by the experimental results either, since I have an impression that the gradient-guessing method only introduces yet another knob to turn when tuning the hyperparameters, and it's not clear at all that this new dimension would indeed unlock levels of performance that were not attainable before. Indeed, the authors seem to fix all hyperparameters across all experiments and only switch around the optimistic components, rather than finding the best tuning for each individual algorithm and comparing the respective results. Also, I don't really see any qualitative improvements in the learning curves due to the new components---but maybe I just can't read these graphs properly since I have more of a "theorist" background.
The writing is mostly OK, although there is room for improvement in terms of English use (especially on the front of articles which seem to be off in almost every sentence).
Overall, I don't feel very comfortable about suggesting acceptance, mostly because I find the results to be rather unsurprising. I suggest that the authors try to convince me of the nontrivial challenges arising in the analysis, or about the definite practical advantage that optimism can buy for large-scale optimization.
Detailed comments
=================
- pp.1, abstract: "We consider new variants of optimization algorithms."---This sentence is rather vague and generic. I guess you wanted to refer to *convex* optimization algorithms, which is actually what you consider in the paper. No need to be embarrassed about assuming convexity...
- pp.1: A general nuisance with the typesetting that already shows on the first page is that italic and small capital fonts are used excessively and without any clearly identifiable logic. Please simplify.
- pp.1: "AdaGrad [...] exploits the geometry of the data and performs informative update"---this makes it sound like other algorithms make non-informative updates.
- pp.1: Regret was not defined even informally in the introduction, yet already some regret bounds are compared, highlighting that one "can be much smaller than O(\sqrt{T})". This is not very friendly for readers with no prior experience in online learning.
- pp.1: "Their regret analysis are the regret analysis in online learning."---What is this sentence trying to say?
- pp.2: For this discussion of FTRL, it would be useful to remark that this algorithm really only makes sense if the loss function is convex. Also related to this discussion: you mention that the bound for optimistic FTRL can be much smaller than \sqrt{T}, but never actually say that \sqrt{T} is minimax optimal---without this piece of context, this statement has little value.
- pp.3: "ADAM [...] does not converge to some specific convex functions."---I guess I understand what this sentence is trying to say, but it certainly doesn't say it right. (Why would an *algorithm* converge to a *function*?)
- pp.3, bottom: This description of "extrapolation methods" is utterly cryptic. What is x_t here? What is the "fixed point x^*"? Why is this scheme applicable at all here? (Why would one believe the errors to be near-linear in this case? Would this argument work at all for non-convex objectives?)
- pp.5, Lemma 1: Why would one expect D_\infty to be finite? In order to ensure this, one would need to project the iterates to a compact set.
- pp.5, right after lemma 1: "it does matter how g_t is generated" -> "it does *not* matter how g_t is generated"
- pp.6, top: "we claimed that it is smaller than O(\sqrt{T}) so that we are good here"---where exactly did this claim appear, and in what sense "are we good here"? Also, the norms in this paragraph should be squared.
- pp.6, Sec 3.2: While this section makes some interesting points, its tone feels a bit too apologetic. E.g., saying that "[you] are aware of" a previous algorithm that's similar to yours and doubling down on the claim that "the goals are different" makes the text feel like you're taking a defensive stance even though I can't see a clear reason for this. In my book, the approach you propose is clearly different and more natural for the purpose of your study.


Depending on how much information an adversary can access to, adversarial attacks can be classified as white-box attack and black-box attack. In both cases, optimization-based attack algorithms can achieve relatively low distortions and high attack success rates. However, they usually suffer from poor time and query complexities, thereby limiting their practical usefulness. In this work, we focus on the problem of developing efficient and effective optimization-based adversarial attack algorithms. In particular, we propose a novel adversarial attack framework for both white-box and black-box settings based on the non-convex Frank-Wolfe algorithm. We show in theory that the proposed attack algorithms are efficient with an O(1/ √ T ) convergence rate. The empirical results of attacking Inception V3 model and ResNet V2 model on the ImageNet dataset also verify the efficiency and effectiveness of the proposed algorithms. More specific, our proposed algorithms attain the highest attack success rate in both white-box and black-box attacks among all baselines, and are more time and query efficient than the stateof-the-art.

The paper investigates the Frank-Wolfe (FW) algorithm for constructing adversarial examples both in a white-box and black-box setting. The authors provide both a theoretical analysis (convergence to a stationary point) and experiments for an InceptionV3 network on ImageNet. The main claim is that the proposed algorithm can construct adversarial examples faster than various baselines (PGD, I-FGSM, CW, etc.), and from fewer queries in a black-box setting.
The FW algorithm is a classical method in optimization, but (to the best of my knowledge) has not yet been evaluated yet for constructing adversarial examples. Hence it is a natural question to understand whether FW performs significantly better than current algorithms in this context. Indeed, the authors find that FW is 6x - 20x faster for constructing white-box adversarial examples than a range of relevant baseline, which is a significant speed-up. However, there are several points about the experiments that are unclear to me:
- It is well known that the running times of optimization algorithms are highly dependent on various hyperparameters such as the step size. But the authors do not seem to describe how they chose the hyperparameters for the baselines algorithms. Hence it is unclear how large the running time improvement is compared to a well-tuned baseline.
- Other algorithms in the comparison achieve a better distortion (smaller perturbation). Since finding an adversarial with smaller perturbation is a harder problem, it is unclear how the algorithms compare for finding adversarial examples with similar distortion. Instead of reporting a single time-vs-distortion data point, the authors could show the full trade-off curve.
- The authors only provide running times, not the number of iterations. In principle all the algorithms should have a similar bottleneck in each iteration (computing a gradient for the input image), but it would be good to verify this with an iteration count vs success rate (or distortion) plot. This would also allow the authors to compare their theoretical iteration bound with experimental data.
In addition to these three main points, the authors could strengthen their results by providing experiments on another dataset (e.g., CIFAR-10) or model architecture (e.g., a ResNet), and by averaging over a larger number of test data points (currently 200).
Overall, I find the paper a promising contribution. But until the authors provide a more thorough experimental evaluation, I hesitate to recommend acceptance.
Additional comments:
The introduction contains a few statements that may paint an incomplete or confusing picture of the current literature in adversarial attacks on neural networks:

* The abstract claims that the poor time complexity of adversarial attacks limits their practical usefulness. However, the running time of attacks is typically measured in seconds and should not be the limiting element in real-world attacks on deep learning systems. I am not aware of a setting where the running time of an attack is the main computational bottleneck (outside adversarial training).
* The introduction distinguishes between "gradient-based methods" and "optimization-based methods". This distinction is potentially confusing to a reader since the gradient-based methods can be seen as optimization algorithms, and the optimization-based methods rely on gradients.
* The introduction claims that black-box attacks need to estimate gradients coordinate-wise. However, this is not the case already in some of the prior work that uses random directions for estimating gradients (e.g., the cited paper by Ilyas et al.)
I encourage the authors to clarify these points in an updated version of their paper.


Representations learnt through deep neural networks tend to be highly informative, but opaque in terms of what information they learn to encode. We introduce an approach to probabilistic modelling that learns to represent data with two separate deep representations: an invariant representation that encodes the information of the class from which the data belongs, and an equivariant representation that encodes the symmetry transformation defining the particular data point within the class manifold (equivariant in the sense that the representation varies naturally with symmetry transformations). This approach to representation learning is conceptually transparent, easy to implement, and in-principle generally applicable to any data comprised of discrete classes of continuous distributions (e.g. objects in images, topics in language, individuals in behavioural data). We demonstrate qualitatively compelling representation learning and competitive quantitative performance, in both supervised and semi-supervised settings, versus comparable modelling approaches in the literature with little fine tuning.

This paper is well written, and the quality of the figures is good. In this paper, the authors propose an invariant-covariant idea, which should be dated back at least to the bilinear models. The general direction is important and should be pursued further.
However, the literature is not well addressed. Eslami et al. 2018 have been cited, but some very important and related earlier works like:
[1] Kulkarni et al. 2015, Deep Convolutional Inverse Graphics Network
[2] Cheung et al. 2015, Discovering Hidden Factors of Variation in Deep Networks
were not discussed at all. The authors should certainly make an effort to discuss the connections and new developments beyond these works. At the end of section 1, the authors argue that the covariant vector could be more general, but in fact, these earlier works can achieve further equivalence, which is much stronger than the proposed covariance.
There is also an effort to compare this work to Sabour et al. 2017 and the general capsule idea. I would like to point out, the capsule concept is a much more fine-grained what & where separation rather than a coarse-grained class & pose separation in one shot. In a hierarchical representation, what & where can appear at any level as one class can consist of several parts each with a geometrical configuration space. So the comparison of this work to the generic capsule network is only superficial if the authors can not make the proposed architecture into a hierarchical separation. Besides different capsule network papers, I found another potentially useful reference on a fine-grained separation:
[3]Goroshin et al., Learning to Linearize Under Uncertainty
In the paper, it is argued several times that the latent vector r_y contains a rich set of global properties of class y, rather than just its label and the aim is that it can learn what the elements of the class manifold have in common. But this point is not supported well since we can always make a label and this latent vector r_y equivalent by a template. I think this point could be meaningful if we look at r_y's for different y, where each of the dimension may have some semantic meaning. Additional interpretation is certainly needed.
Under equation (3), "Note that v is inferred from r_y" should be "inferred from both r_y and x", which is pretty clear from the fig 5. Related to this, I could imagine some encoder can extract the 'style' directly from x, but here both r_y and x are used. I couldn't find any guarantee that v only contains the 'style' information based on the architecture with even this additional complication, could the authors comment on this?
Equation (5) is not really a marginalization and further equation (6) may not be a lower bound anymore. This is probably a relatively minor thing and a little extra care is probably enough.
The numbers in table 2 seems a little outdated.
To conclude, I like the general direction of separating the identity and configurations. The natural signals have hierarchical structures and the class manifold concept is not general enough to describe the regularities and provide a transparent representation. Rather, it's a good starting point. If the authors could carefully address the related prior works and help us understand the unique and original contributions of this work, this paper could be considered for publication.


Studying the evolution of information theoretic quantities during Stochastic Gradient Descent (SGD) learning of Artificial Neural Networks (ANNs) has gained popularity in recent years. Nevertheless, these type of experiments require estimating mutual information and entropy which becomes intractable for moderately large problems. In this work we propose a framework for understanding SGD learning in the information plane which consists of observing entropy and conditional entropy of the output labels of ANNs. Through experimental results and theoretical justifications it is shown that, under some assumptions, the SGD learning trajectories appear to be similar for different ANN architectures. First, the SGD learning is modeled as a Hidden Markov Process (HMP) whose entropy tends to increase to the maximum. Then, it is shown that the SGD learning trajectory appears to move close to the shortest path between the initial and final joint distributions in the space of probability measures equipped with the total variation metric. Furthermore, it is shown that the trajectory of learning in the information plane can provide an alternative for observing the learning process, with potentially richer information about the learning than the trajectories in training and test error.

This paper study the trajectory of H(\hat{y}) versus H(\hat{y}|y) on the information plane for stochastic gradient descent methods for training neural networks. This paper was inspired by (Ziv and Tishby 17'), but instead of measuring the mutual information I(X;T) and I(Y:T), this paper proposed to measure H(\hat{y}) and H(\hat{y}|y), which are much easier to compute but carries similar meaning as I(Y;T) and I(X;T).
The interesting part of this paper appears in Section 4, where the author makes a connection between the SGD training process and \alpha-SMLC(strong Markov learning chain). SMLC is just simply linear combination of the initial distribution and the final stable distribution of the labels. The authors show that the trajectory of the real experiment is similar to that of SMLC.
Generally I think the paper is well-written and clearly present the ideas. Here are some pros and cons.
Pros 1: The trajectory presented in this paper is much more reliable than that in (Ziv and Tishby 17'), since measuring the entropy and conditional entropy of discrete random variables are much easier. Also it is easy for people to believe that the trajectory holds for various neural network structure and various activation functions.
Pros 2: The connection to SMLC is interesting and it may contain lot of insights.
Cons 1: One of my major concern is --- if you look at the trajectory of the experiment v.s. SMLC (Figure 3), they look similar at first glance. But if you look at it carefully, you will notice that the color of them are different! For SGD, the trajectory goes to the turning point very soon (usually no more than 10% of the training steps), whereas SMLC goes to the turning point much slower. How do the authors think about this phenomenon and what does this mean?
Cons 2: This paper is going to be more meaningful if the author can provide some discussions, especially about (1) what does the shape trajectory mean (2) what do the connection between the trajectory and Markov chain means (3) how can these connections be potentially useful to improve training algorithm? I understand that these questions may not be clearly answerable, but the authors should make this paper more inspiring such that other researchers can think deeper after reading this paper.
Cons 3: I suggest the authors using SGD instead of GD throughout the paper. Usually GD means true gradient descent, but the paper is talking about batched stochastic gradient descent. GD does not have Markovity.
Generally, I think the paper is on the borderline. I think the paper is acceptable if the author can provide more insights (against Cons 2).


CMMI10
Traditional exploration methods in reinforcement learning (RL) require agents to perform random actions to find rewards. But these approaches struggle on sparse-reward domains like Montezuma's Revenge where the probability that any random action sequence leads to reward is extremely low. Recent algorithms have performed well on such tasks by encouraging agents to visit new states or perform new actions in relation to all prior training episodes (which we call acrosstraining novelty). But such algorithms do not consider whether an agent exhibits intra-life novelty: doing something new within the current episode, regardless of whether those behaviors have been performed in previous episodes. We hypothesize that across-training novelty might discourage agents from revisiting initially non-rewarding states that could become important stepping stones later in training-a problem remedied by encouraging intra-life novelty. We introduce Curiosity Search for deep reinforcement learning, or Deep Curiosity Search (DeepCS), which encourages intra-life exploration by rewarding agents for visiting as many different states as possible within each episode, and show that DeepCS matches the performance of current state-of-the-art methods on Montezuma's Revenge. We further show that DeepCS improves exploration on Amidar, Freeway, Gravitar, and Tutankham (many of which are hard exploration games). Surprisingly, DeepCS also doubles A2C performance on Seaquest, a game we would not have expected to benefit from intra-life exploration because the arena is small and already easily navigated by naive exploration techniques. In one run, DeepCS achieves a maximum training score of 80,000 points on Seaquest-higher than any methods other than Ape-X. The strong performance of DeepCS on these sparse-and dense-reward tasks suggests that encouraging intra-life novelty is an interesting, new approach for improving performance in Deep RL and motivates further research into hybridizing across-training and intra-life exploration methods.

This paper proposes use of intra-life coverage (an agent must visit all locations within each episode) for effective exploration in Atari games. This is in contrast of approaches that use inter-life coverage or curiosity metrics to incentivize exploration. The paper shows detailed results and analysis on 2 Atari games: Montezuma’s Revenge and Seaquest, and reports results on other games as well.
Strengths
1. Intuitively, the idea of intra-life curiosity is reasonable. The paper pursues this idea and provides experimental evidence towards it on 2 Atari games. It is able to show compelling improvements on the challenging Montezuma’s Revenge game.
Weaknesses
1. The two primary comparison points are missing:
1a. Comparison to other exploration methods. A number of methods that use state visitation counts (also referred to as diversity, eg. [A,B]), or prediction error (also referred to as curiosity, eg [C]) have been proposed in recent years. It is important to place the contributions in this paper in context of these other works. A number of these references are missing and no experimental comparison to these methods has been made.
1b. Comparison between inter and intra life curiosity. One of the central motivation is the utility of intra-life curiosity vs inter-life curiosity, yet no comparisons to this effect have been provided.
2. Additionally, the paper employs a custom way of computing coverage (or diversity). It is in terms of location of agent on the screen, as opposed to featurization of the full game screen as used in prior works. It is possible that a large part of the gain comes from the clever design of the space for computing intrinsic exploration reward. The paper tries to control for it, however that description is rather short and vague (not clear how the proposed reward is computed without there being a grid, or how is the grid useful without the intrinsic reward). More details should be provided, and when comparisons to past works or inter-life curiosity are made, this should be controlled for. The two ideas (use of grids, and intra-life curiosity vs inter-life curiosity) should be independently investigated and put in context of past work.
3. I will encourage investigation on a more varied set of tasks. Perhaps, also using some MuJoCo environments, or 3D navigation environments. Table 1 tries to provide some comparisons on Atari, however number of samples is different for different methods making the comparisons invalid. Additionally, all of these are still on Atari.
[A] Diversity is All You Need: Learning Skills without a Reward Function Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, Sergey Levine
[B] EX2: Exploration with Exemplar Models for Deep Reinforcement Learning Justin Fu, John D. Co-Reyes, Sergey Levine
[C] Curiosity-driven Exploration by Self-supervised Prediction Deepak Pathak, Pulkit Agrawal, Alexei A. Efros and Trevor Darrell International Conference on Machine Learning (ICML), 2017


A KERNEL RANDOM MATRIX-BASED APPROACH FOR SPARSE PCA
In this paper, we present a random matrix approach to recover sparse principal components from n p-dimensional vectors. Specifically, considering the large dimensional setting where n, p → ∞ with p/n → c ∈ (0, ∞) and under Gaussian vector observations, we study kernel random matrices of the type f (Ĉ), where f is a three-times continuously differentiable function applied entry-wise to the sample covariance matrixĈ of the data. Then, assuming that the principal components are sparse, we show that taking f in such a way that f (0) = f (0) = 0 allows for powerful recovery of the principal components, thereby generalizing previous ideas involving more specific f functions such as the soft-thresholding function.

The paper aims at generalizing a result by Deshpande and Montanari.
Deshpande and Montanari prove a result for the covariance thresholding algorithm which consists of (1) removing noise from an empirical covariance matrix of a (single) spiked sparse PCA model using soft thresholding, then (2) computing the leading eigenvector of the denoised matrix and finally (3) picking the leading coordinates of the leading eigenvector.
The current paper focuses on step (1) in the above mentioned process. They claim to generalize the use of soft thesholding to more general functions applied element-wise to the empirical covariance matrix.
There is a questionable term in the paper's title: the word "kernel" is mistakenly used. In fact a symmetric matrix is the Gram matrix of a kernel if it is positive semi-definite. The empirical covariance is PSD, but when you apply a function to it elementwise it has no guarantee of conserving the PSD property.
Regarding the motivation of the paper (1) the paper claims to study the rank K case with arbitrary K>=1. Deshpande and Montanari studied K=1 and the analysis is already an interesting result. Generalizing it to higher ranks poses many questions, including the sparse vector supports' overlaps.
It is not clear to me why authors insist on trying to generalize the result to a function f that has broader properties. The main motivation is to recover the support of the (leading) sparse eigenvector / PC. It should not be to try denoising the empirical covariance with a complicated function f. If the focus is on generalizing the soft-thresholding part of the approach, then the real question can be formulated as what is the optimal f given that we have this or that property in the data? This often leads to Bayesian analysis of the problem.
The numerical experiments do not show any substantial improvement obtained using the prescribed method over using the baseline method (covariance thresholding). I suspect that authors can emphasis the benefit of their method by picking f to hold certain properties that reflect the noise process and beat covariance thresholding in those regimes.
Figure 1 right hand side. I do not see why authors refer to phase transition. I don't see a phase transition happening there.


CRITICAL LEARNING PERIODS IN DEEP NETWORKS
Similar to humans and animals, deep artificial neural networks exhibit critical periods during which a temporary stimulus deficit can impair the development of a skill. The extent of the impairment depends on the onset and length of the deficit window, as in animal models, and on the size of the neural network. Deficits that do not affect low-level statistics, such as vertical flipping of the images, have no lasting effect on performance and can be overcome with further training. To better understand this phenomenon, we use the Fisher Information of the weights to measure the effective connectivity between layers of a network during training. Counterintuitively, information rises rapidly in the early phases of training, and then decreases, preventing redistribution of information resources in a phenomenon we refer to as a loss of "Information Plasticity". Our analysis suggests that the first few epochs are critical for the creation of strong connections that are optimal relative to the input data distribution. Once such strong connections are created, they do not appear to change during additional training. These findings suggest that the initial learning transient, under-scrutinized compared to asymptotic behavior, plays a key role in determining the outcome of the training process. Our findings, combined with recent theoretical results in the literature, also suggest that forgetting (decrease of information in the weights) is critical to achieving invariance and disentanglement in representation learning. Finally, critical periods are not restricted to biological systems, but can emerge naturally in learning systems, whether biological or artificial, due to fundamental constrains arising from learning dynamics and information processing.

The authors analyze the learning dynamics in deep neural networks and identify an intriguing phenomenon that reflects what in biological learning is known as critical period: a relatively short time window early in post-natal development where organisms become particularly sensitive to particular changes in experience. The importance of critical periods in biology is due to the fact that specific types of perturbations to the input statistic can cause deficits in performance which can be permanent in the sense that later training cannot rescue them.
The authors did a great job illustrating the parallelism between critical periods in biological neural systems and the analogous phenomenon in artificial deep neural networks. Essentially, they showed that blurring the input samples of the cifar10 dataset during the initial phase of training had an effect that is very reminiscent of the result of sensory deprivation during the critical periods of visual learning in mammals, resulting in a long-term impairments in visual object recognition that persists even if blurring is removed later in training. The authors go as far as characterizing the effects of the length of the "sensory deprivation" window and its onset during training, and comparing the results to classic neuroscience monocular deprivation experiments in kittens, pointing out very striking phenomenological similarities.
Next, the authors establish a connection between critical periods in deep neural networks and the amount of information that the weights of the trained model contain about the task by looking at the Fisher Information Matrix (FIM). With this method they obtain a host of interesting insights. One insight is that there are two phases in learning: an initial one where the trace of the FIM grows together with a rapid increase in classification accuracy, and a second one where accuracy keeps slightly increasing, but Fisher Information trace globally decreases. They then go into detail and look at how this quantity evolves within individual layers of the deep learning architecture, revealing that the deficit caused by the blurring perturbation during the early epochs training is accompanied by larger FIM trace in the last layers of the architecture at the expense of the intermediate layers.
Besides the fact that deep neural network exhibit critical periods, another important result of this work is the demonstration that pretraining, if done inappropriately can actually be deleterious to the performance of the network.
This paper is insightful, and interesting. The conceptual and experimental part of the paper is very clearly presented, and the methodology is very appropriate to tease apart some of the mechanisms underlying the basic phenomenological observations. Here are some detailed questions meant to elucidate some points that are still unclear.
- Presumably, early training on blurred images prevents the initial conv filters from learning to discriminate high-frequency components (first of all, is this true?). The crucial phenomenon pointed out by the authors is that, even after removing the blur, the lower convolutions aren't able to recover and learn the high-frequency components. In fact, the high FIM trace in the latest layers could be due to the fact that they're trying to compensate for the lack of appropriate low-level feature extractors by composing low-frequency filters so as "build" high-frequency ones. If this makes sense, one would assume that freezing the last layers and only maintaining plasticity in the lower ones could be a way of "reopening" the critical period. Is that indeed the case?
- The authors show that their main results are robust to changes in the learning rate annealing schedule. However, it is not clear how changing the optimizer might affect the presence of the critical period. What would happen for instance using Adam or another optimization procedure that relies on the normalization of the gradient?
- On a related note, the authors point out the importance of forgetting, in particular as the main mechanism behind the second learning phase. They also point out that the deficit in learning the task after sensory deprivation is accompanied by large FIM trace in the last layers. What would happen in the presence of a standard regularizer like weight decay? Assuming that large FIM trace in the last layers is correlated with large weighs, that might mitigate the negative effect of early sensory deprivation.
- In neuroscience the opening of the critical period window if thought to be mechanistically mediated by the maturation of inhibition. Is that view compatible with the results presented in this paper? This is sort of complementary to the FIM analysis, since is mostly about net average input to a neuron, i.e. about the information contained in the activations, rather than the weights.


LEARNING TO MAKE ANALOGIES BY CONTRASTING ABSTRACT RELATIONAL STRUCTURE
Analogical reasoning has been a principal focus of various waves of AI research. Analogy is particularly challenging for machines because it requires relational structures to be represented such that they can be flexibly applied across diverse domains of experience. Here, we study how analogical reasoning can be induced in neural networks that learn to perceive and reason about raw visual data. We find that the critical factor for inducing such a capacity is not an elaborate architecture, but rather, careful attention to the choice of data and the manner in which it is presented to the model. The most robust capacity for analogical reasoning is induced when networks learn analogies by contrasting abstract relational structures in their input domains, a training method that uses only the input data to force models to learn about important abstract features. Using this technique we demonstrate capacities for complex, visual and symbolic analogy making and generalisation in even the simplest neural network architectures.

Cons
1.	It’s unclear why LABC produces lower scores than ‘normal’ training on ‘normal’ testing.
2.	The text says nothing I can find to explain why in Fig 5 the ‘entity’ vectors have all 0s except in one dimension, which seems to make the problem considerably easier.
3.	In a sense, there is no cross-domain adaptation required in the symbolic task: min is min, whether it operates on dimension k of the source vectors or dimension j of the target vectors. On the other hand, dimensions are processed independently in the model, as far as I can tell, so there’s no free transfer of learning min on dimension k to knowing min on dimension j. It would be good to comment on this issue.
4.	There seem to be obvious analogies (so to speak) to GANs, and it is very curious that this is not mentioned anywhere that I can see. This is particularly glaring in Sec. 5.3.
5.	The quantitative results are scattered throughout the prose; it would be challenging, but worthwhile, to gather them into an actual table.
Pros
6.	The basic idea (“We should aspire to select as negative examples those examples that are plausible considering the most abstract principles that describe the data”, p. 14) is very intuitive, common-sensical, bordering on obvious. But it is not at all obvious that the idea has as much power as is demonstrated in the experiments. The transfer to novel domain combinations, novel domains, and novel values of dimensions is impressive and surprising.
7.	The result that the proposed training, designed to promote generalization on analogy tasks, also seems to promote improved sensory processing is interesting. Whether it really instantiates the parallel connection argued for by the High-Level Perception view from psychology/philosophy is debatable, but that is itself an interesting connection that the authors should be praised for identifying.
8.	In general, the connection to the cognitive literature is creative and tantalizing and provides good scientific grounding for the work.
9.	The linking to the flexibility of word meanings in the final paragraph pushes the limit of the plausibility of connection to broader cognitive issues, but I’m inclined to indulge the authors for at least bringing up this important and relevant issue.


BUILDING DYNAMIC KNOWLEDGE GRAPHS FROM TEXT USING MACHINE READING COMPREHENSION
We propose a neural machine reading model that constructs dynamic knowledge graphs from procedural text. It builds these graphs recurrently for each step of the described procedure, and uses them to track the evolving states of participant entities. We harness and extend a recently proposed machine reading comprehension (MRC) model to query for entity states, since these states are generally communicated in spans of text and MRC models perform well in extracting entity-centric spans. The explicit, structured, and evolving knowledge graph representations that our model constructs can be used in downstream question answering tasks to improve machine comprehension of text, as we demonstrate empirically. On two comprehension tasks from the recently proposed PROPARA dataset , our model achieves state-of-the-art results. The model also outperforms previous approaches on the RECIPES dataset (Kiddon et al., 2015), which suggests it may apply broadly to procedural text. Finally, we present some evidence that the model's graphical representations help it to impose commonsense constraints on its predictions.

The paper proposes a recurrent knowledge graph (bipartite graph between entities and location nodes) construction & updating mechanism for entity state tracking datasets such as (two) ProPara tasks and Recipes. The model goes through the following three steps: 1) it reads a sentence at each time step t and identifies the location of each entity via machine reading comprehension model such as DrQA (entities are predefined). 2) Co-reference module adjusts relationship scores (soft adjacency matrix) among nodes, including possibly new nodes introduced by the MRC model. 3) to propagate the relational information across all the nodes, the model performs L layers of LSTM for each entity that attend on other nodes via attention (where the weights come from the adjacency matrix). The model repeats the three steps for each sentence. The model is trained by directly supervising for the correct span by the MRC model at each time step, which is possible because the data provides strong supervision for each sentence (not just the answer at the end).
 The model achieves the state of the art in the two tasks of ProPara and Recipes dataset.
Strengths: The paper provides an elegant solution for tracking relationship between entities as time (sentence) progresses. I also agree with the authors that this line of work (dynamic KG construction and modification) is an important area of research. While the model shares a similar spirit to EntNet, I think the model has enough distinctions / contributions, especially given that it outperforms EntNet by a large margin. The model also obtains non-trivial improvement over previous SOTA models.
Weaknesses: Paper could have been written better. I had hard time understanding it. The notations are overall confusing and not explained well. Also there are a few unclear parts which I discuss in questions below.
Questions:
1. Are e_{i,t} and lambda_{i,t} vectors? Scalars? Abstract node notations? It is not clear in the model section. Also, it took me a long time to figure out that ‘i’ is used to index each entity (it is mentioned later).
2. The paper says v_i (initial representation of each entity) is obtained by looking at the contextualized representations (LSTM outputs) of entity mention in the context. What happens if there are multiple mentions in the text? Which one does it look at?
3. For the LSTM in the graph update, why does it have only one input? Shouldn’t it have two inputs, one for previous hidden state and the other for input?
4. Regarding Recipe experiments, the paper says it reaches a better performance than the baseline using just 10k examples out of 60k. This is great, but could you also report the number when the full dataset is used?
5. What does it mean that in training time the model “updates” the location node representation with the encoding of correct span. Do you mean you use the encoding instead?
6. For ProPara task 2, what threshold did you choose to obtain the P/R/F1 score? Is it the threshold that maximizes F1?


PROMP: PROXIMAL META-POLICY SEARCH
Credit assignment in Meta-reinforcement learning (Meta-RL) is still poorly understood. Existing methods either neglect credit assignment to pre-adaptation behavior or implement it naively. This leads to poor sample-efficiency during metatraining as well as ineffective task identification strategies. This paper provides a theoretical analysis of credit assignment in gradient-based Meta-RL. Building on the gained insights we develop a novel meta-learning algorithm that overcomes both the issue of poor credit assignment and previous difficulties in estimating meta-policy gradients. By controlling the statistical distance of both pre-adaptation and adapted policies during meta-policy search, the proposed algorithm endows efficient and stable meta-learning. Our approach leads to superior pre-adaptation policy behavior and consistently outperforms previous Meta-RL algorithms in sample-efficiency, wall-clock time, and asymptotic performance.

In this paper, the authors investigate the gradient calculation in the original MAML (Finn et al. 2017) and E-MAML (Al-Shedivat et al. 2018). By comparing the differences in the gradients of these two algorithms, the authors demonstrate the advantages of the original MAML in taking the casual dependence into account. To obtain the correct estimation of the gradient through auto-differentiation, the authors exploit the DiCE formulation. Considering the variance in the DiCE objective formulation, the authors finally propose an objective which leads to low-variance but biased gradient. The authors verify the proposed methods in meta-RL tasks and achieves comparable performances to MAML and E-MAML.
Although the ultimate algorithm proposed by this paper is not far away from MAML and E-MAML, they did a quite good job in clarify the differences in the existing variants of MAML from the gradient computation perspective and reveal the potential error due to the auto-differentiation. The proposed new objective and the surrogate is well-motivated from such observation and the trade-off between variance and bias.
My major concern is how big the effect is if we use (3) comparing to (4) in calculate the gradient. As the authors showed, the only difference between (3) and (4) is the weights in front of the term \nabla_\theta\log\pi_\theta: the E-MAML is a fixed weight and the MAML is using a adaptive through the inner product. Whether the final difference in Figure 4 between MAML and E-MAML is all caused by such difference in gradient estimation is not clearly. In fact, based on the other large-scale high-dimension empirical experiments in Figure 2, it seems the difference in gradient estimator (3) and (4) does not induced too much difference in final performances between MAML and E-MAML. Based on such observation, I was wondering the consistent better performance of the proposed algorithm might not because the corrected gradient computation from the proposed objective. It might because the clip operation or other components in the algorithm. To make a more convincing argument, it will be better if the authors can evaluate different gradient within the same updates.
I am willing to raise my score if the author can address the question.
minor:
The gradients calculation in Eq (2) and (3) are not consistent with the Algorithm and the appendix.
The notation is not consistent with common usage: \nabla^2 is actually used for denoting the Laplace operator, i.e., \nabla^2 = \nabla \cdot \nable, which is a scalar.


Learning deep networks is computationally hard in the general case. To show any positive theoretical results, one must make assumptions on the data distribution. Current theoretical works often make assumptions that are very far from describing real data, like sampling from Gaussian distribution or linear separability of the data. We describe an algorithm that learns convolutional neural network, assuming the data is sampled from a deep generative model that generates images level by level, where lower resolution images correspond to latent semantic classes. We analyze the convergence rate of our algorithm assuming the data is indeed generated according to this model (as well as additional assumptions). While we do not pretend to claim that the assumptions are realistic for natural images, we do believe that they capture some true properties of real data. Furthermore, we show that on CIFAR-10, the algorithm we analyze achieves results in the same ballpark with vanilla convolutional neural networks that are trained with SGD.

The paper claims to propose a computationally efficient algorithm for training deep CNNs by making assumptions about the distribution of data. The authors argue that (i) they don't make very simplistic assumptions about the data generating distribution as some other papers do, and (ii) their algorithm resembles the actual methods that are used for training deep models and shows some surprising properties of SGD.
Throughout the paper, the authors make a number of assumptions which seem arbitrary at times; not much justifications are provided. The authors claim that their assumptions are not as simplistic as assuming e.g., the inputs are sampled from Gaussian distribution. Unfortunately this is highly unclear: while the "assumptions" themselves are complex, the combination of those assumptions may make the problem solution trivial. While proving a lower bound to address this issue may be hard, at least the authors should try to illuminate more why the solution is not trivial (e.g., why a linear classifier doesn't work, etc.)
Despite the claims, I find the proposed algorithm very far from the usual SGD-based training methods; this is not a problem per se but I don't think that the result illuminates on the effectiveness of SGD (as the authors suggest). The proposed algorithm is a greedy layer-wise method that in each level does a clustering and also trains a "linear" CNN with SGD. So the hardness of end-to-end training of a deep network does not show up. Furthermore, it is not clear for training a linear CNN the SGD is even needed.
I suggest that the authors name each of the assumptions and clearly say which ones are assumed for which result. Here are some of the assumptions that the authors talk about.
1_ The data is generated by the following recursive procedure: First a small "high-level image" is generated from a distribution, G_0. The "pixels" of this high-level image are supposed to encode semantic classes, e.g., sky or ground. In the next step, each of these high-level pixels are turned into a small (lower-level) image. Therefore, we will have a more refined image after the second step. (each semantic class (e.g., sky) has a corresponding distribution that generates the smaller lower-level image (e.g., uniform over 4 possible types of skies)). This procedure continues recursively until we have the final image.
2_ G_0 is "linearly separable".
3_ Semantic classes defined in the model are different enough from each other
4_ {F_c} corresponding to semantic classes are linearly independent
5_ Patch Orthonormality (apparently not assumed everywhere)
it appears that if one assumes all of 1-5, then the problem becomes trivial (linearly separable). The authors then say that we don't want to make assumption 5 for this reason; still, the problem solution may be trivial (authors should at least intuitively justify why it isn't )
Here are some more uses of the word "assumption".
6_ "For simplicity of analysis, we assume only the first layer of the network is trained".
7_ "We assume the algorithm [KMEANS++] returns a mapping [...] such that [...]"
The experiments do not seem conclusive. Only a few experiments have been done. I think the acquired results for CIFAR-10 are below the usual ones using CNNs, and the effects of various hyper-parameters may have interfered.
--
After reading the authors' response, I still think the way that the contributions are depicted (e.g., a justifying the effectiveness of SGD) are inaccurate/unsupported.
Furthermore, although the authors' suggest that they have tested a linear classifier and observed that the data is not linearly separable, more explanations/intuitions are needed about the assumptions that are made throughout the paper.


Scaling end-to-end reinforcement learning to control real robots from vision presents a series of challenges, in particular in terms of sample efficiency. Against end-to-end learning, state representation learning can help learn a compact, efficient and relevant representation of states that speeds up policy learning, reducing the number of samples needed, and that is easier to interpret. We evaluate several state representation learning methods on goal based robotics tasks and propose a new unsupervised model that stacks representations and combines strengths of several of these approaches. This method encodes all the relevant features, performs on par or better than end-to-end learning, and is robust to hyper-parameters change.

This paper aims at comparing end-to-end learning vs separately learning a state representation and subsequently a controller.
While this would be a relevant and important topic, the paper does not currently present consistent evidence to support this hypothesis.
In particular:
- The approach proposed in the approach is not explained in sufficient details. After reading Sec.4 I have only a very vague and high-level idea of how the proposed approach might work. In Figure.2, what is I_t? What is the model that you are training? How are you learning this model? how do you define L_inverse?
- The cited literature about state representation learning is absolutely incomplete. Papers like Lange et al. , Wahlström et al. and Finn et al. and citations herewithin.
- From the experimental results, it is difficult to say anything definitive about the proposed hypothesis. 1) There are multiple end-to-end approaches in the literature, with significant differences in performance. which one are you using? (it seem A2C and PPO, but to which label do they correspond in the tables?) 2) How do you tune the weights of the reward function proposed? This seems an important design choice, but it is not much discussed. 3) In the table reported (e.g., Table 1) it does not seem to me that SRL consistently outperforms other approaches. Even for the arm tasks, Random features seem to outperform the proposed approach (and indeed all the methods except the ground truth). What is going on there?
Overall¸ the paper would benefit from a clearer and more detailed text, and from improved experiments and comparisons.
Minor comments:
- It is unclear to me what "goal-based robotic tasks" means. How do you define a task without a goal?
- An important and missing characteristic of a suitable state representation should be the generalization. In fact, a good representation would ideally allow the agent to generalize to some degree.
- It seems very odd to me that the "action should be implicitly encoded into the state representation" could you elaborate of the motivation for this and the effects?
References:
- Autonomous reinforcement learning on raw visual input data in a real-world application
S Lange, M Riedmiller, A Voigtlander
Neural Networks (IJCNN), The 2012 International Joint Conference on, 1-8
- From pixels to torques: Policy learning with deep dynamical models
N Wahlström, TB Schön, MP Deisenroth
arXiv preprint arXiv:1502.02251
-  Deep Visual Foresight for Planning Robot Motion
Chelsea Finn, Sergey Levine
International Conference on Robotics and Automation (ICRA), 2017


The Boltzmann softmax operator can trade-off well between exploration and exploitation according to current estimation in an exponential weighting scheme, which is a promising way to address the exploration-exploitation dilemma in reinforcement learning. Unfortunately, the Boltzmann softmax operator is not a non-expansion, which may lead to unstable or even divergent learning behavior when used in estimating the value function. The convergence of value iteration is guaranteed in a restricted set of non-expansive operators and how to characterize the effect of such non-expansive operators in value iteration remains an open problem. In this paper, we propose a new technique to analyze the error bound of value iteration with the the Boltzmann softmax operator. We then propose the dynamic Boltzmann softmax(DBS) operator to enable the convergence to the optimal value function in value iteration. We also present convergence rate analysis of the algorithm. Using Q-learning as an application, we show that the DBS operator can be applied in a model-free reinforcement learning algorithm. Finally, we demonstrate the effectiveness of the DBS operator in a toy problem called GridWorld and a suite of Atari games. Experimental results show that outperforms DQN substantially in benchmark games.

Summary: This work demonstrates that, although the Boltzmann softmax operator is not a non-expansion, a proposed dynamic Boltzmann operator (DBS) can be used in conjunction with value iteration and Q-learning to achieve convergence to V* and Q*, respectively. This time-varying operator replaces the traditional max operator. The authors show empirical performance gains of DBS+Q-learning over Q-learning in a gridworld and DBS+DQN over DQN on Atari games.
Novelty: (1) The error bound of value iteration with the Boltzmann softmax operator and convergence & convergence rate results in this setting seem novel. (2) The novelty of the dynamic Boltzmann operator is somewhat thin, as (Singh et al. 2000) show that a dynamic weighting of the Boltzmann operator achieves convergence to the optimal value function in SARSA(0). In that work, the weighting is state-dependent, so the main algorithmic novelty in this paper is removing the dependence on state visitation for the beta parameter by making it solely dependent on time. A question for the authors: How does the proof in this work relate to / differ from the convergence proofs in (Singh et al. 2000)?
Clarity: In the DBS Q-learning algorithm, it is unclear under which policy actions are selected, e.g. using epsilon-greedy/epsilon-Boltzmann versus using the Boltzmann distribution applied to the Q(s, a) values. If the Boltzmann distribution is used then the algorithm that is presented is in fact expected SARSA and not Q-learning. The paper would benefit from making this clear.
Soundness: (1) The proof of Theorem 4 implicitly assumes that all states are visited infinitely often, which is not necessarily true with the given algorithm (if the policy used to select actions is the Boltzmann policy). (2) The proof of Theorem 1 uses the fact that |L(Q) - max(Q)| <= log(|A|) / beta, which is not immediately clear from the result cited in McKay (2003). (3) The paper claims in the introduction that “the non-expansive property is vital to guarantee … the convergence of the learning algorithm.” This is not necessarily the case -- see Bellemare et al., Increasing the Action Gap: New Operators for Reinforcement Learning, 2016.
Quality: (1) I appreciate that the authors evaluated their method on the suite of 49 Atari games. This said, the increase in median performance is relatively small, the delta being about half that of the increase due to double DQN. The improvement in mean score in great part stems from a large improvement occurs on Atlantis.
There are also a number of experimental details that are missing. Is the only change from DQN the change in update rule, while keeping the epsilon-greedy rule? In this case, I find a disconnect between the stated goal (to trade off exploration and exploitation) and the results. Why would we expect the Boltzmann softmax to work better when combined to epsilon-greedy? If not, can you give more details e.g. how beta was annealed over time, etc.?
Finally, can you briefly compare your algorithm to the temperature scheduling method described in Fox et al., Taming the Noise in Reinforcement Learning via Soft Updates, 2016?
Additional Comments:
(1) It would be helpful to have Atari results provided in raw game scores in addition to the human-normalized scores (Figure 5). (2) The human normalized scores listed in Figure 5 for DQN are different than the ones listed in the Double DQN paper (Van Hasselt et al, 2016). (3) For the DBS-DQN algorithm, the authors set beta_t = ct^2 - how is the value of c determined? (4) Text in legends and axes of Figure 1 and Figure 2 plots is very small. (5) Typo: citation for MacKay - Information Theory, Inference and Learning Algorithms - author name listed twice.
Similarly, if the main contribution is DBS, it would be interesting to have a more in-depth empirical analysis of the method -- how does performance (in Atari or otherwise) vary with the temperature schedule, how exploration is affected, etc.?
After reading the other reviews and responses, I still think the paper needs further improvement before it can published.


We propose a new architecture termed Dual Adversarial Transfer Network (DATNet) for addressing low-resource Named Entity Recognition (NER). Specifically, two variants of DATNet, i.e., DATNet-F and DATNet-P, are proposed to explore effective feature fusion between high and low resource. To address the noisy and imbalanced training data, we propose a novel Generalized ResourceAdversarial Discriminator (GRAD). Additionally, adversarial training is adopted to boost model generalization. We examine the effects of different components in DATNet across domains and languages, and show that significant improvement can be obtained especially for low-resource data. Without augmenting any additional hand-crafted features, we achieve new state-of-the-art performances on CoNLL and Twitter NER-88.16% F1 for Spanish, 53.43% F1 for WNUT-2016, and 42.83% F1 for WNUT-2017   1 .

<Summary>
Authors propose the new “DATNet” for the NER task, which extends the base neural model for NER (Bi-LSTM+CRF sequence model with input represented with CharCNN-word embeddings) with the following two main components: (1) GRAD: a language (or resource) adversarial discriminator with adaptive weights that regularize source-target data imbalance, and (2) additional adversarial training approaches that perturb input samples in the embeddings space.
The paper reports big improvement over their baseline approaches without having to rely on other auxiliary or hand-crafted features. The experiment is performed for various low-resource scenarios (varying training data size).
<Comments>
- While the idea of applying “dual adversarial” approaches is new in the context of NER, the technical novelty of each component is limited. GRAD, for example, is rather a minor modification of Language Adversarial Discriminator (Kim et al., 2017) with a scalar weight parameter on loss. The empirical superiority of the proposed method (GRAD) over normal AD approaches cannot be claimed either -- for this authors need to report quantitative results of the (Base + AT + F/P-transfer with AD -- or e.g. \alpha fixed at 0.5 or at some other rate), which I believe is missing in all figures and tables. Visualization of resulting feature distribution (Figure 3) is interesting to look at, but that alone does not suffice. There is no technical novelty in applying adversarial training either, except that it was used in the context of NER.
- Authors use both source and target data to train their “base model” as well (“... we exploit all the source data and target data ...“), e.g. presumably by merging the source and target dataset as well as their embedding matrices, etc. It is important that authors report if there ever is a negative transfer case (e.g. a base model trained with just target data may outperform models trained with source+target data) at varying resource scenarios -- especially at sufficiently-resourced cases.
If by any chance their “base model” refers to in-domain training / in-domain testing results on target, the aforementioned baseline (naive merge of source and target data) is obviously an important baseline to report. I suggest that authors provide these details or clarify. (The confusion comes mainly because some of the SOTA results the authors quote are in-domain training / in-domain evaluation results on respective languages, and some are cross-domain results -- yet they are all under “cross-lingual/domain” columns in Table 2).
- It would be interesting to report the learned optimal \alpha value for each different setting (at varying r or training size) to see if the authors’ intuition is met. On a related note, from the manuscript alone, it is not entirely clear if \alpha is a learnable parameter or a tunable hyper parameter -- by context I believe it is a model parameter. If they are tuned, authors need to report these values.
<Nit>
- Section 3.2.3, “... (GRAD) to enable adaptive weights for [each sample]” → I think it reads better with [each resource] or [each source and target], unless you meant \alpha_i for each sentence.
- Section 3.2.5, “... recently, adversarial samples are [wisely] incorporated” → [widely]
- Fonts for figures could be bigger.


We propose a method for quantifying uncertainty in neural network regression models when the targets are real values on a d-dimensional simplex, such as probabilities. We show that each target can be modeled as a sample from a Dirichlet distribution, where the parameters of the Dirichlet are provided by the output of a neural network, and that the combined model can be trained using the gradient of the conditional likelihood. This approach provides interpretable predictions in the form of multidimensional distributions, rather than point estimates, from which one can obtain confidence intervals or quantify risk in decision making. Furthermore, we show that the same approach can be used to model targets in the form of empirical counts as samples from the Dirichlet-multinomial compound distribution. In experiments, we verify that our approach provides these benefits without harming the performance of the point estimate predictions on two diverse applications: (1) distilling deep convolutional networks trained on CIFAR-100, and (2) predicting the location of particle collisions in the XENON1T Dark Matter detector.

This paper considers parameterizing Dirichlet, Dirichlet-multinomial, and Beta distributions with the outputs of a neural network. They present the distributions and gradients, discuss appropriate activation functions for the output layer, and evaluate this approach on synthetic and real datasets with mixed results. Overall, I found the writing very clear, the main idea sound, and paper generally well executed, but I have serious concerns about the significance of the contributions that lead me to recommend rejection. It would be very useful to me if the authors would provide a concise list of what they consider the main contributions to be and why they are significant. As I see it, the paper does three main things:
1. In section 2, the authors consider parameterizing Dirichlet, Dirichlet-multinomial, and Beta distributions with the outputs of a neural network (Section 2). As the authors note, parameterizing an exponential family distribution with the outputs of a neural network is not a novel contribution (e.g. Rudolph et al. (2016) and David Belanger's PhD thesis (2017)) and though I have never personally seen the Dirichlet, Dirichlet-multinomial, and Beta distributions used, the conceptual leap required is small. Most of section 2 is dedicated to writing down, simplifying, and deriving gradient equations for these three distributions. The simplifications and gradient derivations are well known and appear in many places (e.g. http://jonathan-huang.org/research/dirichlet/dirichlet.pdf, https://arxiv.org/pdf/1405.0099.pdf) and should not be considered contributions in the age of automatic differentiation (see Justin Domke's blog post on autodiff).
2. In section 3, the authors consider the unique challenges of using the proposed networks. They propose targeted activation functions that will improve the stability of learning. I found this to be the most interesting portion of the paper and the most significant contribution. Unfortunately, it is short on details and empirical results are referenced that do not appear in the paper (i.e. the second to last paragraph on page 5). If I were to rewrite this paper, I would focus on answering the question "What are the unique challenges of parameterizing Dirichlet, Dirichlet-multinomial, and Beta distributions with the outputs of a neural network and how can we address them?", replacing section 2 with an expanded section 3.
3. In section 4, the authors evaluate the proposed networks on a collection of synthetic and real tasks. In the end, the results are mixed, with the Dirichlet network performing best on the XENON1T task and the standard softmax network performing best on the CIFAR-100 task. In general, I don't mind mixed results and I appreciate that the authors included both sets of experiments; however, it is important that there is a convincing argument for why one would prefer the proposed solution even when accuracy is the same (e.g. it is faster, it is interpretable, etc.). The authors briefly argue that the proposed methods are superior because they provide uncertainty estimates for the output distributions. This may be true, but they only perform evaluations on tasks where the primary goal is accuracy. If the main benefit of the proposed networks is proper uncertainty quantification, then the evaluations (even if they are qualitative) should reflect that.
In summary, I do not think the models proposed in section 2 are sufficiently novel to justify publication alone which means that the authors need to either: (1) evaluate novel methods that are critical for use of these models or (2) present a convincing evaluation that strongly motivates the proposed model's use or that provides some novel insight into the model's behavior. I think that the authors are on their way to achieving (1), but do not achieve (2). I would suggest finding an application that requires uncertainty estimates for the distribution and centering the paper around that application.
Minor comments:
- Figure 2 (right) should include a y-axis label (e.g. "parameter value").
- In Figure 3 (right), it is not obvious what the "Sigmoid" line corresponds to.
- It is not clear what the authors are trying to show in section 4.1. The EL activation function is smooth and monotone and the likelihood is convex, so there should be no question that the distribution will concentrate around y.
- Section 4.4 was interesting, but would have been more convincing if paired with an evaluation on real data.


While variational dropout approaches have been shown to be effective for network sparsification, they are still suboptimal in the sense that they set the dropout rate for each neuron without consideration of the input data. With such input-independent dropout, each neuron is evolved to be generic across inputs, which makes it difficult to sparsify networks without accuracy loss. To overcome this limitation, we propose adaptive variational dropout whose probabilities are drawn from sparsity-inducing beta-Bernoulli prior. It allows each neuron to be evolved either to be generic or specific for certain inputs, or dropped altogether. Such input-adaptive sparsityinducing dropout allows the resulting network to tolerate larger degree of sparsity without losing its expressive power by removing redundancies among features. We validate our dependent variational beta-Bernoulli dropout on multiple public datasets, on which it obtains significantly more compact networks than baseline methods, with consistent accuracy improvements over the base networks.

This work proposes Variational Beta-Bernoulli Dropout, a Bayesian way to sparsify neural networks by adopting Spike and Slab priors over the parameters of the network. Motivated by the Indian Buffet Process the authors further adopt Beta hyperpriors for the parameters of the Bernoulli distribution and also propose a way to set up the model such that it allows for input specific priors over the Bernoulli distributions. They then provide the necessary details for their variational approximations to the posterior distributions of both such models and experimentally validate their performance on the tasks of MNIST and CIFAR 10/100 classification.
This work is in general well written and conveys the main ideas in an clear manner. Furthermore, parametrising conditional group sparsity in a Bayesian way is also an interesting venue for research that can further facilitate for computational speedups for neural networks. The overall method seems simple to implement and doesn’t introduce too many extra learnable parameters.
Nevertheless, I believe that this paper needs more work in order to be published. More specifically:
- I believe that the authors need to further elaborate and compare with “Generalized Dropout”; the prior imposed on the weights for the non-dependent case is essentially the same with only small differences in the approximate posterior. Both methods seem to optimise, rather than integrate over, the weights of the network and the main difference is in how to handle the approximate distributions over the gates. Why would one prefer one parametrisation rather than the other? Furthermore, the authors of this work argue that they employ asymptotically unbiased gradients for the binary random variables, which is incorrect as the continuous relaxation provides a biased gradient estimator for the underlying discrete model.
- At section 3.2 the authors argue about the inherent sparsity inducing nature of the IBP model. In the finite K scenario this is not entirely the case as sparsity is only encouraged for alpha < K.
- At Eq. 11 the index “n” doesn’t make sense as the Bernoulli probability for each point depends only on the global pi_k. Similarly for Eq. 12.
- Since you tie q(z_nk|pi_k) = p(z_nk|pi_k) then it makes sense to phrase Eq.16 as just D_KL(q(pi) || p(pi)). Furthermore, I believe that you should properly motivate on why tying these two is a sensible thing to do.
- Figure 1 is misleading; you start from a unimodal distribution and then you simply apply a scalar scale and shift to the elements of that distribution. The output of that will always be a unimodal distribution but somehow you end up with a multimodal distribution on the third part of the figure. As a result, I believe that in this case you will not have two clear modes (one at 0 and one at 1) when you apply the hard-sigmoid rectification.
- The motivation for 21 seems a bit confusing to me; what do you mean with insignificant dimensions? What overflow does the epsilon prevent? If the input to the hard sigmoid is a N(0, 1) distribution then you will approximately have 1/3 of the activations having probability close to 1. Furthermore, it seems that you want beta to be small / negative to get sparse outcomes but the text implies that you want it to be large.
- It would be better to rewrite eq. 22 to include also the fact that you have a separate z per layer as currently it seems that the there is only one z. Furthermore, you have written that the variational posterior distribution depends on x_n on the RHS but not on the LHS.
- Above eq. 23 seems that it should be q(z_nk| pi_k, xn) = p(z_nk| pi_k, xn) rather than q(z_nk| pi_k) = p(z_nk| pi_k, xn)
Regarding the experiments; the MNIST results are not particularly convincing as the numbers are, in general, similar to other methods. Furthermore, Figure 2 is a bit small and confusing to read. Should FLOPS be on the y-axis or something else? Almost zero flops for the original model doesn’t seem right. Finally, at the CIFAR 10/100 experiment it seems that both BB and DBB achieve the best performance. However, it seems that the accuracy /sparsity obtained for the baselines is inferior to the results obtained on each of the respective papers. For example, SBP managed to get a 2.71x speedup with the VGG on CIFAR 10 and an error of 7.5%, whereas here the error was 8.68% with just 1.34x speedup. The extra visualisations provided at Figure 3 do look interesting though as it shows what the sparsity patterns learn.


GENERATING LIQUID SIMULATIONS WITH DEFORMATION-AWARE NEURAL NETWORKS
We propose a novel approach for deformation-aware neural networks that learn the weighting and synthesis of dense volumetric deformation fields. Our method specifically targets the space-time representation of physical surfaces from liquid simulations. Liquids exhibit highly complex, non-linear behavior under changing simulation conditions such as different initial conditions. Our algorithm captures these complex phenomena in two stages: a first neural network computes a weighting function for a set of pre-computed deformations, while a second network directly generates a deformation field for refining the surface. Key for successful training runs in this setting is a suitable loss function that encodes the effect of the deformations, and a robust calculation of the corresponding gradients. To demonstrate the effectiveness of our approach, we showcase our method with several complex examples of flowing liquids with topology changes. Our representation makes it possible to rapidly generate the desired implicit surfaces. We have implemented a mobile application to demonstrate that real-time interactions with complex liquid effects are possible with our approach.

This is an application paper on dense volumetric synthesis of liquids and smoke. Given densely registered 4D implicit surfaces (volumes over time) for a structured scene, a neural-network based model is used to interpolate simulations for novel scene conditions (e.g. position and size of dropped water ball). The interpolation model composes two components -- given these conditions, it first regresses weights combining a set of precomputed deformation fields, and then a second model regresses dense volumetric deformation corrections -- these are helpful as some events are not easily modeled with a set of basis deformations.
I found the paper hard to read at first, since the paper is heavy on terminology, only really understood what is going on when I went through the examples in the appendix, which are helpful and then on a second read the content was clear and appears technically correct. I would advise considering defining in more detail early the problem setup (e.g. Fig 13 was helpful), explain some of the variables in context.
This is primarily an application paper on simulating liquids in controlled scenes using nets and appears novel in that narrow domain. The specific way deformations are composed -- using v_inv to backwards correct basis deformations, following up the mixing of those with a correction model -- is intuitive and is also something I see for the first time.
The experimental results are sufficient for simulating liquids/smoke, except I would like to also see a comparison to using deformation field network only, without its predecessor. This was done for Fig 6, but would be nice to also see it numerically in ablation in Fig. 4. Another useful experiment would be to vary the number of bases and/or the resolution of the deformation correction network and see the effects.
More importantly, it would be very helpful is to try this approach for modeling deforming object and body shapes for which there are many datasets (e.g. Shapenet). Right now the implicit surface deformation model is only tested on liquids examples, which limits the impact to that specialist domain -- it's a bit more of a SIGGRAPH type of paper than ICLR.
---- Post author feedback comment ----
I raised my rating to 7 as the paper itself is solid, main concern as another reviewer points out is it may be a bit too specialist for ICLR. If the AC decides to reject based on this fact I am ok with that as well.
I think it would be helpful to add more ablation (deformation-only results for all cases) and experiments with different numbers of bases in the final version. If that's added it will strengthen the paper.


CODE2SEQ: GENERATING SEQUENCES FROM STRUCTURED REPRESENTATIONS OF CODE
The ability to generate natural language sequences from source code snippets has a variety of applications such as code summarization, documentation, and retrieval. Sequence-to-sequence (seq2seq) models, adopted from neural machine translation (NMT), have achieved state-of-the-art performance on these tasks by treating source code as a sequence of tokens. We present CODE2SEQ: an alternative approach that leverages the syntactic structure of programming languages to better encode source code. Our model represents a code snippet as the set of compositional paths in its abstract syntax tree (AST) and uses attention to select the relevant paths while decoding. We demonstrate the effectiveness of our approach for two tasks, two programming languages, and four datasets of up to 16M examples. Our model significantly outperforms previous models that were specifically designed for programming languages, as well as state-of-the-art NMT models. An online demo of our model is available at http://code2seq.org. Our code, data and trained models are available at

This paper introduces an AST-based encoding for programming code and
shows the effectivness of the encoding in two different task of code
summarization:
1. Extreme code summarization - predicting (generating) function name from function body (Java)
2. Code captioning - generating a natural language sentence for a (short) snippet of code (C#)
Pros:
- Simple idea of encoding syntactic structure of the program through random paths in ASTs
- Thorough evaluation of the technique on multiple datasets and using multiple baselines
- Better results than previously published baselines
- Two new datasets (based on Java code present in github) that will be made available
- The encoding is used in two different tasks which also involve two different languages
Cons:
- Some of the details of the implementation/design are not clear (see some clarifying questions below)
- More stats on the collected datasets would have been nice
- Personally, I'm not convinced "extreme code summarization"
is a great task for code understanding (see more comments below)
Overall, I enjoyed reading this paper and I think the authors did a
great job explaining the technique, comparing it with other baselines,
building new datasets, etc.
I have several clarifying questions/points (in no particular order):
* Can you provide some intuition on why random paths in the AST encode
  the "meaning" of the code? And perhaps qualitatively compare it with
  recording some other properties from the tree that preserve its
  structure more?
* When you perform the encoding of the function body, one sample in a
  training step contains all the k (k = 200) paths and all the 2*k
  terminals (end of Section 2)? Or one path at a time (Section 3.2)?
  I'm guessing is the latter, but not entirely sure. Figure 3 could
  improve to make it clear.
* Can you explain how you came up with k = 200? I think providing some
  stats on the dataset could be helpful to understand this number.
* The results for the baselines - do you train across all projects?
  (As you point out, ConvAttention trained separately, curious whether
  it makes a difference for the 2 datasets med and large not present
  in the original paper).
* I'm not sure I understand parts of the ablation study. In particular
  for point 1., it seems that instead of the AST, only the terminal
  nodes are used. Do you still use 200 random pairs of terminal? Is
  this equivalent to a (randomly shuffled) subset of the tokens in the
  program? Also could you explain why you do the ablation study on the
  validation set of the medium dataset? In fact, the caption of Table
  3 says it's done on the dev set. This part was a bit confusing.
* I would have liked to see more details on the datasets introduced,
  in particular wrt metrics that are relevant for training the model
  you describe (e.g., stats on the ASTs, stats on the number of random
  paths in ASTs, code length in tokens, etc.)
* I'm not convinced that the task of "extreme code summarization" is a
  meaningful task. My main problem with it is that the performance of
  a human on this task would not be that great. On one hand humans
  (I'm referring to "programming humans" :) ) have no problem in
  coming up with a name for a function body; however, I'm not
  convinced they could predict the "gold" standard. Or, another way of
  thinking about this, if you have 3 humans who provided names for the
  same function, my guess it that there will be a wide degree of
  (dis)agreement. Some of the examples provided in the supplementary
  material can serve as confirmation bias to my thought :): Fig 7. I
  claim "choose random prime" and "generate prime number" are
  semantically close, however, the precision and recall for this
  example are both low. All this being said, I understand that it's a
  task for which data can be generated fairly quickly to feed the
  (beast) NN and that helps pushing the needle in understanding code
  semantics.
* It would be nice to see "exact match" as one of the metrics (it is
  probably low, judging by F1 scores, but good to be reported).
* Most likely the following paper could be cited in the related work:
Neural Code Comprehension: A Learnable Representation of Code Semantics
https://arxiv.org/abs/1806.07336
https://nips.cc/Conferences/2018/Schedule?showEvent=11359
Page 5 first phrase at the top, perhaps zi is a typo and it is
supposed to be z1?
----
Update: after all the discussion, I'm lowering my score a bit while still hoping the paper will get published. I'm satisfied with the results and the improvement of the paper. I still find it a bit surprising that the pairs of literals/leaves in the tree are a good approximation for the program itself (as shown in one of the ablation study).


DISTRIBUTION-INTERPOLATION TRADE OFF IN GENERATIVE MODELS
We investigate the properties of multidimensional probability distributions in the context of latent space prior distributions of implicit generative models. Our work revolves around the phenomena arising while decoding linear interpolations between two random latent vectors -regions of latent space in close proximity to the origin of the space are oversampled, which restricts the usability of linear interpolations as a tool to analyse the latent space. We show that the distribution mismatch can be eliminated completely by a proper choice of the latent probability distribution or using non-linear interpolations. We prove that there is a trade off between the interpolation being linear, and the latent distribution having even the most basic properties required for stable training, such as finite mean. We use the multidimensional Cauchy distribution as an example of the prior distribution, and also provide a general method of creating non-linear interpolations, that is easily applicable to a large family of commonly used latent distributions.

The authors study the problem of when the linear interpolant between two random variables follows the same distribution. This is related to the prior distribution of an implicit generative model. In the paper, the authors show that the Cauchy distribution has such a property, however due to the heavy-tails is not particularly useful. In addition, they propose a non-linear interpolation that naturally has this property.
Technically the paper in my opinion is solid. Also, the paper is ok-written, but I think it needs improvements (see comments).
Comments:
#1) In my opinion the motivation is not very clear and should be improved. In the paper is mentioned that the goal of shortest path interpolation is to get smooth transformations. So, in principle, I am really skeptical when the linear interpolant is utilized as the shortest path. Even then, what is the actual benefit of having the property that the linear interpolants follow the same distribution as the prior? How this is related to smoother transformations? What I understand is that, if we interpolate between several random samples, we will get less samples near the origin, and additionally, these samples will follow the prior? But how this induces smoothness in the overall transformation? I think this should be explained properly in the text i.e. why is it interesting to solve the proposed problem.
#2) From Observation 2.2. we should realize that the distribution matching property holds if the distribution has infinite mean? I think that this is implicitly mentioned in Section 2.2. paragraph 1, but I believe that it should be explicitly stated.
#3) Fig.1 does not show something interesting, and if it does it is not explained. In Fig. 2 I think that interpolations between the same images should be provided such that to have a direct comparison. Also, in Fig. 3 the norm of Z can be shown in order to be clear that the Cauchy distribution has the desired property.
#4) Section 2.2. paragraph 6, first sentence. Here it is stated that the distribution "must be trivial or heavy-tailed". This refers only to the Cauchy distribution? Since earlier the condition was the infinite mean. How these are related? Needs clarification in the text.
#4) In Figure 4, I believe that the norms of the interpolants should be presented as well, such that to show if the desired property is true. Also in Figure 5, what we should see? What are the improvements when using the proposed non-linear interpolation?
Minor comments:
#1) Section 1.2. paragraph 2. For each trained model the latent space usually has different structure e.g. different untrained regions. So I believe that interpolations is not the proper way to compare different models.
#2) Section 1.3 paragraph 1, in my opinion the term "pathological" should be explained precisely here. So it makes clear to the reader what he should expect.
#3) Section 2.2. paragraph 2. The coordinate-wise implies that some Z_i are near zero and some others significantly larger?
In generally, I like the presented analysis. However, I do not fully understand the motivation. I think that choosing the shortest path guarantees smooth transformations. I do not see why the distribution matching property provides smoother transformations. To my understanding, this is simply a way to generate less samples near the origin, but this does not directly means smoother transformations of the generated images. I believe that the motivation and the actual implications of the discussed property have to be explained better.


DIVERSITY-SENSITIVE CONDITIONAL GENERATIVE ADVERSARIAL NETWORKS
We propose a simple yet highly effective method that addresses the mode-collapse problem in the Conditional Generative Adversarial Network (cGAN). Although conditional distributions are multi-modal (i.e., having many modes) in practice, most cGAN approaches tend to learn an overly simplified distribution where an input is always mapped to a single output regardless of variations in latent code. To address such issue, we propose to explicitly regularize the generator to produce diverse outputs depending on latent codes. The proposed regularization is simple, general, and can be easily integrated into most conditional GAN objectives. Additionally, explicit regularization on generator allows our method to control a balance between visual quality and diversity. We demonstrate the effectiveness of our method on three conditional generation tasks: image-to-image translation, image inpainting, and future video prediction. We show that simple addition of our regularization to existing models leads to surprisingly diverse generations, substantially outperforming the previous approaches for multi-modal conditional generation specifically designed in each individual task.

The paper proposes a simple way of addressing the issue of mode-collapse by adding a regularisation to force the outputs to be diverse. Specifically, a loss is added that maximises the l2 loss between the images generated, normalised by the distance between the corresponding latent codes. This method is also used to control the balance between visual quality and diversity.
The paper is overall well written, introducing and referencing well existing concepts, and respected the 8 pages recommendation.
Why was the maximum theta (the bound for numerical stability) incorporated in equation 2? What happens if this is omitted in practice? How is this determined?
In section 4, an increase of the gradient norm of the generator is implied: does this have any effect on the robustness/sensitivity of the model to adversarial attacks?
In section 5, how is the “appropriate CGAN” determined?
My main issue is with the experimental setting that is somewhat lacking. The visual quality of the samples illustrated in the paper is inferior to that observed in the state-of-the-art, begging the question of whether this is a tradeoff necessary to obtain better diversity or if it is a consequence of the additional regularisation.. The diversity observed seems to mainly be attributable to colour differences rather than more elaborate differences. Even quantitatively, the proposed method seems only marginally better than other methods.
Update post rebuttal
-----------------------------
The experimental setting that is a little lacking. Qualitatively and quantitatively, the improvements seem marginal, with no significant improvement shown. I would have liked a better study of the tradeoff between visual quality and diversity, if necessary at all.
However, the authors addressed well the issues. Overall, the idea is interesting and simple and, while the paper could be improved with some more work, it would benefit the ICLR readership in its current form, so I would recommend it as a poster -- I am increasing my score to that effect.