Autonomously discovering temporally extended actions, or skills, is a longstanding goal of hierarchical reinforcement learning. We propose a new algorithm that combines skill chaining with deep neural networks to autonomously discover skills in high-dimensional, continuous domains. The resulting algorithm, deep skill chaining, constructs skills with the property that executing one enables the agent to execute another. We demonstrate that deep skill chaining significantly outperforms both non-hierarchical agents and other state-of-the-art skill discovery techniques in challenging continuous control tasks.

Summary
The authors tackle the problem of skill discovery by skill chaining. In particular, the authors claim two key contributions over the state of the art in option discovery 1) learn initiation sets 2) do not need to specify the number of options and this is also learned. Skill discovery is formalized by skill chaining; wherein the skills are chained backward from a goal state, and in a way, such that termination of an option is the initiation of the option that precedes in its chain.
Interesting and useful aspects of this work are goal-based learning of where options should initiate (although clarification on goals would be crucial), the discussion of the optimality of the discovered solutions, scales up various ideas already proposed in Konidaris & Barto (2009b).
Skill discovery through skill chaining, in particular, has not been extensively explored in a deep RL setting and serves as a useful contribution.
Detailed comments:
The authors introduce DSC with an intuition based on a goal. However, it is never mentioned where this goal comes from. In sec 3, the authors describe an algorithm based on the presumed goal. It is not clear to me: the goal is at one instance defined as \beta_{o_i} := \mathcal{I}_{o_i - 1}, and then in the next few lines, it is said “goal state of the MDP or the initiation condition of a previously learned option”. Is it correct to say this is an algorithm (sec 3) for goal-based option discovery using DSC, where the goal is specified in the MDP?
Before going into the details of the architecture, it would be useful for the reader to have a formalism or a clear algorithm where at least the definition of what a goal constitutes here is clearly stated.
Intra-option policy: It is nice to see that the option’s internal policy is not driven by the task-specific reward and has its internal reward. In the sparse reward setting: how is the subgoal reward chosen? In dense reward: what kind of distance metric is used? Please provide details.
Policy over options: The foremost option constructions seems a bit weird: is there a single goal specified which helps determine the termination condition of the global option? What would happen if there are multiple goals in the MDP?
Initiation set classifier: This is an interesting approach. Although, it raises a natural question: “
We continue adding to our chain of options in this fashion until a learned initiation set classifier contains the start state of the MDP. “ What happens if this never happens, or is this process is guaranteed to converge?
Experiments: Initiation set visuals are nice and interpretable. Experiments are not very convincing: authors mostly demonstrate results on control tasks that are specific to navigation and could do a more rigorous analysis by considering different tasks such as visual domains, robot manipulation tasks.  In particular, authors should compare their method and discuss other skill chaining approaches such as [1,2,3].
[1] Shoeleh, Farzaneh, and Masoud Asadpour. "Graph-based skill acquisition and transfer learning for continuous reinforcement learning domains." Pattern Recognition Letters 87 (2017): 104-116.
[2] Metzen, Jan Hendrik, and Frank Kirchner. "Incremental learning of skill collections based on intrinsic motivation." Frontiers in neurorobotics 7 (2013): 11.
[3] Konidaris, George, et al. "Robot learning from demonstration by constructing skill trees." The International Journal of Robotics Research 31.3 (2012): 360-375.
My primary concern is the amount of engineering that is needed to get this to work. There are multiple steps which do not seem to be sequentially dependent on the success of the previous steps (eg: global option construction needs a goal, the next options are only discovered once the global actions has been constructed, there is an initiation period and internal components of an option are all learned through multiple algorithms DQN, SMDP Q learning, DDPG).
Overall:
+Break the assumption that all options are everywhere.
+Number of options per task is also learned and does not need to be specified a priori.
+Chains skills in a smart way - which could be very useful for lifelong learning, but the approach, as it is, is limited by the goal of the MDP (whatever that means in this context, state or a Reward).
+Options are driven by internal rewards
-The heavy machinery used in the core algorithm seems to be inspired by Konandiais 2009b. It would be very useful to discuss, clarify and contrast what is novel and what is borrowed from Konandiais 2009b as is.


Deep neural networks with learnable activation functions have shown superior performance over deep neural networks with fixed activation functions for many different problems. The adaptability of learnable activation functions adds expressive power to the model which results in better performance. Here, we propose a new learnable activation function based on Adaptive Piecewise Linear units (APL), which 1) gives equal expressive power to both the positive and negative halves on the input space and 2) is able to approximate any zero-centered continuous non-linearity in a closed interval. We investigate how the shape of the Symmetric-APL function changes during training and perform ablation studies to gain insight into the reason behind these changes. We hypothesize that these activation functions go through two distinct stages: 1) adding gradient information and 2) adding expressive power. Finally, we show that the use of Symmetric-APL activations can significantly increase the robustness of deep neural networks to adversarial attacks. Our experiments on both black-box and open-box adversarial attacks show that commonly-used architectures, namely Lenet, Network-in-Network, and ResNet-18 can be up to 51% more resistant to adversarial fooling by only using the proposed activation functions instead of ReLUs.

This paper proposes a learnable piece-wise linear activation unit whose hinges are placed symmetrically. It gives a proof on the universality of the proposed unit on a certain condition. The superiority of the method is empirically shown. The change of the activation during training is analyzed and insight on the behavior is provided. The robustness to adversarial attacks is also empirically examined.
This paper discusses a very basic component of neural network models: activation function. Thus, it should be of interest to many researchers. The proposed method is simple and seems easy to use in real settings. A number of experiments are conducted to validate the method and the results look promising. The experiments in Section 5 is particularly interesting. It might give some hints for the following studies.
However, there are several things to be addressed for acceptance.
1) What is actually proposed is not very clear.
S-APL is formulated in Equation 2. However, there are some discussion after that which changes or restricts the equation. For example, it seems that b_i^s^+ = b_i^s^- is assumed throughout the paper. In that case, it should be just reflected in Equation 2. In the third paragraph of Section 3.2, it is mentioned that h_i^s(x) = h_i^s(-x) with b_i^s^+ = b_i^s^-. However, it should also assume that a^s^+ = a^s^-. From the experiments. apparently, a^s^+ = a^s^- is not assumed. It seems that the method has symmetry only for the hinge locations.
In the first paragraph of Section 3.2, it is implied that parameters are shared across layers. It is not very clear what is shared and what is not. Please make that part clear. It will make it easier to understand the experimental settings, too.
2) Theorem 3.1 does not seem to prove the approximation ability of S-APL.
It is clear that g(x, S) can represent arbitrary h(x, S), but I am not sure if it is clear that h(x, S) can represent arbitrary g(x, S). It should also depend on the conditions on a^s^+, a^s^-, b_i^s^+, b_i^s^-. I think it needs to prove that h(x, S) can approximate arbitrary piecewise linear function (i.e., g(x, S)) if you want to prove the approximation ability of h(x, S).
Equation 4 seems to assume that all intervals are the same (i.e., ∀i, B_i - A_i = (B-A) / S). It should be stated explicitly. This relates to the problem 1).
I may not understand some important aspect. I am happy to be corrected.
3) Experimental conditions are not clear.
Please cite the papers which describe the architecture of the models used in the experiments. The effectiveness of the proposed method should depend on the network architecture and it is importable to be able to see the details of the models.
4) On the sensitivity of optimization on the initial value.
It is interesting to see that "fixed trained S-APL" is not comparable with "S-APL positive". If the hypothesis in the paper is correct, it is natural to assume that "fixed trained S-APL" also has some issue on training. It would be interesting to see experimental results with "initialized with trained-S-APL" and "S-APL positive with non-zero initial value".  It is a bit weird to observe that "S-APL positive" never becomes non-zero for x < 0.
5) Comparison results with other activation units in Section 6.
The proposed method is compared only with ReLU. It is important to see comparisons with other activations such as the plain APL.
Some other minor comments:
It is quite interesting that objects are actually modified for adversarial attack for the proposed method in Figure 5. It would be interesting to have some consideration on it.


Neural Architecture Search has shown potential to automate the design of neural networks. Deep Reinforcement Learning based agents can learn complex architectural patterns, as well as explore a vast and compositional search space. On the other hand, evolutionary algorithms offer higher sample efficiency, which is critical for such a resource intensive application. In order to capture the best of both worlds, we propose a class of Evolutionary-Neural hybrid agents (Evo-NAS). We show that the Evo-NAS agent outperforms both neural and evolutionary agents when applied to architecture search for a suite of text and image classification benchmarks. On a high-complexity architecture search space for image classification, the Evo-NAS agent surpasses the accuracy achieved by commonly used agents with only 1/3 of the search cost.

In this paper, the authors proposed to combine both NN-based NAS and Aging EVO to get the benefit of both world: good global and local sample efficiency. The main idea is to use Aging EVO algorithm to guide the overall search process, and a NN predicting the final performance is used to guide the mutation process. The combined EVO-NAS has showed consistent good performance over a range of tasks.
Overall, while the novelty of the paper is not exceptional, since it is a rather straightforward combination of two existing approaches, the end-results is promising. I would like to see more in-depth analysis on the combined algorithm to validate authors' hypothesis on why EVO-NAS works better. More detailed comments can be found below.
1. The experiment on the synthetic task is not very helpful, since the domain can be far apart from the real NAS applications. One evidence is that the NN significantly outperform EVO in this task but not in the other tasks.
2. In difficult tasks, the proposed EVO-NAS and the original Aging EVO are very close in the first few hundreds of trials, however, later the gap remains the same. It would be interesting to see if we can eliminate the gap by adding the NN component in the middle of the Aging EVO experiment (e.g., at the point of 2000 trials).
3. I am also curious about the difference between the NN learned from Neural agent vs. those learned from EVO-NAS agent. Moreover, do we need a NN for the EVO-NAS? Or something simpler would be sufficient to guide the search.


Reduced precision computation is one of the key areas addressing the widening 'compute gap', driven by an exponential growth in deep learning applications. In recent years, deep neural network training has largely migrated to 16-bit precision, with significant gains in performance and energy efficiency. However, attempts to train DNNs at 8-bit precision have met with significant challenges, because of the higher precision and dynamic range requirements of back-propagation. In this paper, we propose a method to train deep neural networks using 8-bit floating point representation for weights, activations, errors, and gradients. We demonstrate state-of-the-art accuracy across multiple data sets (imagenet-1K, WMT16) and a broader set of workloads (Resnet-18/34/50, GNMT, and Transformer) than previously reported. We propose an enhanced loss scaling method to augment the reduced subnormal range of 8-bit floating point, to improve error propagation. We also examine the impact of quantization noise on generalization, and propose a stochastic rounding technique to address gradient noise. As a result of applying all these techniques, our models generalize as well as the full precision models, and achieve baseline accuracy.

This paper is about training deep models with 8-bit floating point numbers. The authors use an enhanced loss scaling method and stochastic rounding method to stabilize training. They do experiments on image classification and NLP tasks.
The paper is clearly written. However, I don’t think this paper passes the bar of ICLR. This paper lacks innovation and insightful analysis.
1.Sec. 3.1 proposes enhanced loss scaling. Loss scaling is a heuristic to train low-precision neural networks. The authors train 8-bit GNMT with a changing scaling factor. However, this looks like some manually tuned result for GNMT only. I doubt if this generalizes to other models. Besides, there is no equation or algorithm flowchart to demonstrate their method. It’s not very readable.
2.The logic of Sec. 3.2 is quite confusing. The authors first empirically show that the performance of ResNet-50 significantly drops with 8-bit training. Then they show the sum of the square of the weights in ResNet-50 is high at the beginning. With this observation, they claim it demonstrates the drawback of ‘rounding-to-nearest-even’. I cannot see the connection between the norm of weights and the rounding technique. Moreover, the stochastic rounding has already been used in 8-bit training.[1]
3.The setting in the experiment section is not stated clearly. For example, what’s the hyper-parameter for loss scaling? Another question is the gradient. In Sec. 3, just above Fig. 1, the authors claim the weight update is performed in full-precision. In contrast, they claim the gradient is 8-bit in table 3. If the update is full-precision, [2] is an important baseline.
Small suggestions:
1.For Fig. 6, I suggest the authors to smooth the loss curves to avoid overlap of two curves.
2.There are two ‘with’s in the last paragraph of page 7.
Reference:
[1]Wang N, Choi J, Brand D, et al. Training deep neural networks with 8-bit floating point numbers[C]//Advances in neural information processing systems. 2018: 7675-7684.
[2]Banner R, Hubara I, Hoffer E, et al. Scalable methods for 8-bit training of neural networks[C]//Advances in Neural Information Processing Systems. 2018: 5145-5153.


We provide a theoretical explanation for the effectiveness of gradient clipping in training deep neural networks. The key ingredient is a new smoothness condition derived from practical neural network training examples. We observe that gradient smoothness, a concept central to the analysis of first-order optimization algorithms that is often assumed to be a constant, demonstrates significant variability along the training trajectory of deep neural networks. Further, this smoothness positively correlates with the gradient norm, and contrary to standard assumptions in the literature, it can grow with the norm of the gradient. These empirical observations limit the applicability of existing theoretical analyses of algorithms that rely on a fixed bound on smoothness. These observations motivate us to introduce a novel relaxation of gradient smoothness that is weaker than the commonly used Lipschitz smoothness assumption. Under the new condition, we prove that two popular methods, namely, gradient clipping and normalized gradient, converge arbitrarily faster than gradient descent with fixed stepsize. We further explain why such adaptively scaled gradient methods can accelerate empirical convergence and verify our results empirically in popular neural network training settings.

In this paper, the authors relax the generally used Lipschitz smoothness condition in optimization, to a more general smoothness condition that may depend on norm of the gradient. The authors proved that, with this relaxed condition, under such cases, both GD and clipped GD can converge within O(1/\epsilon^2) time, but when their exists x where the gradient norm can be very large, the clipped GD will converge provably faster than GD with a proved GD convergence rate lower bound. The authors also generalize their results to SGD. Experiments show that in deep neural network local gradient Lipschitz constant is scale with gradient norm, and use clipped gradient can accelerate convergence while keep good generalization performance as expected, which is well observed by other researchers.
Detailed comments:
1. The f^* In theorem 3 is not defined. I think it’s not the f^* in Assumption 1? Should be more clear. Also, in non-convex optimization, can the deterministic GD find the global optimum? If f^* is the stationary point the algorithm find, then the Assumption 1 is a little weird. The authors should better clarify this.
2. Feel the claim of the Assumption can be relaxed to only hold on \mathcal{S} is a little confusing. Which Assumption can be relaxed? It seems that in the proof of Lemma 9, we need the Assumption 3 holds for the set \|x^+-x\| \leq min{1/L_1, 1}. I think maybe the other Assumption will also be needed globally, unless the authors can prove that the optimization is only on a compact set in S.
3. Maybe a discussion connected the existing Lipschitz constant based results to the new results in this paper can make the readers more aware of the contribution in this paper. For example, the Lipschitz constant of neural network can be of order O(L_1 M) due to the definition, so if L_1 M is large, for clipped GD we can still have expected decreasing of O(1/L_0) each turn while vanilla GD with the best O(1/L) step size can still only have O(1/L) where L = L_0 + L_1 M due to the traditional results based on Lipschitz smoothness.
I don’t totally go through the whole proof, but I think the results are reasonable and the most of the techniques are standard to the whole community. So there may be no fatal error that violates the conclusion. I find the whole paper interesting and match the intuition from the practice. However, I think it can be polished to make the whole idea more clear and more easy to accept by the potential theorists and practitioner audience.


Reinforcement learning agents can include different components, such as policies, value functions, state representations, and environment models. Any or all of these can be the loci of knowledge, i.e., structures where knowledge, whether given or learned, can be deposited and reused. Regardless of its composition, the objective of an agent is behave so as to maximise the sum of a suitable scalar function of state: the reward. As far as the learning algorithm is concerned, these rewards are typically given and immutable. In this paper we instead consider the proposition that the reward function itself may be a good locus of knowledge. This is consistent with a common use, in the literature, of hand-designed intrinsic rewards to improve the learning dynamics of an agent. We adopt the multi-lifetime setting of the Optimal Rewards Framework, and propose to meta-learn an intrinsic reward function from experience that allows agents to maximise their extrinsic rewards accumulated until the end of their lifetimes. Rewards as a locus of knowledge provide guidance on "what" the agent should strive to do rather than on knowledge of "how" the agent should behave that is more directly captured in policies or value functions for example. Thus, our focus here is on demonstrating the following: (1) that it is feasible to meta-learn good reward functions, (2) to show that the learned reward functions can capture interesting kinds of "what" knowledge, and (3) that because of the indirectness of this form of knowledge the learned reward functions can generalise to other kinds of agents and to changes in the dynamics of the environment.
Reinforcement learning agents can store knowledge in their policies, value functions, state representations, and models of the environment dynamics. These components can be the loci of knowledge in the sense that they are structures in which knowledge, either learned from experience by the agent's algorithm or given by the agent-designer, can be deposited and reused. The objective of the agent is defined by a reward function, and the goal is to learn to act so as to optimise cumulative rewards. In this paper we consider the proposition that the reward function itself is a good locus of knowledge. This is unusual in that most prior work treats the reward as given and immutable, at least as far as the learning algorithm is concerned. At the same time, especially in challenging reinforcement-learning problems, agent designers do find it convenient to modify the reward function given to the agent to facilitate learning. It is therefore useful to distinguish between two kinds of reward functions : extrinsic rewards define the task and capture the designer's preferences over agent behaviour, whereas intrinsic rewards serve as helpful signals to improve the learning dynamics of the agent. Intrinsic rewards are typically hand-designed and then often added to the immutable extrinsic rewards to form the reward optimised by the agent.
Most existing work on intrinsic rewards falls into two broad categories: task-dependent and taskindependent. Both are typically designed by hand. Hand-designing task-dependent rewards can be fraught with difficulty as even minor misalignment between the actual reward and the intended bias can lead to unintended and sometimes catastrophic consequences (Clark & Amodei, 2016). Task-independent intrinsic rewards are also typically hand-designed, often based on an intuitive understanding of animal/human behaviour or on heuristics on desired exploratory behaviour. It can, however, be hard to match such task-independent intrinsic rewards to the specific learning dynamics induced by the interaction between agent and environment. The motivation for this paper is our interest in the comparatively under-explored possibility of learned (not hand-designed) taskdependent intrinsic rewards (see Zheng et al., 2018, for previous work).
We emphasise that it is not our objective to show that rewards are a better locus of learned knowledge than others; the best locus likely depends on the kind of knowledge that is most useful in a given
task. In particular, knowledge captured in rewards provides guidance on "what" the agent should strive to do while knowledge captured in policies provides guidance on "how" an agent should behave. Knowledge about "what" captured in rewards is indirect and thus slower to make an impact on behaviour because it takes effect through learning, while knowledge about "how" can directly have an immediate impact on behaviour. At the same time, because of its indirectness the former can generalise better to changes in dynamics and learning architectures. Therefore, instead of comparing different loci of knowledge, the purpose of this paper is to show that it is feasible to capture useful learned knowledge in rewards and to study the kinds of knowledge that can be captured.
How should we measure the usefulness of a learned reward function? Ideally, we would like to measure the effect the learned reward function has on the learning dynamics. Of course, learning happens over multiple episodes, indeed it happens over an entire lifetime. Therefore, we choose lifetime return, the cumulative extrinsic reward obtained by the agent over its entire lifetime, as the main objective. To this end, we adopt the multi-lifetime setting of the Optimal Rewards Framework (Singh et al., 2009) in which an agent is initialised randomly at the start of each lifetime and then faces a stationary or non-stationary task drawn from some distribution. In this setting, the only knowledge that is transferred across lifetimes is the reward instead of the policy. Specifically, the goal is to learn a single intrinsic reward function that, when used to adapt the agent's policy using a standard episodic RL algorithm, ends up optimising the cumulative extrinsic reward over its lifetime.
In previous work, good reward functions were found via exhaustive search, limiting the range of applicability of the framework. Here, we develop a more scalable gradient-based method (Xu et al., 2018c) for learning the intrinsic rewards by exploiting the fact the interaction between the policy update and the reward function is differentiable (Zheng et al., 2018). Since it is infeasible to backpropgate through the full computation graph that spans across the entire lifetime, we truncate the unrolled computation graph of learning updates up to some horizon. However, we handle the long-term credit assignment by using a lifetime value function that estimates the remaining lifetime return, which needs to take into account changing policies. Our main scientific contributions are a sequence of empirical studies on carefully designed environments that show how our learned intrinsic rewards can capture interesting regularities in the interaction between a learning agent and an environment sampled from a distribution, and how the learned intrinsic reward can generalise to changed dynamics and agent architectures. Collectively, our contributions present an effective approach to the discovery of intrinsic rewards that can help an agent optimise the extrinsic rewards collected in a lifetime.

The paper proposes a meta-learning approach to learn reward functions for reinforcement learning agents. It defines an algorithm to optimize an intrinsic reward function for a distribution of tasks in order to maximise the agent’s lifetime rewards. The properties of this reward function and meta-learning algorithm  are investigated through a number of proof-of-concept experiments.
The meta-learning algorithm and the corresponding empirical investigation are the main contributions of the paper. The algorithm seems to be similar to previous meta-learning approaches, but differs by introducing a lifetime value function. While I thought the paper raises some interesting possibilities, I am currently leaning towards rejection. The proposed algorithm does not seem like a major innovation over cited previous work. The empirical evaluation provides a number of proof-of-concept ideas, but no in depth investigation of the properties of the approach. The theoretical properties of the approach are barely discussed.
Detailed remarks:
* The main addition to the meta-learning algorithm is the lifetime value function. The authors mention multiple times that this is crucial to learning, but the properties of this value function are not really investigated or discussed in depth:
- The authors mention that the value function must take into account changing future policies, but do not discuss this further. The value function update seems to be a standard on-policy TD update with the lifetime return and the complete history as input. The policy for this value function, however, is still a standard policy with only state as input (but it will be non-stationary over the agent lifetime). It would be good to discuss this learning problem in more detail.
- The algorithm uses an n-step return. Is this important? What effect does n have on learning?
* Another issue which I would have liked being discussed in more detail is the non-stationarity of the learning problem in general. Most of the approaches discussed in related work (e.g. shaping)  are aimed at learning/designing more informative reward functions. These reward functions still fit in the MDP framework, however, and map from states and actions to rewards. In the case of shaping approaches guarantees can be given that this does not alter the learning problem. The intrinsic reward functions used in this paper map the full life-time history of the agent to rewards. While this is a richer framework that can express more complicated tasks (like exploration over multiple episodes), it also invalidates many of the basic assumptions of reinforcement learning. The rewards are now no longer Markovian when only observing the current state. Moreover, the reward function will change over time. To what extent does this require non-stationary / history-based policy and value function learning to solve these issues? While some of these issues also apply to count based exploration strategies, (Strehl and Littman,2008 ) provided results that the  exploration bonuses result a Bellman Equation that accounts for uncertainties. No real guarantees seem to exist here.
* The empirical contribution focuses on trying to answer a number of questions regarding the properties of the learnt intrinsic rewards. I found these questions to be very broad, while the answers are mostly anecdotal evidence through proof-of-concept examples.  These examples do show potential benefits of meta-learning intrinsic rewards, but I was somewhat disappointed that there was no more systematic investigation. For example, questions like ‘how does the distribution of tasks affect intrinsic rewards’ or ‘does intrinsic reward generalise’ are not really answered by providing metrics of performance or generalisation in controlled experiments, but by providing some example cases. Several of these questions (including optimising exploration and dealing with non-stationarity) also seem to have been investigated to some extent in the original Optimal reward papers (Singh, 2009/2010). It would be good to clearly indicate what we have learned beyond these previous results.
* There seems to be a bit of a mismatch between the learning objective for intrinsic rewards in the optimal reward framework and the results shown in the experiments. The learning objective aims to optimise lifetime rewards for a distribution of tasks. Most of the experiments seem to analyse episodic reward performance and compare against single-task (or task agnostic) methods.
Minor comments:
- The architecture / parameterization of the lifetime value function does not seem to be defined anywhere. Given that it takes histories as input I assume this is another RNN?
- There seems to be some small overloading in the notation with \eta occasionally being used to denote the parameters of the reward function r_eta or the reward function itself.


Are Graph Neural Networks (GNNs) fair? In many real world graphs, the formation of edges is related to certain node attributes (e.g. gender, community, reputation). In this case, standard GNNs using these edges will be biased by this information, as it is encoded in the structure of the adjacency matrix itself. In this paper, we show that when metadata is correlated with the formation of node neighborhoods, unsupervised node embedding dimensions learn this metadata. This bias implies an inability to control for important covariates in real-world applications, such as recommendation systems. To solve these issues, we introduce the Metadata-Orthogonal Node Embedding Training (MONET) unit, a generalizable neural network architecture for performing training-time linear debiasing of graph embeddings. MONET achieves this by ensuring that the node embeddings are trained on a hyperplane orthogonal to that of the node metadata. This effectively organizes unstructured embedding dimensions into an interpretable topology-only, metadata-only division with no linear interactions. We illustrate the effectiveness of MONET though our experiments on a variety of real world graphs, which shows that our method can learn and remove the effect of arbitrary covariates in tasks such as preventing the leakage of political party affiliation in a blog network, and thwarting the gaming of embedding-based recommendation systems.

The paper presents an approach to debiasing graph embeddings from known, given node attributes/metadata.
Specifically, the paper proposes to learn an embedding that is orthogonal to given node attributes, ensuring that there is no *linear* function which can extract the node attributes from the learned embedding.
Strength:
-	The paper addresses an interesting and relevant problem on debiasing graph embeddings.
-	The paper presents a, to my knowledge, novel approach, to avoid the leakage of meta-data in the embedding, effectively debiasing it from this information (although some discussion of prior should be addressed, see below)
-	The paper shows that the approach is effective compared to two baselines on two datasets (although some aspects of the experiments can be improved, see below)
Weaknesses:
1.	Experimental evaluation:
1.1.	The paper only evaluates on the training set. Specifically, in experiment 1, the paper learns an embedding and supervises it to be orthogonal to given labels; this is good, but it also somewhat expected that this could be learned when tested on the same dataset. An important question is, if the model actually learned a generalizable embedding or just overfit to the training set. If the learned embedding is applied to new data, is it still not possible to extract political affiliation from it?
I would suggest splitting the dataset in two part: One part to train the debiased embedding and one to train and test the Linear SVM.
1.2.	It would be great if the authors describe better and quantify how they ensure in experiment 1 that the learned embedding of the MONET model is measured, and how this compares to the three baselines (e.g. a random or constant embedding would also be perfectly debiased).
1.3.	While the paper clearly states that the approach is restricted to linear relationships, it would be interesting to look at non-linear classifiers and see how well this works in practice, also in comparison to the baselines.
1.4.	Figure 3(c) visualizes that
2.	Related work:
2.1.	The comparison to related work could be improved. Specifically, a discussion relating this work to adversarial training, e.g. as in domain confusion networks [A] or in [5].
2.2.	The authors argue that [5] is independent/concurrent work. I agree with the authors that [5] is sufficiently different to this work, but it should be discussed thoroughly as it has been published at ICML 2019. Unfortunately, the authors also miss to include in the references that it has been published at ICML 2019.
While the paper explores an interesting direction and approach, there are several concerns which speak against acceptance (see Weaknesses above); however, I believe they can be a addressed/clarified in a further revision.
References:
[A] Tzeng et al, Adversarial Discriminative Domain Adaptation, CVPR 2017
=== Post author response
I thank the authors for their response and additions as well as clarifications.
I agree to other reviewers, that the limitation to linear de-biasing is a concern for the paper, but the authors have clarified it now in the abstract an other locations; the additional experiments with and RBF kernel have shown that  indeed the formulation only does mainly do linear decorrelation.
Table 3 could be further clarified (what is the goal? 50% accuracy, i.e. chance level?) and made more similar to Table 1, or ideally merged with it.
It is also a bit unfortunate that all this additions are in supplement and not merged in the main paper.
Overall I still lean more towards accept.


Most research on lifelong learning applies to images or games, but not language. We present LAMOL, a simple yet effective method for lifelong language learning (LLL) based on language modeling. LAMOL replays pseudo-samples of previous tasks while requiring no extra memory or model capacity. Specifically, LAMOL is a language model that simultaneously learns to solve the tasks and generate training samples. When the model is trained for a new task, it generates pseudosamples of previous tasks for training alongside data for the new task. The results show that LAMOL prevents catastrophic forgetting without any sign of intransigence and can perform five very different language tasks sequentially with only one model. Overall, LAMOL outperforms previous methods by a considerable margin and is only 2-3% worse than multitasking, which is usually considered the LLL upper bound. The source code is available at https://github.com/xxx.

Summary:
The paper proposes to use the same language model to learn multiple tasks and also to generate pseudo-samples for these tasks which could be used for rehearsal while learning new tasks. The authors demonstrate that this idea works well compared to other SOTA lifelong learning methods for learning various NLP tasks using a single model.
My comments:
1. Please change the title! Language modeling is NOT all you need for lifelong language learning. Also, not every NLP task is a QA task. I do not want more papers to over-trivialize NLP by following Bryan McCann and Socher, 2018. I will not increase my scores until the title is changed.
2. A relevant model architecture based method is Sodhani et al. 2018 (Towards Training Recurrent Neural Networks for Lifelong Learning) who use Net2Net to do zero-shot expansion of the model parameters.
3. Section 3.2 - you mention that any pseudo-example which does not have only one ANS token is discarded. Can you comment on how much discarding is needed to generate the required number of pseudo-samples?
4. Why is it that every task was trained only for 9 epochs?
5. On page 5, you mention k=20. What is k? Where is this introduced?
6. On page 5, you mention that MTL is used to determine whether forgetting is caused by a lack of model capacity. I am not sure if it is correct. Can you explain?
7. Why not compare the approach with models like GEM? Keeping very few examples is ok. Even though you don’t beat GEM, it is good to see the comparison.
8. Page 7: Is there any reason why you choose to go from large to small tasks? I feel like this is a favorable order. I would like to see how the model performs if you do the reverse order.
9. Please remove the last line.
10. I assume that the authors will release the code upon acceptance of the paper.
Minor comments:
1. Page 2, 4th contribution: check the spelling for “pseudo-samples”
2. Page 2, 5th last line: “After a completing a task” - fix it.
3. Table 1: I think the description is not correct. 1fEM is for wikiSQL, not WOZ. Also, it is better if you can describe these metrics in detail in the appendix.
==================================
After rebuttal:
I am happy with the authors' response and name change. I am increasing my score.


We propose a meta path planning algorithm named Neural Exploration-Exploitation Trees (NEXT) for learning from prior experience for solving new path planning problems in high dimensional continuous state and action spaces. Compared to more classical sampling-based methods like RRT, our approach achieves much better sample efficiency in high-dimensions and can benefit from prior experience of planning in similar environments. More specifically, NEXT exploits a novel neural architecture which can learn promising search directions from problem structures. The learned prior is then integrated into a UCB-type algorithm to achieve an online balance between exploration and exploitation when solving a new problem. We conduct thorough experiments to show that NEXT accomplishes new planning problems with more compact search trees and significantly outperforms state-of-the-art methods on several benchmarks.

Summary:
Motion-planning in high dimensional spaces is challenging due to the curse of dimensionality. Sampling-based motion planners like PRM, PRM*, RRT, RRT*, BIT* etc have been the go-to solution family. But often these algorithms solve every planning problem tabula rasa. This work combines learning with sampling-based planning such that the parent-sampling and expansion steps instead of being done by common heuristics are learnt in an online manner. Also the resulting exploration-exploitation problem is naturally dealt via using a UCB-style contextual bandit algorithm. Since the number of parents are always varying the common trick of 'describe the action choices with respect to the environment' is adopted so that varying number of actions (states to be sampled from) can be naturally incorporated.
The other significant aspect of this paper is that there is a self-improving component (Algorithm 3) where a dataset is built up every time step, of environments where either an expansion with RRT or the learnt expansion policy is attempted with the policy being invoked more as time goes on and it trains more. If the process succeeds in finding a path to the goal then this example is added to a dataset and the dataset used to update the policy and associated value function to guide it towards the feasible paths found in the tree.
Comments:
- Algorithm 3: "Reconstruct optimal path". These paths are not really optimal for the problem. They are optimal in the tree T that is built so far for example U. But for the problem they are feasible and if RRT* were to be run asymptotically then perhaps near-optimal. The accompanying text should be updated accordingly so that there isn't confusion.
- Here is my main concern with Algorithm 3: For equation 6  where the policy and value functions are updated, the policy is inevitably going to suffer from covariate shift. This is because the algorithm is essentially doing behavior cloning (BC) with respect to the feasible paths found on the planning examples. Since we are inherently in a sequential setting (non-iid) where the states visited by the policy are a direct result of its own decisions the error bound will be quadratic in the horizon (path-length) for equation 6. This phenomenon has been well-understood in imitation learning literature and algorithms like DAgger, AggreVate or online versions like AggreVateD, LOLS already address these problems in a principled manner. Equation 6 should ideally be replaced with an inner DAgger/AggreVateD like loop (with an RRT* dynamic oracle) for stable learning of policy and value function. I am happy to be convinced that covariate shift and resulting quadratic mistake-bound problems are not present here.
- Application of imitation learning to both self-improvement style path planning and leveraging experience in planning has been done before: See "Learning to Search via Retrospective Imitation
Jialin Song, Ravi Lanka, Albert Zhao, Aadyot Bhatnagar, Yisong Yue, Masahiro Ono, 2018" (this is unpublished it seems so it is unfair of me to mention this perhaps but I wanted to give an example of how to use dynamic oracles for stable imitation in planning.) and "Data-driven Planning via Imitation Learning
Sanjiban Choudhury, Mohak Bhardwaj, Sankalp Arora, Ashish Kapoor†, Gireeja Ranade, Sebastian Scherer and Debadeepta Dey", IJRR 2018. At least the last paper should be cited and discussed in related work.
- Also would be curious how the authors would situate methods which are non-learning based but leverage experience in planning (example E-Graphs: Bootstrapping Planning with Experience Graphs, Phillips et al, RSS 2012) via graphs discovered in other problems directly. Perhaps a discussion in related work is warranted?
Update: After rebuttal updating to Accept.


It has been of increasing interest in the field to develop automatic machineries to facilitate the design process. In this paper, we focus on assisting graphical user interface (UI) layout design, a crucial task in app development. Given a partial layout, which a designer has entered, our model learns to complete the layout by predicting the remaining UI elements with a correct position and dimension as well as the hierarchical structures. Such automation will significantly ease the effort of UI designers and developers. While we focus on interface layout prediction, our model can be generally applicable for other layout prediction problems that involve tree structures and 2-dimensional placements. Particularly, we design two versions of Transformer-based tree decoders: Pointer and Recursive Transformer, and experiment with these models on a public dataset. We also propose several metrics for measuring the accuracy of tree prediction and ground these metrics in the domain of user experience. These contribute a new task and methods to deep learning research.

Summary: This paper introduces the task of using deep learning for auto-completion in UI design. The basic idea is that given a partially completed tree (representing the design state of the UI), the goal is to predict or "autocomplete" the final tree. The authors propose a transformer-based solution to the task, considering three variants: a vanilla approach where the tree is flattened to a sequence, a pointer-network style approach, and a recursive transformer. Preliminary experiments indicate that the recursive model performs best and that the task is reasonable difficulty.
Assessment: Overall, this is a borderline paper, as the task is interesting and novel, but the presentation is lacking in technical detail and there is a lack of novelty on the modeling side.
In particular, the authors spend a bulk of the paper describing the three different baselines they implement. However, despite the fact that most of the paper is dedicated to the explanation of these baselines. There is not sufficient detail to reproduce the models based on the paper alone. Indeed, without referencing the original Pointer Network and (and especially the) Transformer papers, it would not be possible to understand this paper at all. Further technical background and detail would drastically improve the paper. Moreover, it seems strange that significant space was used to give equations describing simple embedding lookups (i.e., matrix multiplications with one-hot vectors), but the basic technical foundations of Transformers were not adequately explained.  In addition, only the transformer baselines were considered, and it would seem natural to consider LSTM-based baselines, or some other related techniques.  In general, the space that was used to explain the Transformer baselines---which are essentially straightforward ways to adapt transformers to this task---could have been used to give more detail on the dataset. For example, one question is how often a single partial tree has multiple possible completions in the data.
A major issue---mainly due to the lack of technical details and the lack of promise to provide code/data (unless I missed this)---is that the paper does not appear to be reproducible. Given the intent to have this be a new benchmark, ensuring reproducibility seems critical.
Reasons to accept:
- Interesting new application of GNNs
Reasons to reject:
- Incremental modeling contribution
- Lack of sufficient technical detail on models and dataset
- Does not appear to be reproducible


Exciting new work on generalization bounds for neural networks (NN) given by Bartlett et al. (2017); Neyshabur et al. (2018)  closely depend on two parameterdependant quantities a) the Lipschitz constant upper bound and b) the stable rank (a softer version of rank). Even though these bounds typically have minimal practical utility, they facilitate questions on whether controlling such quantities together could improve the generalization behaviour of NNs in practice. To this end, we propose stable rank normalization (SRN), a novel, provably optimal, and computationally efficient weight-normalization scheme which minimizes the stable rank of a linear operator. Surprisingly we find that SRN, despite being non-convex, can be shown to have a unique optimal solution. We provide extensive analyses across a wide variety of NNs (DenseNet, WideResNet, ResNet, Alexnet, VGG), where applying SRN to their linear layers leads to improved classification accuracy, while simultaneously showing improvements in generalization, evaluated empirically using shattering experiments (Zhang et al., 2016); and three measures of sample complexity by Bartlett et al. (2017), Neyshabur et al. (2018), & Wei & Ma. Additionally, we show that, when applied to the discriminator of GANs, it improves Inception, FID, and Neural divergence scores, while learning mappings with a low empirical Lipschitz constant.

Stable Rank Normalization for Improved Generalization in Neural Networks and GANs
Summary:
This paper proposes to normalize network weights using the Stable Rank, extending the method of spectral normalization. The stable rank is the ratio of the squared frobenius norm to the squared spectral norm (the top singular value). The authors motivate this decision in the context of lipschitz constraints and noise sensitivity. The proposed method (combined with spectral norm as SRN alone does not explicitly subsume SN) is tested for both classification (using a wide range of popular models on CIFAR) and GAN training. Performance (classification accuracy and FID/IS) is measured and several auxiliary investigations are performed into generalization bounds, sample complexity, and sensitivity to hyperparameters.
My Take:
This paper motivates and presents an interesting extension of spectral norm, and evaluates it quite well with thorough experiments in a range of settings. The method looks to be reasonably accessible to implement, although its compute cost is not properly characterized and some details (like the orthogonalization step necessary in power-iteration for more than one SV) seem to be omitted. My two main concerns are that the results, while good, are not especially strong (the relative improvement is not very high) and that the paper could be made substantially more concise to fit within the 8 page soft limit (I felt there was plenty of material that could be moved to the appendix). All in all this is a reasonably clear accept to me (7/10) that with some cleanup could be a more solid 8, and I argue in favor of acceptance.
Notes
-The paper should characterize the runtime difference between SRN and SN. It is presently unclear how computationally intensive the method is. What is the difference in time per training iteration? The authors should also indicate their hardware setup and overall training time.
-I found table 1 confusing as it lacks the test error. Are the test errors the same for all these models and the authors are just showing that for certain settings the SRN models have higher training error? If there is a difference in testing error, then this table is misleading, as one cares little about the training error if the test errors vary. If the test errors are approximately the same, then why should I care if the training error is higher? This would just be a way to decrease the stated “generalization gap,” which is not necessarily indicative of a better model (vis-à-vis the commonly held misconception that comparing training error between models is properly indicative of relative overfitting).
-Nowhere (that I could spot) in the body of the paper is it explained what “Stable-50”, “SRN-50”, and “SRN-50%” are. I assume these all mean the same thing and it refers to the choice of the c hyperparameter, but this should be explicitly stated so that the reader knows which model corresponds to which settings.
Minor
-The footnotes appear to be out of order,  footnote 1 appears on page 9
-There are typos such as “simiplicity,” please proofread thoroughly.


Neural Architecture Search (NAS) has shown great potentials in finding a better neural network design than human design. Sample-based NAS is the most fundamental method aiming at exploring the search space and evaluating the most promising architecture. However, few works have focused on improving the sampling efficiency for a multi-objective NAS. Inspired by the nature of the graph structure of a neural network, we propose BOGCN-NAS, a NAS algorithm using Bayesian Optimization with Graph Convolutional Network (GCN) predictor. Specifically, we apply GCN as a surrogate model to adaptively discover and incorporate nodes structure to approximate the performance of the architecture. For NAS-oriented tasks, we also design a weighted loss focusing on architectures with high performance. Our method further considers an efficient multi-objective search which can be flexibly injected into any sample-based NAS pipelines to efficiently find the best speed/accuracy trade-off. Extensive experiments are conducted to verify the effectiveness of our method over many competing methods, e.g. 128.4× more efficient than Random Search and 7.8× more efficient than previous SOTA LaNAS for finding the best architecture on the largest NAS dataset NASBench-101.

This paper proposed BOGCN-NAS that encodes current architecture with Graph convolutional network (GCN) and uses the feature extracted from GCN as the input to perform a Bayesian regression (predicting bias and variance, See Eqn. 5-6). They use Bayesian Optimization to pick the most promising next model with Expected Improvement, train it and take its resulting accuracy/latency as an additional training sample, and repeat.
They tested the framework on both single-objective and multi-objective tasks.  On the single-objective (accuracy task). They tested it on NasBench and LSTM-12K, two NAS datasets with pre-trained models and their performance. They obtained very good performance on both, beating LaNAS (previous SoTA) by 7.8x higher sample efficiency. On multiple-objective, they show higher efficiency in finding Pareto frontier models, compared to random search.
One main question I have is whether the next model is chosen given the current prediction model? For NasBench, did you run your predictor for all (420k minus explored) models and pick the one that maximizes Expected Improvement? Note that LaNAS is more efficient in that manner by sampling directly on polytopes formed by linear constraints. If so, how do you pick the next candidate models in open domain setting?
It looks like Eqn. 9 biases the training heavily towards high accuracy region, which is a hack. Although in the Appendix (Fig. 7) the authors have already perform some analysis on the effect of different weight terms, I wonder whether there is a more problem-independent way of doing it. The MCTS in LaNAS is one way that automatically focuses the search on the important regions. Currently, the proposed approach might limit the usability of the proposed method to other situations when accuracy is no longer that important.
The performance is really impressive in the NasBench and LSTM dataset. The paper mentioned that “BO can really predict the precious model to explore next” but didn’t provide an examples in the paper. I would whether the author could visualize it in the next revision, which would be very cool.
Do you have open domain search research in single-objective search?
Why not use NasNet architecture for a fair comparison with other NAS papers?
In Appendix, Fig. 6 shows that even without GCN and BO, a single MLP already achieves global optimal solution in LSTM-12K dataset with ~850 samples, already beating all the previous methods (Random, Reg Evolution, MCTS and LaNAS). If that’s the case, I wonder how much roles the proposed methods (BO + GCN) play during search? Also what do you mean by “without BO”? Do you only predict the mean and assume all variance is constant?
=====Post Rebuttal======
I have read other reviewers' comments and the rebuttal.
One of the main problems in this paper is an unfair comparison against LaNAS. LaNAS only uses a single sample at each leaf, while they sample 100% to 0.1% of the models, evaluate them with the current BO model and find the best. For NasBench-101 with 420K models, even sampling 0.1% each time means ~400 samples and the performance (from the rebuttal) seems to degrade substantially from 100% case (1464.4->4004.4). This means that almost 3x more samples are needed, compared to what they claimed.
I agree with the authors that calling BO function is super fast so maybe this is fine. However, on the open domain experiments, their performance is also not better than LaNAS+c/o, which they didn't list in the rebuttal. I listed it here:
Model                      	Params       	Top-1 err	No. of samples truly evaluated
-----------------------------------------------------------------------------------------------------------------
BOGCN+cutout (V1)	   3.1M 	          2.74	                      200
BOGCN+cutout (V2)	   3.5M 	          2.61	                      400
LaNAS+c/o                       3.2M              2.53±0.05                    803
Overall this paper is on the borderline. I don't mind if the paper gets rejected. For now I lower the score to 3.


Recently, non-Euclidean spaces became popular for embedding structured data. However, determining suitable geometry and, in particular, curvature for a given dataset is still an open problem. In this paper, we define a notion of global graph curvature, specifically catered to the problem of embedding graphs, and analyze the problem of estimating this curvature using only graph-based characteristics (without actual embedding the graph). We show that optimal curvature essentially depends on the dimensionality of the embedding space and loss function one aims to minimize via embedding. We review existing notions of local curvature (e.g., Ollivier-Ricci curvature) and analyze their properties theoretically and empirically. In particular, we show that such curvatures are often unable to properly estimate the global one. We also make an important observation: larger dimensionality usually leads to less negative optimal curvature. We propose a new estimator of global graph curvature which, in contrast to the existing ones, reflects this property as it is dimension-specific. We demonstrate that for a given dimension the new estimator is able to distinguish between "hyperbolic" and "Euclidean" networks.

Summary:
This paper is about curvature as a general concept for embeddings and ML applications. The origin of this idea is that various researchers have studied embedding graphs into non-Euclidean spaces. Euclidean space is flat (it has zero curvature), while non-Euclidean spaces have different curvature; e.g., hyperbolic space has a constant negative curvature. It was noted that trees don't embed well into flat space, while they embed arbitrarily well into hyperbolic space.
All of the notions of curvature, however, are defined for continuous spaces, and have to be matched in some sense to a discrete notion that applies to graphs beyond a particular class like trees. The authors study this setting, consider a variety of existing notions of curvature for graphs, introduce a notion of global curvature for the entire graph, and now to efficiently compute it. They also consider allowing these concepts to vary with the downstream task, as represented by the loss function.
Pros, Cons, and Recommendation
The study of the various proposed distances is fairly interesting, although it's hard to say what the takeaway here is. I think the part I'm struggling with the most is the motivation. Why do we care about using a space of constant curvature? True, we do so when it's appropriate---we embed trees into hyperbolic space. But when we have a more complicated and less regular graph, then compressing all of that curvature information into a scalar doesn't seem like a good idea, and indeed that's the point of the Gu et al work that's being built on here: it mixes and matches various component spaces that each have constant curvature, but altogether have varying curvature.
At the same time, I like the idea of studying a bunch of proposed measures and attempting to gain new insights. This is a pretty unusual paper for ICLR, since the experimental section is really barely there, and what's most interesting are really these atomic insights. If the authors work on the motivation I would consider accepting it---for now I gave it weak accept.
Comments:
- The distinction between what the authors think of as "global" and "local" curvatures is confusing and should be explained further. From what I can see, the authors think of global as being a scalar, and local as being defined at each point; intuitively these seem like pretty bad labels. I would think of the "global" one as being coarse, and the "local" as being more refined, since it contains a lot more information. This is also related to the motivation: why stuff all of this information into one single scalar curvature? It forces you to take averages, while Gu et al defined a distribution over the local curvatures.
If the idea is to simply use one space and not a product, there are in fact various spaces with non-constant curvature, e.g., the complex manifold CH^n.
- Why do your need your graph to be unweighted at the very beginning of Section 2? On the other hand, you may want to define your graph to be connected for the distortion function to be well-defined.
- The statement "1, graph distances are hard to preserve:..." isn't really meaningful, since for the example in 4.3.1, it is possible to embed that graph arbitrarily well. That is, even if the distortion isn't 0, it can be made as small as we desire. There are indeed graphs that are hard to embed (i.e., have lower bounds that do not go to 0) in reasonably tractable spaces, and the authors actually prove such a result, but the star graph is not one of these.
- There's various tricks that actually make some of these graphs very easy to embed. One example is K_n in 4.33. Instead of just embedding K_n, embed the star graph on n+1 nodes, and place a weight of 1/2 on each edge. Now every pair of (non-central) nodes is at distance exactly. 1, and this thing is embeddable into hyperbolic space, etc. Interestingly, this is actually predicted by the Gromov hyperbolicity (for K_n_ that the authors briefly mention.
The reason I bring this up is that even if the authors' project is successful, simple graph transformations may induce much better embeddings. That's fine, though, although it should be mentioned.
- Can the authors write out what's going on for the hyperbolic lower bound on D_min in the  proof of Thm. 4.1?


In recent years, stochastic gradient descent (SGD) and its variants have been the dominant optimization methods for training deep neural networks. However, SGD suffers from limitations such as the lack of theoretical guarantees, vanishing gradients, excessive sensitivity to input, and difficulties solving highly non-smooth constraints and functions. To overcome these drawbacks, alternating minimizationbased methods for deep neural network optimization have attracted fast-increasing attention recently. As an emerging and open domain, however, several new challenges need to be addressed, including 1) Convergence depending on the choice of hyperparameters, and 2) Lack of unified theoretical frameworks with general conditions. We, therefore, propose a novel Deep Learning Alternating Minimization (DLAM) algorithm to deal with these two challenges. Our innovative inequality-constrained formulation infinitely approximates the original problem with non-convex equality constraints, enabling our proof of global convergence of the DLAM algorithm under mild, practical conditions, regardless of the choice of hyperparameters and wide range of various activation functions. Experiments on benchmark datasets demonstrate the effectiveness of DLAM.

The motivation for this paper is as follows:
Why SGD:
1. Easy
2. applied in online settings
However:
1. Convergence proof has been done by others, but the assumptions of their proofs cannot be applied to problems involving deep neural networks, which are highly nonsmooth and nonconvex
2. Suffer from gradient vanish
3. sensitive to the input
The paper is based on new ideas  on alternative minimization method. In particular the  loss function is reformulated as nested function associated with multiple linear and nonlinear transformations. This nested structure is then decomposed into a series of linear and nonlinear equality constraints by introducing auxiliary variables and penalty hyperparameters. Then  multiple subproblems are generated which can be minimized alternatively. (using ADMM, BCD)
However, these methods suffer from problems:
1. Convergence properties are sensitive to penalty parameters.
2. Lack of unified theoretical frameworks with general conditions. (Not fully proved the
convergence. Most of the work is based on some assumptions)
Main Assumption: Activation functions are quasilinear functions.
pros (+):
1. Generic, easy to extend it to convolutional layers and recurrent layers.
2. Transform the Neural network optimization problem into inequality constrained
problem (new point of view)
3. Ensure convexity of subproblems. (quadractic approximation of activation function).
4. Ensure global minima, and converges to the critical point (where 0 is in the set of
partial differential of F(F(W,b,,z,a) is the augment loss function) with respect to W*,
and set of partial differential of F with respect to b*).
5. The convergence rate is O(1/k), k is the number of iterations of updating (W,b,z,a).
6. inequality-constraint prevents the output of a nonlinear function from changing much
and reduces sensitivity to the input.
7. Matrix inversion is avoided by quadratic approximation
8. No need strict and complex condition, such as KL properties to prove convergence.
Instead, this algorithm needs simple and mild conditions to guarantee convergence,
and cover most of the loss function and activation functions.
9. The choice of hyperparameters has no effect on convergence.
10. Much better performance on MNIST
cons(-):
1. penalty parameters need to be tuned as ADMM does.
2. The regularizations $\omega$ is l1 and l2, otherwise, it will not be closed-form
solutions. The regularization like dropout, batch-norm can not be added into this algorithm.
3. The dataset is relatively naive. Only MNIST and Fashion MNIST. It would be better to include little more sophisticate dataset like ImageNet or Cifar10. Since the computation complexity is O(d^2) where d is the dimension of the features, large scale of image like ImageNet could cause problem. (Comparison: features of MNIST: 196, features of Fashion MNIST:784, features of ImageNet: 65536).
4. The Adagrad has better performance than DLAM on 500*500 on Fashion MNIST and close performance on MNIST. As the authors suggest: DLAM performed competitively for the Fashion MNIST dataset. This can be interpreted as follows:  the proposed algorithm performs well on easy datasets, however the performance is not much better than other algorithms  on Fashion MNIST, which is little more challenging. What would be your comment about this criticism? Can you provide any theoretical insight?  Certainly  it would strengthen the paper if there were more comparisons of  DLAM on more complex dataset.


We propose a novel node embedding of directed graphs to statistical manifolds, which is based on a global minimization of pairwise relative entropy and graph geodesics in a non-linear way. Each node is encoded with a probability density function over a measurable space. Furthermore, we analyze the connection between the geometrical properties of such embedding and their efficient learning procedure. Extensive experiments show that our proposed embedding is better in preserving the global geodesic information of graphs, as well as outperforming existing embedding models on directed graphs in a variety of evaluation metrics, in an unsupervised setting.

This paper proposed another graph embedding method. It focuses on directed graphs, and it embedded the graph nodes into exponential power distributions, which include the Gaussian distribution as a special case. The method is implemented by optimizing with respect to the free distributions on a statistical manifold so as to achieve the minimum distortion between the input/output distances.  The method is tested on several directed graph datasets and showed superior performance based on several metrics.
Overall, the submission forms a complete and novel contribution in the area of graph embeddings. A key novelty is that the authors used the asymmetry of KL divergences to model the asymmetry of the distances in directed graphs, and they use the fact that KL is unbounded to model the infinite distances in undirected graphs. The proposed method has three main hyperparameters, \lambda in eq.(1), \beta in eq.(2), and the dimensionality of the target embedding. The author showed that \lambda and \beta are not sensitive and can be set to the default values, and 2-dimensional distributions already give much better results as compared to alternative embeddings. Moreover, the author proposed a scalable implementation based on sampling. Furthermore, the authored justified their choice of the target embedding space through some minor theoretical analysis given in section 3.
The writing quality and clarity are good (well above average).
To further improve this paper (e.g., in the final version), the authors are suggested to incorporate the following comments:
In the experimental evaluation, it should include some cases when the dimensionality of the target embedding has a large value (e.g., 50). This will make the evaluation more complete.
There are some typos and unusual expressions.  For example, page 3, what is "a good proposal function"?
After eq.(1), mention \lambda is a hyperparameter (that is not to be learned).
Theorem 1 (2), mention the Fisher information matrix is wrt the coordinate system (\sigma^1, \cdots,\sigma^k, \mu^1, \cdots, \mu^k)
Ideally, the experiments can include an undirected graph and show for example that the advantages of the proposed method become smaller in this case.


For linear classifiers, the relationship between (normalized) output margin and generalization is captured in a clear and simple bound -a large output margin implies good generalization. Unfortunately, for deep models, this relationship is less clear: existing analyses of the output margin give complicated bounds which sometimes depend exponentially on depth. In this work, we propose to instead analyze a new notion of margin, which we call the "all-layer margin." Our analysis reveals that the all-layer margin has a clear and direct relationship with generalization for deep models. This enables the following concrete applications of the all-layer margin: 1) by analyzing the all-layer margin, we obtain tighter generalization bounds for neural nets which depend on Jacobian and hidden layer norms and remove the exponential dependency on depth 2) our neural net results easily translate to the adversarially robust setting, giving the first direct analysis of robust test error for deep networks, and 3) we present a theoretically inspired training algorithm for increasing the all-layer margin. Our algorithm improves both clean and adversarially robust test performance over strong baselines in practice.

The paper proposes a new generalization bound for deep neural networks and develops a regularizer which optimize quantities related to the bound and improve generalization error on competitive baselines. The paper treats deep neural network as a composition of functions (i.e. the layers) and introduces a new complexity measure, all layer margins, which depends on norm-based perturbation simultaneously applied to all the hidden layers of the neural networks. The all-layer margin can be seen as a generalization of the popular margin definition in SVM’s. Similarly, a generalization bound can be derived based on such all layer margins. The paper also shows that the bound has nice properties that improve upon some short-comings of previous efforts and compare favorably in terms of tightness. The paper also shows that this bound can be adapted to adversarial robustness of the deep model. Finally, a practical algorithm is proposed to optimize the approximation of the all-layer margin and improve upon the baseline models (i.e. trained with only cross-entropy) in a non-trivial manner.
I really like the paper! Jiang et al. 19 showed a surprisingly strong correlation between the margins at the hidden layers and generalization, which begs the question whether a generalization bound can be proven based on these perturbations. This paper shows that it is indeed the case with a much more sophisticated yet intuitive definition of margin that depends all hidden layers at the same time. Other than the nice theoretical properties outlined in the paper, the most exciting property of the all layer margin is that it can be directly optimized which the paper shows at the end. On the technical front, I have carefully read section 2 and Appendix A and read the other proofs at a coarser granularity. Barring some small notational errors, I did not find major errors in the derivation since standard techniques are used after the definition of the all layer margin. For the algorithm, while I have some questions about the thoroughness and accuracy of the approximation the results are quite compelling.
I think this paper should be accepted as I consider generalization to be one of the most important topics in deep learning and the paper is a valuable contribution both for theory and practices. In addition, the paper is relatively easy to follow for a topic so technically involved.
Barring my endorsement, I have some small complaints and questions:
    1. The notation for summation in the definition of \psi is quite difficult to parse. It would be good to either adopt a more explicit notation or have some extra clarification.
    2. The symbol for function F and the symbol for function class \mathcal{F} are sometimes mixed. For example, in lemma 2.1, the m_F on LHS should be m \circ \mathcal{F}.
    3. I am a little confused on the role of section 3 since, like the paper states, the bound in section 3 is always looser than section 2. Is this bound just for comparison with previous work or provide a tractable upper bound? If so this should be discussed.
    4. In lemma B.1, why are there 2 reference matrices and how should they be chosen?
    5. Condition A.1 assumes the function classes of each layer have some kind of bounded complexity. In a feedforward neural network, a single \mathcal{F}_i is just a linear operator which has pretty small complexity, but how about networks with residual connections? Is this assumption still reasonable? Likewise, since the proposed AMO is applied to residual networks, I want to hear the authors thought on this.
    6. Following the previous point, perhaps it would be nice to have some experiments on conventional feedforward networks and applied the proposed algorithm to all layers instead of skipping the contents of the residual block.
    7. Can the bound be accurately approximated? To me this seems almost combinatorially hard. If so, do the authors expect it to be non-vacuous? Given the results of Jiang et al., it seems that it is plausible to obtain fairly tight bound conditioned on the data and architecture.
    8. Adversarial robustness is a highly active and empirical-results-driven field, so some experiments on the defense against adversarial attack would greatly strengthen the theoretical results. Otherwise it’s hard to gauge how useful it is.


Building accurate language models that capture meaningful long-term dependencies is a core challenge in natural language processing. Towards this end, we present a calibration-based approach to measure long-term discrepancies between a generative sequence model and the true distribution, and use these discrepancies to improve the model. Empirically, we show that state-of-the-art language models, including LSTMs and Transformers, are miscalibrated: the entropy rates of their generations drift dramatically upward over time. We then provide provable methods to mitigate this phenomenon. Furthermore, we show how this calibrationbased approach can also be used to measure the amount of memory that language models use for prediction.

This paper highlights and studies the problem of "entropy rate drift" for language models: the entropy rate of the language generated by a trained model is much higher than the entropy of ground truth sequences and this discrepancy worsen with the length of generation. The authors interestingly claim that the well-known lack of coherence in long-term model generations is due to this entropy rate drift. Valuably, the entropy rate drift is characterized mathematically. The authors propose a calibration method and prove that their method can interestingly reduce *both* the entropy rate drift and the perplexity of a miscalibrated model, *even though* they assume a rather simplistic model of miscalibration (one that leaks a small amount of mass to all the sequences of a given length). The author quantitatively show that their calibration method reduces the entropy rate drift and qualitatively show that their generations "make sense" in the long-term. As an auxiliary result, they show that a similar calibration method can be used to quantify the past information used by the model by upper-bounding the mutual information between the current prediction and the long-term past, given the short-term past, e.g. I(W_t | W<{t-\tau} | W_{t-\tau:t-1}).
I really enjoyed reading this paper and was really intrigued by the author's solution. However, in the current form, this paper is below the bar of acceptance due to some weaknesses: (i) rather strong assumption for the main derivation; (ii) lack of clarity and computational complexity of the proposed algorithms; (iii) weak experimental results and missing hint to current models / applications / concurring models. I would be more than happy to increase my score if the authors could kindly respond to the points below.
1) About the assumptions: in your theory, you show that the proposed solution is guaranteed to improve a particular constrained form of miscalibrated model P^\epsilon, in which a epsilon uniform distribution over all possible sentences of length K is added to the model distribution. This seems a rather strong assumption to me and bumps up to increasing the probability of each word by a small amount for each per-step conditional distribution estimated by the model.
1.1) Could you elaborate on why intuitively this assumption is something that happens in current models like GPT-2 ? Do you have a way of quantifying whether this assumption reasonably holds ?
2) About complexity:
2.1) What’s roughly the computational complexity of Algorithm 2 ?
2.2) Do you need to compute H(W_{t+1}|w_{<= t}) for all possible 1-step continuations w_t ? That seems quite expensive to me in the case of a large vocab. (e.g. sample a word, run forward one step and compute future entropy).
2.3) If this is correct, how to address this issue ? If I am missing something, it would be good to add to the paper some explicit considerations about the technical feasibility of the algorithm.
3) The amazing thing to me is that you reduce entropy-rate drift (although wrt a particular miscalibrated model, cf. 1) by re-minimizing cross-entropy using the ground-truth corpus. If you could give intuition for this , it would be great and make a much stronger paper !
3.1) Could you explain why intuitively this works ?
3.2) If \alpha is positive, Algorithm 2 is basically penalizing the model for producing words that lead to higher entropy in the future. Why this correlates to less cross-entropy with the data distribution ?
3.3) Why would calibrating a model that may not conform to the assumption 1) in general prefer to have an \alpha > 0 ?
4) About the experiments:
4.1) Where are the perplexities of the calibrated models ?
4.2) Are the perplexity improving as the theory would suggest ?
4.3) Is there any small , systematic human study that you could perform apart from just showing few examples of generations ?
4.4) Could you report some quantitative metrics of the generations from your models and the baselines, for example as computed in previous papers (https://arxiv.org/abs/1801.07736, https://arxiv.org/abs/1908.04319)
4.5) How do your generations change if you use beam search or arg-max, top-k sampling ? Does your method also help in preventing repetitions ?
Minor:
- I think you are missing a minus in Algorithm 2 inside the exponential.
====
Updated review:
I thank the authors for their answer. I think that overall this paper provides interesting insight on a hard problem.
I am raising my score and strongly hope that the authors add the considerations made in their response in the main paper: warning about complexity, optimal \alphas for each model and details.


Multi-hop text-based question-answering is a current challenge in machine comprehension. This task requires to sequentially integrate facts from multiple passages to answer complex natural language questions. In this paper, we propose a novel architecture, called the Latent Question Reformulation Network (LQRnet), a multi-hop and parallel attentive network designed for question-answering tasks that require reasoning capabilities. LQR-net is composed of an association of reading modules and reformulation modules. The purpose of the reading module is to produce a question-aware representation of the document. From this document representation, the reformulation module extracts essential elements to calculate an updated representation of the question. This updated question is then passed to the following hop. We evaluate our architecture on the HOTPOTQA question-answering dataset designed to assess multi-hop reasoning capabilities. Our model achieves competitive results on the public leaderboard and outperforms the best current published models in terms of Exact Match (EM) and F 1 score. Finally, we show that an analysis of the sequential reformulations can provide interpretable reasoning paths.

This paper proposes a model for multi-hop question answering, specifically in a closed domain setting where the relevant paragraphs are present within a few distractor paragraphs. Their model has the following components — (a) a reader module that reads and collect information from paragraphs, (b) a query reformulation module, that reformulates the query to retrieve the next relevant document required to answer the question. Following the effectiveness of multi-head attention, the reader and reformulation module has K-heads that forms independent reformulations. The k-heads are aggregated via a simple summing and the reformulated query representation is used in the next step of the pre-defined hop.
Specifically, they start by encoding the question and paragraphs by BERT embeddings. Next they concatenate the fixed set of paragraphs (10 in their experiments) and encode it with a recurrent NN (bi-GRU) to produce token-level recurrent representation. This is followed by a reading module which dies document question attention in BiDAF (Seo et al., 2017) style followed by a self attention module. Next, in the query reformulation phase a convolution style filter is passed across the token embeddings of the document to gather / pool information required for querying. The current query representation is added to the new pooled representation to obtain the updated query representation. This is followed by an answering module that runs 4 layers of Bi-GRU and has an outlet for computing loss for each of the following — supporting facts , start , end of the answer span representation and a classifier to determine if its a yes-no question or a span answer. The network is trained via supervision for all the aforementioned 4 outlets.
Empricially, this paper shows gain in the distractor setting for HotpotQA dataset.
Strengths:
Query reformulation is an important strategy for IR, and multi-hop QA and it is nice to see that this model adapts it. Each module of the network seems to be carefully designed and the ablation results and analysis are helpful.
Weaknesses:
1. Related work: One of the major weakness of the paper is the missing related work. There are two well-established paper Das et al (ICLR 2019) — Multi-step Retriever-Reader Interaction for Scalable Open-domain Question Answering and Feldman and Yaniv (ACL 2019) — Multi-Hop Paragraph Retrieval for Open-Domain Question Answering that show query reformulation is effective for multi-hop question answering. Das et al 2019 proposes that the query reformulation module is independent of the reader module as long as it has access to the hidden representation of the reader module. This paper builds off from the original papers by carefully designing the reader and the reformulation modules (which is great!), but never mentions any of the above papers. I think that should be fixed.s
2. The setting considered in this paper is closed-domain where the number of paragraphs to be read are fixed and pre-determined. That is a very unrealisitic setting for a general purpose QA system. The paper should consider testing on the open domain setting of the HotpotQA dataset. The other two papers mentioned above (Das et al 2019, Feldman & Yaniv 2019) both test on open-domain setting. Moreover, it has been also shown recently that the distractor setting can easily be fooled and is not a great benchmark for testing the reasoning capabilities.
3. For a real QA system to be deployed in production setting, the system needs to be fast. I am afraid the model proposed by this paper is very computationally expensive. Apart from encoding the question and paragraphs by BERT, they have multiple Bi-GRU encodings (8 additional). That is going to be computationally intensive and the authors should strongly consider other approaches such as replacing bi-grus with transformers that can encode sequences in parallel, and parameter sharing.
For the above strong weaknesses, I am forced to give a low score to the current paper.


We present local ensembles, a method for detecting extrapolation at test time in a pre-trained model. We focus on underdetermination as a key component of extrapolation: we aim to detect when many possible predictions are consistent with the training data and model class. Our method uses local second-order information to approximate the variance of predictions across an ensemble of models from the same class. We compute this approximation by estimating the norm of the component of a test point's gradient that aligns with the low-curvature directions of the Hessian, and provide a tractable method for estimating this quantity. Experimentally, we show that our method is capable of detecting when a pretrained model is extrapolating on test data, with applications to out-of-distribution detection, detecting spurious correlates, and active learning.

# Summary of contribution
- The paper provides a novel fast and simple approximation of second-order local parameter sensitivity of neural networks, to estimate a form of uncertainty wrt to a test sample, which is further used and tested as a novelty detector.
- The method analyzes the most significant eigenvector/eigenvalues of the Hessian (of training loss), and use the compliment of their span to get directions of local perturbations to network parameters that affect training loss little ("ensemble subspace"). The novelty score is then based on how much the prediction is influenced by these perturbations.
- The idea of estimating "ensemble subspace" is interesting and computationally effective. Compared to other recent methods that also use second-order gradients for uncertainty, this paper is more generally applicable, and can be faster at test time. The paper demonstrates good performance on both simulated data and real data (CelebA faces with CNN, etc.).
# Decision TL;DR
I am giving a weak reject. The paper is strong in its idea, formulation, and theory, but is too similar to recent related works which this paper is reluctant to compare to (either in theory, efficiency, or performance). Since the contribution of the paper lies in the efficient approximation of local ensemble methods, readers cannot gauge how beneficial the contribution is compared to other approximations of ensembles.
# Pros
- Novel way to estimate a local neighborhood that affects training loss little (Note: not an expert in this line of research, not sure if it is completely novel) by estimating significant eigenvectors and using their compliment space.
- The paper is well-written, and relatively easy to understand, despite a few hard-to-follow spots
- Widely applicable post-hoc to any trained neural networks, and potentially faster training than ensembles / Bayesian approximated ensembles
- (Theoretical) stability compared to full Hessian inversion
# Cons
Motivation wrt other papers unclear, and a lack of comparison.
- Two of the cited papers (Gal & Ghahramani, 2016; Blundell et al., 2015) both work on local ensembles. The former uses MC dropout, the latter estimates a diagonal covariance of a Gaussian distribution of network parameters. These methods are not mentioned in the motivation or related work, which makes it hard to say this paper is well-placed in the literature.
- The reason that these methods are not compared to is insufficient. The paper only argues that they are not "post-hoc" methods. It is very unclear why in any circumstance (or use case) a post-hoc estimation of local ensemble must be (or is preferred to be) used, rather than having network parameters and local neighborhood jointly estimated. If it is for efficiency reasons, the paper does not provide any experimental comparison of the efficiency. Also, it is hard to argue that the "post-hoc" nature of this paper makes it so different from the two prior work. For the first prior work, the only time it is not post-hoc is when the original network does not have any dropout layer, and that circumstance is not very common. For the second prior work, one can easily make it post-hoc by training the network first, and estimate the diagonal covariance post-hoc using their loss.
- The advantage of this paper is that it is more efficient and stable than alternatives, but only  the full hessian inversion is discussed. In particular, it may be necessary to discuss this paper's efficiency against MC-dropout (Gal & Ghahramani, 2016). This method can be done in mini-batches, while the proposed method has to run forward and back-propagation separately for each sample to get gθ*(x'), and it is unclear how well that scales.
Efficiency analysis lacking
- As discussed above, the proposed method seems to need to back-prop for each test sample separately without using a batch. How much this affects test efficiency is unclear.
Experimental comparison with similar methods missing.
- The paper would benefit from comparing to the two cited papers (among which MC-dropout is so easy to implement) as well as a full hessian estimation (for toy datasets at least).
- The paper poses itself as an efficient alternative, so it would be essential to gauge experimentally how fast each method is.
Others. (not crucial issues)
- The performance of the paper's main method (LE w/ predictions) underperforms in Table 2, and a variant had to be proposed to make up for the performance drop. This suggests instability of the proposed method wrt new datasets.
- The claim in contribution "We identify underdetermination as a key factor in the unreliability of predictions" is not verified.
- Inability to scale up to large networks with larger m needed, compared to real ensemble methods or Bayesian networks with Gaussian distributions.
# Room for improvement (decreasing order of importance)
- Improve placement in the literature by discussing when this paper is more useful than prior work (Gal & Ghahramani, 2016; Blundell et al., 2015).
- Detailed theoretical or experimental analysis of efficiency against alternative approximations of ensembles.
- Performance comparison to ensemble and local Bayesian methods.
# Editorial issues
- Figure 3(c) x axis meaning unclear
- Figure 4 not mentioned in text, and unclear which experiment this refers to
- Table 2 experiment's loss gradient version is not explained.
#############################################################
POST REBUTTAL
#############################################################
TL;DR: The rebuttal addresses some but not all of my concerns. The MC-dropout comparison especially shows the difference between some approximate ensemble methods and ensembles that specifically changes the loss little (this paper). In the end, it is a good paper in terms of theory, although the experiments is lacking in crucial places (no comparison with many existing papers that do attempt to invert hessian) and lack of analysis of claimed efficiency, which I can only hope don’t turn out to be a big deal. I am increasing the score, to marginally above borderline, but please consider the following feedback for the camera-ready version.
> MC Dropout and Bayes by Backprop effectively measure different types of uncertainty from what our method targets, and so they should not be considered competing methods.
I disagree; by the same logic we can never compare SVM with random forest because they are so different.
> However, as we discussed in 4.1, small eigenvalues make turning this representation into a proper posterior distribution difficult, because these eigenvalues need to be inverted to obtain a covariance matrix, but these inverted eigenvalues can be numerically infinite.
I agree; but the main difference is the full hessian is hard to implement while there are papers that have implemented the diagonal hessian. It would benefit the paper to compare to that and prove the issue of instability on top of arguing theoretically.
> First, we clarify that our method can in fact be performed using mini-batches of test points (both the forward and backward passes).
As far as I know, with mini-batches, the gradients averaged over all samples in the mini-batch are computed for each network parameter. But this method needs the gradient wrt each sample separately.
If the authors have some implementation trick that allows computing network parameters’ gradients wrt each sample in the mini-batch separately, please include in the implementation details. Otherwise, please mention this drawback in any “fast” claim.
I will change the score upwards due to the informative rebuttal. Please include much of the discussion in the paper or appendix.


CAT: Compression-Aware Training for bandwidth reduction
Convolutional neural networks (CNNs) have become the dominant neural network architecture for solving visual processing tasks. One of the major obstacles hindering the ubiquitous use of CNNs for inference is their relatively high memory bandwidth requirements, which can be a main energy consumer and throughput bottleneck in hardware accelerators. Accordingly, an efficient feature map compression method can result in substantial performance gains. Inspired by quantization-aware training approaches, we propose a compression-aware training (CAT) method that involves training the model in a way that allows better compression of feature maps during inference. Our method trains the model to achieve low-entropy feature maps, which enables efficient compression at inference time using classical transform coding methods. CAT significantly improves the state-of-the-art results reported for quantization. For example, on ResNet-34 we achieve 73.1% accuracy (0.2% degradation from the baseline) with an average representation of only 1.79 bits per value. Reference implementation accompanies the paper.

In this article, the authors propose a compression-aware method to achieve efficient compression of feature maps during the inference procedure. When in the training step, the authors introduce a regularization term to the target loss that optimizes the entropy of the activation values. At inference time, an entropy encoding method is applied to compress the activations before writing them into memory. The experimental results indicate that the proposed method achieves better compression than other methods while holding accuracy.
There are still some issues as follows:
1.	The authors should carefully check the format of the references in the whole article. For example, in section 2, line 5 from the top and line 8 from the bottom, “Xiao et al. (2017), Xing et al. (2019)” and “(Chmiel et al., 2019)” are in the wrong format.
2.	It is suggested that the authors swap the order of formulation (8) and (9) in section 3.2 so that it will be a good correlation with the formulation (3) and (4).
3.	I am interested in learning the time taken by the proposed method during the inference procedure vs other related methods.
4.	The authors studied two differentiable entropy approximation in the paper, and they stated that they calculate soft entropy only on the part of the batch for the reduction of both memory requirements and time complexity in training. I hope the authors will clarify 1) Whether the accuracy will be affected by other differentiable entropy approximations; 2) what is the impact on accuracy if only part of the batch is considered.