Time is an important feature in many applications involving events that occur synchronously and/or asynchronously. To effectively consume time information, recent studies have focused on designing new architectures. In this paper, we take an orthogonal but complementary approach by providing a model-agnostic vector representation for time, called Time2Vec, that can be easily imported into many existing and future architectures and improve their performances. We show on a range of models and problems that replacing the notion of time with its Time2Vec representation improves the performance of the final model.

This paper introduces a particular learnable vector representation of time which is applicable across problems without the use of a hand-crafted time representation. Their representation makes use of a feed-forward layer with sine activations which operates on time data. As it is a vector representation, it combines well with other deep neural network methods. They motivate their problem well, explaining why time data is important to a variety of problems and situate their solution as an orthogonal approach to many current solutions in the literature. They make reference to fourier analysis as motivation for their representation. Finally, they provide experimental results to support their claims using fabricated and real-world time series datasets, as well as ablation studies to support their design decisions.
While I think this work has the potential to be a significant contribution, I rate this a weak reject because the theoretical motivation and analysis of the experimental results are lacking the depth of evidence I would expect for an ICLR paper. If you provide a deeper discussion of the provable claims about the power of your model via Fourier analysis and provide a table of test accuracy/recall@K with/without your representation for more than one other state of the art algorithm for these datasets, I would be convinced to strong accept.
Specific comments:
* p.3 third paragraph: you repeat yourself in math notation a few times here. Repeated equations usually indicate that there is something new happening, but all of these are just restatements of your theta sin(omega tau + phi) term. I would introduce the notation for t2v(tau) upfront and use that to define a(tau, k)[j] and f_j
* p.3 A clearer explanation of the theory here would help, as I think Fourier's theorem nicely supports your claims.
* p.4 first paragraph you claim that this method responds well to data which exhibits seasonality, but none of your datasets deal with data that would exhibit seasonality. There are plenty of simple real-world datasets available which show multi-scale periodic phenomena (activity or location data, weather data, travel data, etc.). In fact, segmentation and recognition of wearable device activity would be a great application for this method.
* p.4 third paragraph: Your claim of invariance to time rescaling is technically correct, but I am not convinced that a model can learn the correct omega values for an arbitrary rescaling (e.g. if the period is smaller than the time unit). You show that this works for a rescaling from 2pi/7 to 2pi/14, but it would be nice if there was experimental confirmation of this property with frequency > 1.
* p.6 Showing accuracy/recall across training epochs is not sufficient evidence to show that this is a useful representation. There should be some kind of comparison with test set results from other state-of-the-art work on these datasets. If adding your representation to the SOTA model improved test set performance (or at least sped up training without hurting test set performance), then that would be better evidence. If LSTM+T is the SOTA, say so and restate the author's test performance compared to yours. If this is what these graphs show, consider using a different visualization to make it clearer that you're improving the final performance, not just the training process.
* p.8 I think sine functions make optimization harder because they make the gradient function periodic with respect to the weights, creating infinitely many local extrema. Historically this may have been an issue, but deep neural networks have so many local minima it might not matter. Still, it would be good to show that trained performance doesn't depend on the initialization values more than a standard LSTM+T model.
* You have an interesting corner case where your neural network parameters are interpretable: you can interpret the omega values from your model as frequencies and investigate their values to see which kinds of periodicity your model uses. You do something like this on p.7, but it would be neat to see a histogram like the one you have for EventMNIST for one of the real-world datasets to see if it learns the domain-relevant time knowledge you claim that it should learn.


Applying reinforcement learning (RL) to physical systems presents notable challenges, given requirements regarding sample efficiency, safety, and physical constraints compared to simulated environments. To enable transfer of policies trained in simulation, randomising simulation parameters leads to more robust policies, but also significantly extends training time. In this paper, we exploit access to privileged information (such as environment states) often available in simulation, in order to improve and accelerate learning over randomised environments. We introduce Attention Privileged Reinforcement Learning (APRiL), which equips the agent with an attention mechanism and makes use of state information in simulation, learning to align attention between state-and image-based policies while additionally sharing generated data. During deployment we can apply the image-based policy to remove the requirement of access to additional information. We experimentally demonstrate accelerated and more robust learning on a number of diverse domains, leading to improved final performance for environments both within and outside the training distribution.
1 .

Summary - Building on top of the domain randomization principle (used to train policies robust to domain-variations) to learn policies which transfer well to new domains, the paper proposes an approach to improve and speed-up learning / training over randomized environments. The paper operates in a settings where the policy to be transferred only has access to observations -- images, etc -- and not the complete underlying state of a (simulated) environment. The underlying idea is to -- (1) maintain two sets of actor-critic networks - a symmetric pair where the actor has access to the underlying state and an asymmetric pair where the actor has access only to image observations; (2) evenly gather experiences from behavioral policies of both actors and store them in a shared replay buffer and (3) learn to align the attention placed by the policies over objects in the environment for the state and observation based actors. The idea is to leverage privileged information about the state (which is strictly more informative compared to observations) to learn robust observation based policies. Experimental results indicate the proposed approach improves generalization performance compared to several ablations of the same on both in-distribution and out-of-distribution environments.
Strengths
- The paper is generally well-written and easy to follow. The authors generally do a good job of motivating the proposed approach by leveraging access to privileged information in the environment / simulation / renderer during training to get more robust observation policies to transfer to novel settings. The proposed approach is presented after appropriately grounding the problem setting and preliminaries and the authors clearly state and evaluate on the axes of research questions they care about.
- The proposed approach is somewhat novel and extends prior work on using shared replay buffers and asymmetric actor critic methods to accelerate training. Although the specific focus is on aligning object-level attention from a state-based actor and an observation based actor, the authors adopt design choices that help in preventing degenerate solutions -- for instance, the object-weighted squared error loss component to learn the observation attention module.
- The experimental results more-or-less support the claims of the paper (however, only in comparison with ablations and 1 baseline) in the sense we see improvements in terms of average returns and sample-efficiency plots. Furthermore, the authors conduct ablations to understand which components of the proposed approach contribute significantly in different environments -- for both extrapolated and interpolated environments.
- Sec 4.6 presents interesting analysis of the learned attention (observation) attention mechanism for both interpolated and extrapolated environments. It seems that attention is generally placed over relevant aspects (objects / links) of the environment and the associated takeaways seem feasible.
Weaknesses
- Having said that, there some weaknesses / questions which if addressed would make the paper stronger and help in increasing the rating of the paper.
- Access to the object-specific attention maps seems to be an assumption that might not scale well across simulators. More realistic / richer simulations (say, 3D reconstructions of indoor rooms) may not always offer this much privileged information -- one might have access to fine-grained reconstructions from multiple viewpoints, but not object level maps. This, combined with the fact that major gains have only been demonstrated over the specified continuous control domains (ignoring the Atari results), makes me slightly concerned about the scalability of the proposed approach in terms of more real-world + applicable domain-transfer scenarios. Maybe a more general approach that learns to match some intermediate representations of the state and observation based actors is a more general approach. Can the authors comment on this?
- Including the entropy loss while training the attention mechanism for the state based actor is justified only through feasible interpretations of the attention visualizations for the JacoReach experiments. I’m curious how important is it for the state-based attention to be sparse? Does it actually affect performance if the entropy loss is not included while learning the state-based attention module?
- Although the paper mentions experience is gathered evenly over the behavioral policies of both the actors it’s slightly unclear how that is being performed without actually referring to the algorithm pseudocode in the appendix. I would encourage the authors to include / move the same to the main paper. It makes the entire pipeline much easier to grasp.
- Transfer to novel environments (not necessarily with domain-shifts) has also been studied in context of providing exploration incentives (see InfoBot - https://arxiv.org/abs/1901.10902) in addition to a (sparse / dense) episodic reward. I would be curious to see how well does APRiL compare to such approaches in a setting where both are applicable - say the MultiRoomNXSY set of experiments in InfoBot (pointed above). This is to understand if the gains obtained from APRiL are very specific to domain-shifts or are largely applicable to any form of novel environment transfer. Can the authors comment on this?
Reasons for rating
Beyond the above points of discussion, I don’t have major weaknesses to point 	out. I generally like the paper. The authors do a good job of identifying the sliver in which they make their contribution and motivate the same appropriately. My major point of concern is centered is around the fact that APRiL (probably) assumes access to much privileged information which may not be generally available across all kinds of environments, etc. My rating of the paper is based on the strengths and weaknesses highlighted above. Addressing / Responding to those appropriately would definitely help in improving the rating of the paper.


Although adaptive algorithms have achieved significant success in training deep neural networks with faster training speed, they tend to have poor generalization performance compared to SGD WITH MOMENTUM(SGDM). One of the state-of-the-art algorithms, PADAM, is proposed to close the generalization gap of adaptive methods while lacking an internal explanation. This work proposes a general framework, in which we use an explicit function Φ(·) as an adjustment to the actual step size, and present a more adaptive specific form ADAPLUS(ADA+). Based on this framework, we analyze various behaviors brought by different types of Φ(·), such as a constant function in SGDM, a linear function in ADAM, a concave function in PADAM and a concave function with offset term in ADAPLUS. Empirically, we conduct experiments on classic benchmarks both in CNN and RNN architectures and achieve better performance(even than SGDM). The code is anonymously provided in https: //anonfiles.com/daV7Ed6enb/AdaPlus_zip.

This paper proposes a new variation on adaptive learning rate algorithm that builds on a prior work Padam, which is also a concurrent submission to this conference. The method is empirically validated through through various domains and neural networks architecture. Though the empirical results are extensive, I am leaning towards reject because (1) The method is a very small variation; no insight is provided for why. (2) The theory is not very useful in justifying the method. (3) The empirical results are weak.
(1) This work builds on the prior work Padam, which is not a well justified algorithm. Same criticisms hence can be made to this paper. I don't see any strong arguments from the paper that well justifies the methodology either.
The problems are listed below:
Here's what I can find of the authors' interpretation of the prior work Padam:
"The internal cause is that a concave function is applied rather than the linear function in ADAM.
Once ε is extremely small and |g| ∈ (0, ε), the mapping value of Φ(·) would be much
larger in PADAM than in ADAM; therefore, PADAM can adapt to larger learning rate α,
thus flexibly adapting to the variable learning rate scheme."
(i) I don't think phrasing the cause as a result of concavity gives arise to any new insights. Simply, what this is trying to say is that Φ(·) is large when |g| is small, that's why PADAM can adapt to large learning rate.
Building on Padam, the paper further justifies the proposed method by:
" This form of Φ(·) not only directly inherits advantages of PADAM,
as is depicted in Figure 1, but also makes a better guarantee for larger learning rates. The
offset ∆ makes sure that Φ(·) can altogether avoid the extreme situation. Even when
|g| → 0, a more extensive learning rate αt is allowed. "
(ii) The paper inherits a serious problem from Padam, that is to assume large learning rate is important for learning rate decay. Padam didn't explain this, nor does this paper. So the justification is very weak.
(iii) The introduction of the offset is not novel. Just as the author noted, this is almost the same term epsilon in original updates. It's not rare that one also tunes epsilon for some optimization problems.
(2) Continuing the last point, the theoretical analysis focus on the convergence proof of the algorithm. The proof is not new. Theory also is not useful in justifying the method.
(3) The method seems not to be able to beat the previous baseline Padam, which makes it questionable as a practical algorithm.


In the field of Generative Adversarial Networks (GANs), how to design a stable training strategy remains an open problem. Wasserstein GANs have largely promoted the stability over the original GANs by introducing Wasserstein distance, but still remain unstable and are prone to a variety of failure modes. In this paper, we present a general framework named Wasserstein-Bounded GAN (WBGAN), which improves a large family of WGAN-based approaches by simply adding an upper-bound constraint to the Wasserstein term. Furthermore, we show that WBGAN can reasonably measure the difference of distributions which almost have no intersection. Experiments demonstrate that WBGAN can stabilize as well as accelerate convergence in the training processes of a series of WGAN-based variants.

The paper proposes a new way of stabilizing Wasserstein GANs by using Sinkhorn distance to upper-bound the objective of WGAN's critic's loss during the training. GAN stabilization is a well-motivated problem and limiting the dramatic changes of discriminator loss clearly helps achieving this goal. Experiments show that in few settings the proposed method successfully addresses this issue. However, the theoretical insights have multiple flaws and incorrect proofs, while the choice of experiments raises few questions. Finally, the resolution limitations suggest the overall contribution is rather incremental.
The paper thus requires further work in terms of design, theory and experiments; for instance the choice of Sinkhorn distance as a boundary-heuristic might be too limiting. Although the main bounding strategy seems to be a promising idea, this work does not meet the quality requirements of ICLR.
Pros:
A. Conceptual simplicity of the general method of bounding discriminator's objective.
B. The method works well for a few variants of Wasserstein GAN on medium resolutions (64x64, 128x128).
Issues:
1. Resolution limitation. The upper-bound cannot be reasonably computed for higher resolutions in mini-batch training.
2. It is unclear how costly the method is in terms of resources and training time.
3. Experiments were done with CelebA/CelebA-hq with DCGAN and BigGAN architectures at 64x64 resolution and BigGAN architecture for 128x128 resolution. However, they lack comparison with vanilla BigGAN (trained with hinge loss) and the explanation of the architectual differences. Authors only mention the lack of self-attention, yet the used architecture has much less parameters than the original BigGAN; it is also unclear how BigGAN for 64x64 resolution should look like. In fact, the choice of BigGAN for 64x64 resolution is questionable, as this model has been designed for datasets of much higher complexity.
4. In two out of three experiments, WBGAN-GP does not beat WGAN-GP. Given BigGAN's results at 64x64, it might be the case that bounding strategy just helps in the case of heavily-overparametrized model.
5. Definiton (8) is unclear: firstly, the objective L_theta is parametrized by parameters of discriminator (theta), over which the maximum on the rhs is taken (this follows in Appendices). Secondly, it has non-deterministic term d^lambda(..) which depends on random empirical distributions \hat(P_r), \hat(P_g) (in Proposition 8. which follows, authors take expectation over a Wasserstein distance between these distributions).
6. Remark 2's proof is incorrect. At the beginning of p.13 authors take D_theta(x) = sign(P_r(x) - P_g(x)), which is not a Lipschitz function. Also P_r and P_g may not be discrete so the entire expression does not make sense.
7. Remark 2 is not true non-deterministic formulation of (8). It is possible that P_g != P_r and \hat(P_g) = \hat(P_r); in such case sinkhorn distance is 0 and the remaining terms of rhs of (8) cancell out (e.g. P_r is a Dirac delta at 0 and P_g = Bernoulli(0.1), then it is possible that \hat(P_r) = \hat(P_g) = delta(0)). Perhaps re-shaping definition (8) so that it is deterministic would help achieving correctness of Remark 2.
8. Remark 3. is unclear: in what sense expression (8) can be optimized by gradient descent?
9. The paper seems rougly written (see below) and the text needs some polishing.
Typos/ requires clarification:
[p2.] Backgrounds -> background
[p.3] corresponds a \lambda -> corresponds to \lambda
[p.3] 'The proposed bounded strategy' -> bounding
[p.3] 'We name it general in two folds' (?)
[p.7] 'High-resolution' - 128x128 is not high.
[p.8] Discussions -> Discussion


Combining domain knowledge models with neural models has been challenging. End-to-end trained neural models often perform better (lower Mean Square Error) than domain knowledge models or domain/neural combinations, and the combination is inefficient to train. In this paper, we demonstrate that by composing domain models with machine learning models, by using extrapolative testing sets, and invoking decorrelation objective functions, we create models which can predict more complex systems. The models are interpretable, extrapolative, dataefficient, and capture predictable but complex non-stochastic behavior such as unmodeled degrees of freedom and systemic measurement noise. We apply this improved modeling paradigm to several simulated systems and an actual physical system in the context of system identification. Several ways of composing domain models with neural models are examined for time series, boosting, bagging, and auto-encoding on various systems of varying complexity and non-linearity. Although this work is preliminary, we show that the ability to combine models is a very promising direction for neural modeling.

The authors present a new hybrid models that incorporates traditional models with neural networks by boosting and reducing the residuals in stages. They show 3 different boosting schemes: sequential boosting, parallel boosting and cyclical boosting (inspired by variational autoencoders). Furthermore the authors present the new Ljung-Box (LJB) loss function which reduces autocorrelation. They test on 2 simulated systems and on a physical DC Motor and put an emphasis on testing their result on extrapolated (unseen) data.
Strengths:
- The authors combine traditional models with deep learning approaches.
- The models can extrapolate to unseen data better than conventional deep learning approaches.
- The results show that the authors found a loss function that can reduce autocorrelation.
Weaknesses:
- The paper lacks an exact explanation for every model that was used. This is made worse by a lack of consistency, e.g. the model Physics-Boost-Dense doesn't explain which of the 3 boosting schemes is used.
- The authors propose a new Ljung-Box (LJB)  loss function. It contains hyper parameter L "which should be larger than possible correlations". It is not mentioned how to find such a value or what the value for the experiments for. They show that this loss function can reduce autocorrelation, but it comes at the expense of higher RMSE. It is not clear in what scenario this is useful.
- The methodology is difficult to follow. E.g. there is a mathematical explanation in 3.1, but not for 3.2 or 3.3. In Fig. 1(1) and Fig. 1(2), they incorporate a traditional model, but not in Fig. 1(3). Later they show results for a cyclic method that incorporates a physical model, but it's not clear how that was done.
- The authors claim that they are using "a new method for creating testing data". However, testing on extrapolating data is not a new method.
Additional Comments:
- Equation (2): I assume the hat should only be above f, not the entire f(x). If not, I don't see this symbol explained or used anywhere else in the paper.
- h_t in equation (3) and (4) are not explained in the text
- Fig. 1(1) and Fig. 1(3) are not mentioned in the text
- Figure 3 and 4 are hard to understand. Instead of State 0/1/2, it should say angle / angular velocity / acceleration. There should be a table to accompany this figure, so that it's easier to compare the models. As is, it's very hard to compare e.g. Physics-Ensemble-Dense with Dense-Ensemble-RNN.
- There are plenty of typos and a general lack of clarity. E.g. this sentence: "The  alternative  of  performing  end-to-end  of  all  the  models  at  once  is  an  significant  alternative."
- Add reference for original Ljung-Box test


In the twilight of Moore's law, GPUs and other specialized hardware accelerators have dramatically sped up neural network training. However, earlier stages of the training pipeline, such as disk I/O and data preprocessing, do not run on accelerators. As accelerators continue to improve, these earlier stages will increasingly become the bottleneck. In this paper, we introduce "data echoing," which reduces the total computation used by earlier pipeline stages and speeds up training whenever computation upstream from accelerators dominates the training time. Data echoing reuses (or "echoes") intermediate outputs from earlier pipeline stages in order to reclaim idle capacity. We investigate the behavior of different data echoing algorithms on various workloads, for various amounts of echoing, and for various batch sizes. We find that in all settings, at least one data echoing algorithm can match the baseline's predictive performance using less upstream computation. We measured a factor of 3.25 decrease in wall-clock time for ResNet-50 on ImageNet when reading training data over a network.

The authors propose a simple method for avoiding bottlenecks during NN training, whereby training examples are utilized multiple times per read. The work focuses on cases where the cost of preparing a minibatch exceeds that of a training step (both a forward pass and, subsequently, a parameter update). Working within said regime, the authors investigate different strategies for 'echoing' examples.
Feedback:
  The proposed method itself is very simple: that's fine. While some cursory analysis of data echoing's theoretical implications would be appreciated, I am fine with practically motivated solutions that address real issues. Simple 'tricks' that are easy to implement and widely applicable are often useful tools. Especially when little theoretical analysis is provided, introducing such a trick requires strong empirical evidence to validate its efficacy. As it stands, failure to provide crucial information forces the reader to suspend disbelief when evaluating the proposed method's impact.
  The authors seem to tiptoe around the issue of the relative cost of prefetching a batch versus that of a combined forward pass and parameter update. The work is predicated upon the assumption that the ratio of said costs $R > 1$, but the authors state that an unspecified subset of their experiments violate this assumption. What's more, real-world values of $R$ are not reported (or even measured!). As per Amdahl's law, $R$ upper bounds the potential benefits for data echoing; hence, failing to report $R$ is more than a little bit concerning. By the same token, the appropriate statistic for various result figures would seemingly be time rather than, e.g., the number of fresh examples read. As a reviewer, I would rather see evidence that data echoing provides modest benefits in realistic scenarios than x3.25 speedup in self-described "contrived" examples. Alternatively, consider providing real-world examples where $R > 1$ to help ground your arguments.
Questions:
  - Why was extensive hyperparameter tuning necessary?
  - Why are most results reported in terms of time/steps to achieve a target value? If nothing else, consider providing the corresponding learning curves (as in Figure 8) as an appendix (incl. means and standard errors).
Nitpicks, Spelling, & Grammar:
  - Metaparameters -> hyperparameters
  - Streamline list at end of introduction:
    """
    In this paper, we demonstrate that data echoing:
        1. reduces the...
    """


It is important to collect credible training samples (x, y) for building dataintensive learning systems (e.g., a deep learning system). In the literature, there is a line of studies on eliciting distributional information from self-interested agents who hold a relevant information. Asking people to report complex distribution p(x), though theoretically viable, is challenging in practice. This is primarily due to the heavy cognitive loads required for human agents to reason and report this high dimensional information. Consider the example where we are interested in building an image classifier via first collecting a certain category of highdimensional image data. While classical elicitation results apply to eliciting a complex and generative (and continuous) distribution p(x) for this image data, we are interested in eliciting samples x i ∼ p(x) from agents. This paper introduces a deep learning aided method to incentivize credible sample contributions from selfish and rational agents. The challenge to do so is to design an incentive-compatible score function to score each reported sample to induce truthful reports, instead of an arbitrary or even adversarial one. We show that with accurate estimation of a certain f -divergence function we are able to achieve approximate incentive compatibility in eliciting truthful samples. We then present an efficient estimator with theoretical guarantee via studying the variational forms of f -divergence function. Our work complements the literature of information elicitation via introducing the problem of sample elicitation. We also show a connection between this sample elicitation problem and f -GAN, and how this connection can help reconstruct an estimator of the distribution based on collected samples.

- Summary
This paper studies the sample elicitation problem where agents are asked to report samples. The goal is then to evaluate the quality of these reported samples by means of a scoring function S. Following previous related works, the authors use the equivalence between maximizing the expected proper score and minimizing some f-divergence. Their approach relies on the dual expression of the f-divergence which writes as a maximum over a set of functions t. Theoretical guarantees are given for f-scorings obtained (with or without ground truth samples) by first computing the empirical optimal function t, then plugged to estimate the f-divergence. Finally, a deep learning approach is proposed by considering functions f parameterized as sparse deep neural networks.
- Critics
The paper is globally well written but not well motivated and sometimes difficult to understand.
In particular, the notions of "elicitation", "reports" and "score function" should be defined mathematically more clearly.
Moreover, the deep learning aspect of the paper is not well motivated and is introduced in a very arbitrary way. Why not choosing another parametric family of functions? Is there another (broad) family of functions for which the computation of the argmin in Equation (4.3) is more tractable in practice?
A convincing way to motivate this deep learning approach would be to include numerical experiments and to compare to other parametric families.


While the celebrated graph neural networks yield effective representations for individual nodes of a graph, there has been relatively less success in extending to deep graph similarity learning. Recent work has considered either global-level graph-graph interactions or low-level node-node interactions, ignoring the rich cross-level interactions between parts of a graph and a whole graph. In this paper, we propose a Hierarchical Graph Matching Network (HGMN) for computing the graph similarity between any pair of graph-structured objects. Our model jointly learns graph representations and a graph matching metric function for computing graph similarity in an end-to-end fashion. The proposed HGMN model consists of a multi-perspective node-graph matching network for effectively learning crosslevel interactions between parts of a graph and a whole graph, and a siamese graph neural network for learning global-level interactions between two graphs. Our comprehensive experiments demonstrate that our proposed HGMN consistently outperforms state-of-the-art graph matching network baselines for both classification and regression tasks.

In this paper, a hierarchical graph matching network, which considers both graph-graph interaction and node-graph interaction, is proposed. Specifically, the graph-graph interaction is modeled through graph level embeddings learned by GCN with pooling layers. While node-graph interaction is modeled using node embedding learned by GCN and attentive graph embedding aggregated from node embedding.
Some concerns about the paper are as follows:
1)	The novelty of the paper is incremental. The major contribution of the paper lies in the propose of multi-perspective matching function $f_m()$, which is somewhat similar to the Neural Tensor Networks proposed in [1] and utilized in [2] Although, in [2], the Neural Tensor Network is used to measure the similarity between graph-level embeddings.
2)	Some of the technical details of the paper is not clearly presented or well explained.
a.	In Eq. (7), attentive graph-level embeddings are calculated using weighted average of its node embeddings. However, it is not clear which node $i$ from the other graph should be used to calculate the weights (\alpha_{I,j}, \beta_{i,j}). Furthermore, it is also not clear why the attention score should solely base on a single node from the other graph rather than the information of the entire graph.
b.	In Eq. (10), it would be better if the authors could provide more motivations about using Bi-LSTM aggregator. Especially, the embeddings to be aggregated are unordered. What are the two directions in Bi-LSTM in this case? What is the benefit of using Bi-LSTM as aggregator compared with LSTM aggregator or other aggregators?
Some other questions to be clarified:
1)	Why different similarity score functions are adopted for the classification task and the regression task?
2)	For the classification task, the mean squared error loss is adopted. Why not using other more commonly used loss for classification task?
Suggestions:
It would be better if the authors could empirically show the effectiveness of the Bi-LSTM aggregator.
It would be helpful if the authors could conduct some investigation on how the number of perspectives affect the performance of the model.
[1] Reasoning With Neural Tensor Networks for Knowledge Base Completion
[2] SimGNN: A Neural Network Approach to Fast Graph Similarity Computation


We investigate the capacity control provided by dropout in various machine learning problems. First, we study dropout for matrix sensing, where it induces a datadependent regularizer that, in expectation, equals the weighted trace-norm of the product of the factors. In deep learning, we show that the data-dependent regularizer due to dropout directly controls the Rademacher complexity of the underlying class of deep neural networks. These developments enable us to give concrete generalization error bounds for the dropout algorithm in both matrix completion as well as training deep neural networks. We evaluate our theoretical findings on real-world datasets, including MovieLens, Fashion MNIST, and CIFAR-10.

Post Discussion Update:
The authors vehemently disagree with my critiques about discussion of / attribution to prior work.  They seem to think that the differences from Cavazza et al. [AIStats 2018] would be obvious to "even someone who has taken a basic course in machine learning."  However, both Reviewer #2 and Reviewer #3 mentioned the same issues, saying "much of the technical leverage exploited in this paper comes from earlier work...results a lot like Proposition 2 can be found in [Cavazza et al.]" (#2), and "I am surprised that [Cavazza et al.] is not cited with its due credit in the writing. I hope to see some concrete statement about the difference between the authors' contribution and the existing literature" (#3).  Since 3 out of 4 reviewers did not recognize a clear distinction, it is fair to say the differences to which the authors refer in their rebuttal should be discussed at length in the paper.  However, the authors have not posted a revised draft or any sample text , leaving me to keep my recommendation at 'reject.'
______________________________________________________________________________________________________________________________________
Summary:  This paper studies (Bernoulli) Dropout regularization in the context of matrix sensing and neural networks (when dropout is applied to only the last hidden layer).  The paper first focuses on matrix sensing, showing an explicit regularizer with connections to trace norm regularization and proving a generalization bound.  The paper then moves to neural networks, first showing an explicit regularizer in the case of a squared loss and dropout on the last hidden layer only.  When the input distribution is symmetric and isotropic, the explicit regularizer is shown to have connections to path norm regularization.  From this explicit regularizer, the authors then derive an upper bound on the generalization gap (Theorem 2).  Experiments are reported that show (#1) (stochastic) dropout does improve generalization in a matrix completion task and (#2) the theoretical results do predict generalization as tested on MNIST and CIFAR-10.
Pros:  Extending dropout to other tasks and understanding its general method of action is an important problem, thus making the paper well motivated.  Moreover, the approach of deriving explicit regularizers from the stochastic objective is a commendable strategy that could improve the stability and speed of converge.  Furthermore, understanding the generalization properties of neural networks is important, and as dropout seems to be a well-established tool for improving generalization, the paper’s approach is sensible.
Cons:  I find this paper to severely over-claim its contributions---in particular #1 and #3 from the introduction...
(#1) Dropout for matrix completion:  Claimed contribution #1 is incremental as it merely adapts the dropout strategy of Cavazza et al. [AIStats 2018] to matrix sensing.  Cavazza et al. [AIStats 2018]’s procedure “drop[s] columns of the factors” (a quote from their abstract), and this is exactly what is done in this paper: “...a procedure that randomly drops the columns of the factors during training” (p 1).  I find it suspicious that the only time Cavazza et al. [AIStats 2018]’s work is mentioned is in the Introduction’s long list of citations of previous dropout work.  In effect, this equates Cavazza et al. [AIStats 2018]’s work with much less related work (e.g. Bayesian interpretations).  The Cavazza et al. [AIStats 2018] work should surely be cited in the vicinity of Equation 2.  Furthermore, Cavazza et al. also discuss connections to trace norm regularization (Section 4) and should also be included in the paper’s discussion on page 3.
(#3) Dropout in NNs: The explicit regularizer derived in Prop 3 was previously derived by Wang & Manning [ICML 2013]; see their Section 3.1.  This work is not cited---another significant oversight.  However, the resulting complexity bounds derived from Wang & Manning [ICML 2013]’s explicit regularizer are original, to the best of my knowledge.
In general, the paper makes several claims that are at best ungenerous to previous work.  For instance, the Introduction claims “none of these [previous] works adequately address the following basic question: how does dropout control the capacity of deep neural networks?” (p 1).  This is an unsubstantiated claim given that the paper contains no Related Work section, which is needed since there have been dozens of papers written on dropout.  Moreover, this work considers only the case in which dropout is applied to the last hidden layer, not to the full network (which is a valid and sensible restriction).  Yet this essentially reduces the results to a study of dropout in linear models (except perhaps in Prop 4) and therefore I don’t see how one could claim the previous work of Wager et al. [NeurIPS 2013], which also studies dropout for linear models, doesn't address similar questions.  For another example, the paper claims on page 3: “These observations are specifically important because they connect dropout, an algorithmic heuristic in deep learning, to strong complexity measures that are empirically effective as well as theoretically well understood.”  I agree that the connections are important, but similar connections have already been established by (at least) Cavazza et al. [AIStats 2018], Wang & Manning [ICML 2013], and Wager et al. [NeurIPS 2013].  Such connections are not unique to this paper, as the text implies.
As for experiments, the matrix completion results to not validate any of the results in Section 2.  A comparison of training under the stochastic objective vs with the explicit regularizer is never performed (as is done in Cavazza et al. [AIStats 2018]).  Similarly, the generalization bounds are not shown to be useful.  The only thing that is shown is that the stochastic objective (again, which is a minor adaptation from Cavazza et al. [AIStats 2018]) does improve generalization.
Minor comments:
> ERM is never defined as an initialism for “empirical risk minimization”
> While the assumptions of an isotropic and symmetric input distribution in Prop 4 are unrealistic in general, such conditions would be satisfied by hybrid architectures defined by making the early layers of the network a (isotropic Gaussian) normalizing flow [Nalisnick et al., ICML 2019].
Final Evaluation:  I find the paper's only original and validated contribution to be using Wang & Manning [ICML 2013]’s explicit regularizer to derive the complexity upper bound in Lemma 1.  Due to the paper's lack of discussion and, at times, mischaracterization of previous work, the text needs to be significantly revised before it can be accepted.  A proper Related Work section must be added to discussion the previous literature on understanding dropout.
__References__
Nalisnick, Eric, et al. "Hybrid Models with Deep and Invertible Features." International Conference on Machine Learning. 2019.
Wang, Sida, and Christopher Manning. "Fast dropout training." International Conference on Machine Learning. 2013.


We describe a simple and general neural network weight compression approach, in which the network parameters (weights and biases) are represented in a "latent" space, amounting to a reparameterization. This space is equipped with a learned probability model, which is used to impose an entropy penalty on the parameter representation during training, and to compress the representation using a simple arithmetic coder after training. Classification accuracy and model compressibility is maximized jointly, with the bitrate-accuracy trade-off specified by a hyperparameter. We evaluate the method on the MNIST, CIFAR-10 and ImageNet classification benchmarks using six distinct model architectures. Our results show that state-of-the-art model compression can be achieved in a scalable and general way without requiring complex procedures such as multi-stage training.

The authors propose a novel, information theoretic approach to learning compressed neural networks, whereby model parameters $\theta$ are expressed in an (approximately discrete) latent space as $\theta = f_{\psi}(\phi)$. During training, "reparameterizations" $\Phi$ are treated as draws from a generative distribution $q(\phi)$. As per the source coding theorem, penalizing the entropy of $q$ (more accurately, of $\hat{q}$) directly promotes compressible choices of $\Phi$ and, therefore, of $\Theta$. After training, arithmetic coding of $q$ is used to compress $\Phi$. Empirical results on a suite of common benchmarks demonstrates the proposed algorithm gracefully trades off between model performance and size.
Frequency Domain Interpretation:
  Here, I am operating off of the assumptions that, in Sect 3.2 paragraph 3, $\phi_{w}$ should be $\psi_{w}$. If so, can "Our Method (DFT)" also be interpreted as frequency domain SQ (of convolutional filters)? When trained on natural images, CNN filters are typically "smooth" (see any visualization thereof) and this smoothness translates as a prior of sorts on $\Phi_{w}$. This insight was previously explored in [1, 2] for purposes of compressing CNNs. Does evidence of this manifest in your experiments? For example, do the empirical distributions of high- and low-frequency components of $\phi_{w}$ differ?
Feedback:
  The authors propose a general framework where in (seemingly) arbitrary decoders $f$ can be used for purposes of "reparameterization". It is therefore somewhat disappointing that, in practice, $f$ is restricted to affine transformations and, in some cases, even fixed. Memory issues notwithstanding, it is unclear how well complex decoders $f$ could be jointly learned with (surrogate) encodings $\tilde{\Phi}$ and density models $\tilde{q}$.
  The lack of comparison between the proposed method's test-time throughput and that of its baselines leaves open questions regarding potential real-world use cases. Much of the time, methods for compressing neural networks divide their attention between both memory footprint and prediction speed. Outright ignoring this aspect of the problem seems odd. As a reviewer, I would much rather see the authors take a proactive approach in addressing this issue.
Questions:
  - What do results look like if convolutional decoders $f_{\text{conv}}$ are jointly learned during training?
  - How sensitive is the algorithm to different groups assignments? Can assignments be reliably made by simply looking at the architecture?
Minor Comments:
  - The general notation of the paper is cumbersome at times, can $\phi$ to be changed to, e.g., $\tilde{\theta}$? Similarly, can notation such as $\theta_{k, W}$ be simplified as $W^{k}$?
  - Section 2 jumps around a bit. Consider trying to order things "chronologically" such that training comes before compression? By the same token, details regarding "Model Partitioning" could be move to the end of the section such that the preceding material just refers to $\theta$ more abstractly?
  - Some additional details regarding training (esp. discretization and model distributions $\tilde{q}$) would be appreciated.
[1] "Compressing convolutional neural networks in the frequency domain", Chen et al. 2016
[2] "CNNpack: Packing convolutional neural networks in the frequency domain", Wang et al. 2016


We study the most practical problem setup for evaluating adversarial robustness of a machine learning system with limited access: the hard-label black-box attack setting for generating adversarial examples, where limited model queries are allowed and only the decision is provided to a queried data input. Several algorithms have been proposed for this problem but they typically require huge amount (>20,000) of queries for attacking one example. Among them, one of the state-of-the-art approaches  showed that hard-label attack can be modeled as an optimization problem where the objective function can be evaluated by binary search with additional model queries, thereby a zeroth order optimization algorithm can be applied. In this paper, we adopt the same optimization formulation but propose to directly estimate the sign of gradient at any direction instead of the gradient itself, which enjoys the benefit of single query. Using this single query oracle for retrieving sign of directional derivative, we develop a novel query-efficient Sign-OPT approach for hard-label black-box attack. We provide a convergence analysis of the new algorithm and conduct experiments on several models on MNIST, CIFAR-10 and ImageNet. We find that Sign-OPT attack consistently requires 5× to 10× fewer queries when compared to the current state-of-the-art approaches, and usually converges to an adversarial example with smaller perturbation.

In this paper, the authors propose a new algorithm for evaluating adversarial robustness of black-box models. The aim of the proposed algorithm, SIGN-OPT is generating adversarial examples as close as possible to the decision boundary using as less queries of the black-box model as possible.
The authors follow the approach of (Cheng et al 2019) by modeling the problem as an optimization problem, where the objective function is to find the direction with the shortest distance to the decision boundary. They propose a smart modification of the previous approach by evaluating the sign of the gradient rather than the gradient itself. The advantage is that the sign of the gradient can be evaluated using a single query while the estimation of the gradient needs many queries.
The authors have analyzed the proposed algorithm. They showed that using SIGN-OPT, the expectation of the gradient tends to zero in $O(1/\sqrt(T))$, meaning that a (local) minimum is reached.
The algorithm is favorably compared with the state-of-the-art on three image test sets (MNIST, CIFAR-10n ImageNet).
This paper is technically sound, well-written and propose an interesting modification of a previous algorithm. I vote for acceptance.
However, I have some concerns:
-	I think that the L-smoothness assumption should be discussed. Is it realistic for Deep Learning models? Does it hold for the three attacked CNN networks?
-	The analytical results of the previous algorithm RGF and the proposed algorithm SIGN-OPT are not compared and discussed. It is a pity.


Domain shifts are frequently encountered in real-world scenarios. In this paper, we consider the problem of domain-invariant deep learning by explicitly modeling domain shifts with only a small amount of domain-specific parameters in a Convolutional Neural Network (CNN). By exploiting the observation that a convolutional filter can be well approximated as a linear combination of a small set of basis elements, we show for the first time, both empirically and theoretically, that domain shifts can be effectively handled by decomposing a regular convolutional layer into a domain-specific basis layer and a domain-shared basis coefficient layer, while both remain convolutional. An input channel will now first convolve spatially only with each respective domain-specific basis to "absorb" domain variations, and then output channels are linearly combined using common basis coefficients trained to promote shared semantics across domains. We use toy examples, rigorous analysis, and real-world examples to show the framework's effectiveness in cross-domain performance and domain adaptation. With the proposed architecture, we need only a small set of basis elements to model each additional domain, which brings a negligible amount of additional parameters, typically a few hundred.

The paper proposes an approach to learning domain invariant representations using the adaptive decomposition of the convolutional filters. The approach is similar to methods that use multi-stream networks (a stream for each domain), but using the filer decomposition scheme, the authors avoid the issue of excessive increase in the number of parameters typical in fully multi-stream architectures.
This is achieved by learning a separate basis for convolutional filters for each domain while sharing the basis coefficients across domains. This encourages shared semantics across domains while maintaining a balance between the network expressiveness and the computational complexity.
The authors argue that the basis learned for each domain can be understood as correction/alignment mappings to bring together the representations of each domain. A toy example presented is convincing and shows the correction basis learned in a simple synthetic case. Theoretical arguments are provided to show that the proposed correction scheme covers a large range of possible domain shifts.
The authors also show that plugging the decomposition scheme into existing CNN based unsupervised domain adaptation algorithms results in consistent improvements across methods and datasets.
How does the method compare to approaches that learn to adapt the representations using conditional/adaptive batch norm [1,2]?
Overall, the paper was well motivated and easy to read. The methods appear to be a useful addition to tools available for domain invariant learning.
[1] Chang, Woong-Gi, Tackgeun You, Seonguk Seo, Suha Kwak, and Bohyung Han. "Domain-Specific Batch Normalization for Unsupervised Domain Adaptation." In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7354-7362. 2019.
[2] Kumar, Abhishek, Prasanna Sattigeri, Kahini Wadhawan, Leonid Karlinsky, Rogerio Feris, Bill Freeman, and Gregory Wornell. "Co-regularized alignment for unsupervised domain adaptation." In Advances in Neural Information Processing Systems, pp. 9345-9356. 2018.


We conduct a large experimental comparison of various robustness metrics for image classification. The main question of our study is to what extent current synthetic robustness interventions ( p -adversarial examples, noise corruptions, etc.) promote robustness under natural distribution shifts occuring in real data. To this end, we evaluate 147 ImageNet models under 199 different evaluation settings. We find that no current robustness intervention improves robustness on natural distribution shifts beyond a baseline given by standard models without a robustness intervention. The only exception is the use of larger training datasets, which provides a small increase in robustness on one natural distribution shift. Our results indicate that robustness improvements on real data may require new methodology and more evaluations on test sets representing natural distribution shifts.

============================================ Update after rebuttal =========================================
I thank the authors for their detailed rebuttal. I'm in general satisfied with the authors' response to my concerns, so as promised, I'm happy to increase my score.
========================================================================================================
This paper considers the relationship between various measures of synthetic robustness and two distinct measures of natural robustness in large scale image classification models. The authors argue that the synthetic robustness measures considered in this paper are not predictive of natural robustness when the effect of baseline accuracy is subtracted. I think, if true, this is an important message that researchers in this area need to be aware of. However, I have a number of questions and concerns about the results. I would be happy to increase my score if the authors could address some of these issues:
1) Another important recent benchmark not mentioned in the paper is ImageNet-A (https://github.com/hendrycks/natural-adv-examples). I would encourage the authors to include this benchmark among their natural robustness measures (in addition to ImageNetV2 and ImageNetVidRobust). The advantage of this dataset is that because it samples from the error distribution of a high-performing ImageNet model, to a large extent, it already comes with the baseline subtracted, so it essentially obviates the need for the indirect effective robustness measure introduced here. The raw accuracies on ImageNet-A would be directly interpretable and they would also answer the question “why should we care?” in a more visceral way, because even the Instagram trained state-of-the-art ImageNet models seem to achieve a mere 17% accuracy on this benchmark: https://arxiv.org/abs/1907.07640
2) There seems to be a direct conflict between the main conclusion of this paper (that synthetic robustness measures do not predict natural robustness) and an opposite conclusion reached in an earlier paper (https://arxiv.org/abs/1904.10076) where the authors claim that robustness against synthetic perturbations like translation, hue, and saturation are actually highly predictive of video robustness (not sure if this would generalize to ImageNetV2). As far as I can see, these particular perturbation types are not included among the synthetic perturbations considered in this paper. Can you please clarify this discrepancy?
3) Relatedly, looking at the scatter plots of effective robustness vs. robustness against individual perturbations in the appendix, especially for the video robustness measure, some of the correlations seem to be pretty significant (for example, video robustness vs. jpeg compression robustness, p. 19). So, I am wondering to what extent the main conclusion of this paper might just be driven by the averaging of a large number of non-predictive perturbations and a smaller number of more predictive perturbations.
4) Also, ImageNet-P perturbations are not included in the paper. If the authors want to make their claims more reliable, I would encourage them to consider these among their synthetic perturbations as well. The translation perturbation in ImageNet-P, in particular, would be particularly important to consider for video robustness given the results from the arxiv pre-print mentioned in 2) above.


Inspired by the advances in biological science, the study of sparse binary projection models has attracted considerable recent research attention. The models project dense input samples into a higher-dimensional space and output sparse binary data representations after Winner-Take-All competition, subject to the constraint that the projection matrix is also sparse and binary. Following the work along this line, we developed a supervised-WTA model when training samples with both input and output representations are available, from which the optimal projection matrix can be obtained with a simple, efficient yet effective algorithm. We further extended the model and the algorithm to an unsupervised setting where only the input representation of the samples is available. In a series of empirical evaluation on similarity search tasks, the proposed models reported significantly improved results over the state-of-the-art methods in both search accuracy and running time. The successful results give us strong confidence that the work provides a highly practical tool to real world applications.

The paper takes the binary projection framework used for LSH-type applications (based on Dasgupta et al. 2017) and given training data from that framework, shows an efficient closed-form solution update for the "projection" matrix, and also derives an alternating minimization algorithm for when training data are not available (note that these are not training data in a typical supervised learning sense, i.e., they are not data features, but rather the output of a projection of the very specific type discussed in the paper).
The idea is interesting, and the recent literature suggest that the overall idea works well. This new paper seems to work much better than recent literature (and a lot faster than some), which is very encouraging. The derivation of the closed-form updates is clever and seems to be correct.  Overall, I am quite favorable for this paper.
My chief critical comment is that given the complexity of the model, is is feasible to compare to PCA-based projections as well. That is, usually PCA is not compared with these methods since a random projection method is faster. But this paper proposes data-dependent projection matrices, which is exactly what PCA does.  This paper has been implicitly assuming d << n and trying to avoid the O(n^2) cost of nearest neighbor search. Given that d << n, the complexity of the algorithm in the paper is, roughly (avoiding some logs), O((#iter) d^2 n ), since the constant "c" depends on d, and presumably d' also scales linearly with d. The (#iter) refers to the number of iterations of the alternating minimization. The actual constants are small, but it is still quadratic in d.  PCA takes O(d^2n + d^3) operations when d<n, so therefore PCA is of roughly the same complexity, and thus I'd expect you to compare to it in the numerical comparisons.
With a valid response to the issue of PCA, I would probably "strongly accept" this paper.
(Another note on comparisons: it seems Johnson-Lindenstrauss projections would be applicable, so I'd suggest running a Fast Johnson-Lindenstrauss (Ailon and Chazelle) to compare with also).
Minor notes:
- In section 3.3, how do you get complexities like d log(c) for finding the top c entries? Please cite an algorithm textbook. I might be rusty on this, but the algorithms I can think of to do this are O(  min( d log(d), d c )) operations. Same for the  d' log(k).
- A good number of sentences were grammatically incorrect or sounded awkward, e.g., paragraph 2, "people's much attention"; second sentence in section 2.2, the phrase "; while the rest get inhibited and remain silent" is not a complete phrase by itself (maybe change the semicolon to a comma). Overall, another round of proof reading would help.


We propose using active learning based techniques to further improve the stateof-the-art semi-supervised learning MixMatch algorithm. We provide a thorough empirical evaluation of several active-learning and baseline methods, which successfully demonstrate a significant improvement on the benchmark CIFAR-10, CIFAR-100, and SVHN datasets (as much as 1.5% in absolute accuracy). We also provide an empirical analysis of the cost trade-off between incrementally gathering more labeled versus unlabeled data. This analysis can be used to measure the relative value of labeled/unlabeled data at different points of the learning curve, where we find that although the incremental value of labeled data can be as much as 20x that of unlabeled, it quickly diminishes to less than 3x once more than 2,000 labeled example are observed.

Summarize the paper:
This paper proposes a method that can deal with an active-learning scenario for the recently proposed semi-supervised learning method: MixMatch.  More specifically, the proposed method considers uncertainty measures to choose samples and a diversification step to ensure diversity within the sampled batch.  For uncertainty measures, the paper considers the simple maximum confidence and the gap between two most likely classes.  Additional augmentation techniques inspired from MixMatch are used.  For diversification, a clustering method and an information density method are considered.  Furthermore, the paper proposes a cost analysis model to compare labeled and unlabeled samples.  Experiments demonstrate the behavior of the proposed method.
Pros of the paper:
- The experimental results seem to be strong and encouraging.
- The discussions on the cost of labeled and unlabeled samples seems to be an important contribution for semi-supervised active learning.
- The motivation and direction of the paper is simple and easy to follow: Take the state-of-the-art semi-supervised learning algorithm and propose an active-learning version of it.
- It is not a straightforward combination of MixMatch and active learning, and there are some specialized techniques such as “aug” used in the design of the proposed algorithm.
Cons of the paper:
- Only uncertainty based sampling methods are considered, but is this enough?  There seems to be no other papers that deal with active semi-supervised learning for a deep learning context, so it might be important to really explore the many sampling methods (e.g., from survey of Settles 2009).
- A more minor comment: The same issue goes for the semi-supervised learning side.  MixMatch is the state of the art in terms of accuracy for image domains, but it is an ensemble of several semi-supervised learning methods, and have strong assumptions, e.g., smoothness assumption, small distribution overlap, etc.  This will mean the proposed method will also have those strong assumptions and limits the method’s applicability.
- In experiments, it would be better to have figures that are usually used in active learning experiments, where the x-axis is the remaining budget and y-axis is the performance measure.
Additional comments:
Active learning methods gives labels to unlabeled samples in different epochs until the budget is used up, but it would be interesting to give the final labeled and unlabeled dataset after budget is used up as a fixed dataset, and then train the traditional passive MixMatch with this.  Then we can really compare the original MixMatch and active MixMatch.  If the proposed method still works better,  then the proposed method might be meaningful not only as an active learning method but also as a curriculum learning method.
********************
I would like to thank the authors for answering my questions and updating the paper, but would like to keep my score due to the 2nd point of the cons.  A minor comment on the second point of the author response:  The sharpening step in MixMatch can be regarded as an entropy minimization procedure, which I think is based on the assumption of low distribution overlap.


Decoding in autoregressive models (ARMs) consists of searching for a high scoring output sequence under the trained model. Standard decoding methods, based on unidirectional greedy algorithm or beam search, are suboptimal due to error propagation and myopic decisions which do not account for future steps in the generation process. In this paper we present a novel decoding approach based on the method of auxiliary coordinates (Carreira-Perpinan & Wang, 2014) to address the aforementioned shortcomings. Our method introduces discrete variables for output tokens, and auxiliary continuous variables representing the states of the underlying ARM. The auxiliary variables lead to a factor graph approximation of the ARM, whose maximum a posteriori (MAP) solution is found exactly using dynamic programming. The MAP solution is then used to recreate an improved factor graph approximation of the ARM via updated auxiliary variables. We then extend our approach to decode in an ensemble of ARMs, possibly with different generation orders, which is out of reach for the standard unidirectional decoding algorithms. Experiments on the text infilling task over SWAG and Daily Dialogue datasets show that our decoding method is superior to strong competing decoding methods.

I think this is a good study, unless I miss something. It proposes a new solution to one of the fundamental problems in the field, which significantly improves previous solutions in terms of accuracy. I will recommend it for acceptance unless I miss something. (I’m not an expert in the field and the problem seems to be so fundamental, making me cautious about judging the novelty of the proposed solution.)
There seems to be some unclarity about the optimization algorithm. In short, I suspect that the proposed optimization has some difficulty with its convergence.
-	In Sec.4, the authors suggest “a two-step block coordinate descent algorithm to alternate between (i) optimizing y’s while g’s are fixed, and (ii) optimizing g’s while y’s are fixed”. They further suggest `a variant of the Viterbi algorithm’ for (i) and gradient-based algorithms for (ii). These seem to make sense to me so far.
-	Then, in Sec.6, they state that “We use the Nesterov optimizer with a learning rate of 0.1. All \mu’s are initialised with 0.5 and are multiplied by 1.2 after 5 epochs”. I think this needs some explanation.
-	For instance, what is the thought behind the choice of the optimizer? Why was Nesterov’s acceleration necessary instead of plain GD? Is it essential to use the scheduled adjustment of \mu?
-	It states “All \mu’s” but there seems only a single \mu in Eq.(5). Am I missing something?
-	 Some explanation on what “Nesterov optimizer” and \mu would also be necessary for a wide range of readers.
-	There is a statement on the initialization of the optimization in p.7: “h corresponding to the beam search solution to initialize the g variables”. How sensitive is the optimization to the initial values? For instance, will the results change if g’s are initialized by the solution of the greedy method.
-	There is a plot without a figure caption in p.8, which shows how the objective cost decreases with parameter updates. Why are their values at ‘epoch’=0 lower than those at the subsequent epochs? Does this mean the initial values give more optimal parameters?
Additional comments:
-	It would be more friendly to the readers to show the definition of the notations like y^n_1.
-	There is a statement like “Decoding was run for 10 epochs.” I suppose epochs here mean iteration count of the alternated parameter updates in the proposed algorithm. Why do the authors call it `epoch’?
-	No figure caption for the plot in p.8. It also uses `epochs’ for the horizontal axis title. No axis title for the vertical axis.


This paper handles a challenging problem, unseen attribute-object pair recognition, which asks a model to simultaneously recognize the attribute type and the object type of a given image while this attribute-object pair is not included in the training set. In the past years, the conventional classifier-based methods, which recognize unseen attribute-object pairs by composing separately-trained attribute classifiers and object classifiers, are strongly frustrated. Different from conventional methods, we propose a generative model with a visual pathway and a linguistic pathway. In each pathway, the attractor network is involved to learn the intrinsic feature representation to explore the inner relationship between the attribute and the object. With the learned features in both pathways, the unseen attribute-object pair is recognized by finding out the pair whose linguistic feature closely matches the visual feature of the given image. On two public datasets, our model achieves impressive experiment results, notably outperforming the state-of-the-art methods.

The paper attempts to tackle unseen object-attribute recognition in still images. This follows a line of works that addresses such a problem using embedding (with operators), classification and generative approaches. The paper makes use of the visual attractor network, extracting different visual representations for objects and attributes. A collection of losses (some regularisation and some distance in embedding space) are proposed. I believe the paper (except A below) is technically correct.
This paper was particularly tricky to read. While convincing at parts, there are two worrying statements. One I currently believe (pending the authors' response) is a scientific flaw (A), the other causes the results to be potentially incomparable to published works (B). I detail these next.
(A) Scientific Flaw: Quoting from the paper, at inference (i.e. for a test image), "we propose a voting inference method. For an input image, its visual attractor feature AV is computed, ... By computing the L2 distance ... the pair that corresponds to the minimum distance is taken as the initial attribute-object pair recognition for the input image. For the images belonging to the same pair, we use the recognitions for all images to vote a pair label to be the final attribute-object pair recognition."
In my interpretation, the authors are taking all test images that are labeled using the same pair (e.g. 'sliced-apple' - assuming this is an unseen pair). They then use all these to make a final decision for all these images.
I believe my interpretation is correct because their statement "images belonging to the same pair" can only exist in testing for their target scenario (unseen object-attribute pair).
If my interpretation is correct, this is A MAJOR FLAW in the approach. How do the authors know that a collection of test images "belong to the same pair"?? This knowledge can only be acquired from labels of the test set and cannot be used as part of any algorithm.
I would very much like for my interpretation to be naive, but I could not find a different explanation for this statement in page 6. The authors note in the conclusion that this voting approach is part of their contribution.
(B) In reporting the results, the authors state that: "Using the grid-search method, we set parameters \alpha; ... [5 parameters]  in Eq.17 to be 1:0; 2:0; 6:0; 3:0; 2:0 for the MIT-States dataset and 1:0; 3:0; 6:0; 1:0; 2:0 for the UTZappos50K
dataset. Attractor parameters cv; cl; Tv; Tl are set as 0:1; 0:2; 20; 10 for the MIT-States dataset and 0:08; 0:16; 25; 15 for the UT-Zappos50K dataset."
This assumes that the authors are grid-searching all parameters, and evaluating the test set performance for each case, then reporting the maximum possible test performance per dataset. The authors do not reference a validation set they use to set these parameters instead. Importantly, the values are SIGNIFICANTLY different for each test set and there is no explanation of why the parameters vary largely. It is not clear how the performance on an new dataset would be,
Up to my knowledge, the other methods that they compare to in Table 1, have not reported a grid-search over the parameter space for maximum performance of the test set. This IMO makes the comparative evaluation, which the authors deem to be 'impressive' quite unfair. It is not possible to assess the method's performance using these results.


Consistency regularization describes a class of approaches that have yielded ground breaking results in semi-supervised classification problems. Prior work has established the cluster assumption -under which the data distribution consists of uniform class clusters of samples separated by low density regions -as key to its success. We analyze the problem of semantic segmentation and find that the data distribution does not exhibit low density regions separating classes and offer this as an explanation for why semi-supervised segmentation is a challenging problem. We then identify the conditions that allow consistency regularization to work even without such low-density regions. This allows us to generalize the recently proposed CutMix augmentation technique to a powerful masked variant, CowMix, leading to a successful application of consistency regularization in the semi-supervised semantic segmentation setting and reaching state-of-the-art results in several standard datasets.

# Summary
This paper proposes a method for semi-supervised semantic segmentation.
The authors tackle this problem through consistency regularization, a successful technique in image classification, that encourages the network to give consistent predictions on unlabeled samples which are perturbed in multiple ways. The authors argue that the cluster assumption (to which effectiveness of consistency regularization has been partially attributed) does not hold in semantic segmentation. Thus, in order to enable class boundaries to become low-density regions and then better guide contrastive regularization, the authors argue that a stronger perturbation must be inserted. To this effect they first propose looking at CutOut and CutMix types of methods. They improved upon them by putting forward a variant of CutMix, coined CowMix, with more degrees of freedom and using flexible masks instead of rectangular ones. CowMix is evaluated on the Cityscapes and PascalVOC 2012 datasets in the semi-supervised regime and showing encouraging results.
# Rating
I find the paper and the advanced ideas of interest for the community and I consider they are novel. I'm currently on the fence between Weak Accept and Weak Reject, mostly due to incomplete evaluations and support for claims made in the introduction regarding the infeasibility of contrastive regularization methods for semantic segmentation. I would be happy to upgrade my rating if authors addressed these concerns.
# Strong points
- The paper is well written and mostly clear with a good coverage and positioning w.r.t. related work. The authors illustrate well the reasoning and the choices they have made. The author provide plenty of ablation studies (e.g., per class statistics) and implementation details, improving significantly the reproducibility of the contribution.
- The flexible masking technique that is advanced here is novel and experimentally seems effective.
- This work is among the few that address semi-supervised semantic segmentation in a non-adversarial manner, so I would give it some novelty credit.
- I appreciate the evaluation protocol of averaging across multiple runs.
# Weak points
## Unclear aspects
- The authors argue that consistency regularization has had little success so far in semantic segmentation problems since low density regions in input data do not align well with class boundaries. It would be useful to provide a reference to this claim or at least validate it experimentally on a large dataset.
- In Figure 1, it is not clear on which features where the distances between patches computed? Is it on raw pixels or intermediate feature maps from a CNN?
If the distances are made over raw pixels, I find it difficult to make the connection between distances in the pixel space and distances in the class space.
Are the neighbor patches overlapping with the central/query patch?
## Experiments
- The authors compare against other methods on the CamVid dataset. CamVid is a small and relatively limited dataset (~367 images for training from the streets of Cambridge). I'm worried that this dataset might not be enough to conclude and emphasize the benefits of this method over other semi-supervised techniques. For instance CowOut does not seem do be above CutOut, while CutMix has convergence problems and low scores.
The other experiments on Cityscapes and Pascal VOC are certainly interesting, but the method is compared only against Hung et al. which a different family of methods and the subset baseline (which is useful but not enough). I think this work would benefit from an additional baseline in the style of contrastive regularization methods, e.g. ICT, and eventually CutOut, to support the initial arguments regarding the limitations of these methods in semantic segmentation and respectively the effectiveness of the flexible masks over the rectangular ones in this setup.
# Suggestions for improving the paper:
1) It would be useful to include other semi-supervised baselines, e.g. ICT, and the baseline perturbation CutMix on larger experiments, in order to better emphasize the contributions of this work.
2) Did the authors try the flexible masking on image classification? How is it expected to perform over ICT, MixUp or MixMatch?


While progress has been made in crafting visually imperceptible adversarial examples, constructing semantically meaningful ones remains a challenge. In this paper, we propose a framework to generate semantics preserving adversarial examples. First, we present a manifold learning method to capture the semantics of the inputs. The motivating principle is to learn the low-dimensional geometric summaries of the inputs via statistical inference. Then, we perturb the elements of the learned manifold using the Gram-Schmidt process to induce the perturbed elements to remain in the manifold. To produce adversarial examples, we propose an efficient algorithm whereby we leverage the semantics of the inputs as a source of knowledge upon which we impose adversarial constraints. We apply our approach on toy data, images and text, and show its effectiveness in producing semantics preserving adversarial examples which evade existing defenses against adversarial attacks.

Major caveat: I have published in the area of adversarial attacks on NLP models, but the specifics of the methods presented in this paper are quite outside of my expertise, and I do not have time to become familiar with them for this review.  I hope there are other reviewers that are more qualified than I am to check the specifics of the methods.
This paper presents a new technique for generating adversarial examples, by first learning the data manifold in an embedding space, then finding an adversarial example that lies on the manifold.  I like this idea, it intuitively seems like a promising method for obtaining semantically meaningful adversarial examples.
As I said above, I do not feel qualified to review whether the method should _theoretically_ accomplish its goals, so my judgment of this paper is on the intuition behind the idea (which I like), and the results that I can see (which are less promising).  In order to have a "semantics preserving" attack, the method needs to (1) remain on the data manifold, and (2) not change the label a human would give to the input.
For (1), this appears to have been accomplished on most datasets, though it seems pretty hard to argue that the artifacts seen in the MNIST examples shown are on the data manifold - there are no such artifacts in any of the inputs, or in the clean reconstruction.  How do the authors claim that this actually did a reasonable job of staying in the data manifold?
For (2), most of the images do indeed look like they should retain their human labels, which is good (but also not hard for adversarial images).  Almost all of the textual examples, however, have correct predictions from the model after the adversarial change to the input.  You can't really argue that these are "semantics preserving", or even "successful attacks", as they change the expected input label.  This is why semantics-preserving attacks are so hard in NLP, and I don't think that this method has accomplished its goal here at all, at least for text.  The authors should consult with experts in NLP before making claims about successfully constructing semantics preserving attacks on NLP models.
I'm pretty on the fence about this paper, as I like the intuition, and the method appears to work reasonably well for vision.  It does not work as claimed for text, however, and that should be fixed before this paper is published (either with softened claims or with better results).  Hopefully people from other perspectives can pipe in and give a more clear picture on this paper.
EDIT: See discussion below for my justification for reducing my score from a 3 to a 1.
EDIT 11/14: The authors' revisions have satisfied my concerns about how the NLP attacks are described.  I'm a little bit nervous about how the examples were changed - it seems that nothing changed about the method itself, so the authors probably cherry-picked better examples - but that's not sufficiently worrying to me to justify rejection.  The pilot study is also quite weak, as the number of inputs that were evaluated was only 20, and the questions presented don't appear to ask about changes in the label.  I don't know how you could get 100% on that given the examples that I saw in the previous version of the paper.  This is all to say that I don't think the NLP attacks are actively problematic anymore, as they were previously, now they are just weak.  The main contribution here is the technical contribution, anyway, so weaker results on one of the datasets tested is not a deal-breaker to me.  Assuming the technical contributions pass muster (which, as I said, I don't really feel qualified to judge), I'm satisfied with this paper as it is now.


We propose an implementation of GNN that predicts and imitates the motion behaviors from observed swarm trajectory data. The network's ability to capture interaction dynamics in swarms is demonstrated through transfer learning. We finally discuss the inherent availability and challenges in the scalability of GNN, and proposed a method to improve it with layer-wise tuning and mixing of data enabled by padding.

Review for "Towards Scalable Imitation Learning For Multi-Agent Systems with Graph Neural Networks".
The paper proposes a new time series model for learning a sequence of graphs.
I vote to reject the paper for three reasons.
1. Lack of significance. Algorithm 1 is essentially supervised learning of an autoregressive model with a GNN. The GNN definition is expanded out in the pseudocode, but seems to be completely standard. Moreover, Section 2.5 is also completely standard - you are using a sum of standard MSE losses. It should be much more concise. Also, there is typically no point scaling the loss by D if you use Adam (because the scaling they will eventually cancel out).
2. Poor awareness of prior work. What you call "scalability issue caused by poor extrapolation" is normally called poor generalization. Improving generalization is an established problem in ML. You should cite some of that work. See [1] and follow the references.
3. Poor experimental evaluation. Your experiments show that the competing GNN method (what you call Kipf's GNN) didn't converge in certain settings. Robust GNN implementations exist that converge for a wide range of reasonable graphs. You should have used one of them.
As things stand, because the criticisms concern the core of the submission, I have doubts the quality of the paper can be improved enough within the ICLR revision time to obtain an "accept" score. However, I wanted to encourage you not to give up. The building blocks that you have in the paper are very relevant and can be a basis for impactful work. You also write well (despite some minor issues). I hope these skills will help you make great submissions in the future!
Minor points:
1. I would not call what the paper is doing "imitation learning". Imitation learning normally means you have a controllable system and want to learn a policy from expert demonstrations. What this paper does is commonly referred to as learning an autoregressive time series model.
2. Avoid colorful language. For example, the sentence: "the apparent resemblance of the corrected distributions in the final output may inspire an inductive bias as part of training objective" is unclear.
[1] https://papers.nips.cc/paper/7176-exploring-generalization-in-deep-learning.pdf


The formal understanding of deep learning has made great progress based on quantum many-body physics. For example, the entanglement entropy in quantum many-body systems can interpret the inductive bias of neural network and then guide the design of network structure and parameters for certain tasks. However, there are two unsolved problems in the current study of entanglement entropy, which limits its application potential. First, the theoretical analysis of entanglement entropy was only investigated in the representation of a single object (e.g., an image or a sentence), but has not been well studied in the matching of two objects (e.g., question-answering pairs). Second, the entanglement entropy can not be qualitatively calculated since the exponentially increasing dimension of the matching matrix. In this paper, we are trying to address these two problem by investigating the fundamental connections between the entanglement entropy and the attention matrix. We prove that by a mapping (via the trace operator) on the high-dimensional matching matrix, a low-dimensional attention matrix can be derived. Based on such a attention matrix, we can provide a feasible solution to the entanglement entropy that describes the correlation between the two objects in matching tasks. Inspired by the theoretical property of the entanglement entropy, we can design the network architecture adaptively in a typical text matching task, i.e., question-answering (QA) task. Our adaptive strategy sets a new state-of-the-art performance in TREC-QA task, with 2.9% absolute improvement over a recent model.

In this paper, a method for leveraging entanglement entropy for understanding attention matrices is proposed. Specifically, the paper aims at solving two problems: 1) to study the theoretical analysis of entanglement entropy for the matching of two objects (question-answering pairs), and (2) to qualitatively calculate the matching matrix. The introduced approach is based on fundamental connections between the entanglement entropy and the attention matrix. The main goal of the paper is to show that a low-dimensional attention matrix can be derived from a high-dimensional matching matrix. Results are shown for a text matching task on two datasets (TREC-QA, YAHOO-QA).
My main concerns with this paper are that the approach is not well described and that the proposed contribution appears quite narrow. Overall it is not clear what the actual contribution of this work really is. Several sections of the paper appear convoluted and could be more concise; the text is often written unnecessarily complicated which makes it hard to follow and to comprehend details. While the experiments seem sound the overall improvement of 2.9% compared to SOTA work appear rather shallow. Therefore, in its current state I cannot recommend accepting this paper.
Detailed comments:
- The abstract of the paper is very cryptic and the motivation for the proposed approach is not sufficiently described.
- The paper is not well-written and difficult to follow. Several sentences and sections are left too unclear. E.g. in the introduction sentences like "... but such an indicator only reflects the intricate correlation structures of each single input object (e.g., an image or a text)" or "This is due to the fact that the tensor product occurs in the quantum many-body function for representing the image and text " are confusing as they come without much context.
- It is not clear what is meant by "matching problem".
- A few sentences in the introduction are exact copies of the abstract, which makes the text appear redundant. It would help to clearly state what the contribution of this work is, instead of repeating sentences.
- It is not clear what is meant by "relatively-deeper" and "relatively-shallower" layers.
- The text shows several spelling and grammar issues (such as "for the more complex the inputs").
- Several sentences don't make sense and are difficult to read (e.g. "Since our work is mainly for the text matching task of a sentence pair, we briefly introduce a recent Quantum Many-body Wave Function inspired Language Modeling").
- The notation and equations in Section 2 are mostly common knowledge and could be moved to the appendix.
- In Section 3.1. what is meant by "subsystem in deep neural networks"? Later in the text in becomes more clear. So it would be helpful to rearrange the text.
- "probability amplitude distributions " is not clear.
- Sentences like "which often correspond to the important information hidden in the matrix" should be accompanied with a reference.
- Section 3.2 is titled "Network Design Based on Entanglement Entropy", but the section does not actually describe a network architecture, but instead just describes how to obtain the attention matrix and the sample differences.
- Section 4.1 (first paragraph): it would help to discuss the connection of many-body wave functions to represent questions and answer sentences as two subsystems more clearly and earlier in the text.
- Section 4.2 (second paragraph) appears quite repetitive. Many sentences have been used in previous sections of the text.
- The results and discussions shown in Sections 4.4 and 4.5 are interesting and seem sound. However, it is not clear what exactly are the "adaptive settings for kernels" (e.g. in Figure 2)
- Limitations of the approach are not sufficiently discussed.
- It is not clear what is meant by "we will investigate the entanglement entropy under high-order conditions" in the conclusion.


A key component of most neural network architectures is the use of normalization layers, such as Batch Normalization. Despite its common use and large utility in optimizing deep architectures, it has been challenging both to generically improve upon Batch Normalization and to understand the circumstances that lend themselves to other enhancements. In this paper, we identify four improvements to the generic form of Batch Normalization and the circumstances under which they work, yielding performance gains across all batch sizes while requiring no additional computation during training. These contributions include proposing a method for reasoning about the current example in inference normalization statistics, fixing a training vs. inference discrepancy; recognizing and validating the powerful regularization effect of Ghost Batch Normalization for small and medium batch sizes; examining the effect of weight decay regularization on the scaling and shifting parameters γ and β; and identifying a new normalization algorithm for very small batch sizes by combining the strengths of Batch and Group Normalization. We validate our results empirically on six datasets: CIFAR-100, SVHN, Caltech-256, Oxford Flowers-102, CUB-2011, and ImageNet.

The authors discuss four techniques to improve Batch Normalization, including inference example weighing, medium batch size, weight decay, the combination of batch and group normalization. Equipped with the proposed techniques, the authors obtain promising results when training deep models with various batch sizes. However, the novelty of this paper seems very limited and more experiments are required.
Please see my detailed comments below.
Positive points：
1. The proposed inference example weighing method yields promising results and does not require any re-training.
2. The combination of batch and group normalization makes it possible to train deep models with very small batch size.
3. By combining all the techniques, the proposed method yields promising performance when training deep models with different batch sizes.
Negative points:
1. Some notations are very confusing. For example, the authors use B to represent the size of a minibatch. However, why do the authors only consider B-1 samples in Eq. (2), i.e., selecting the minimum possible output among x_1, …, x_{B-1}?
2. The proposed inference example weighing method seems very similar to Batch Renormalization. Both methods seek to use a linear function to combine the batch statistics and the moving statistics. What is the essential difference between these methods?
3. What model do the authors use in the experiment of Figure 2? Why do the authors conduct experiments on different datasets in Section 3.1 (including ImageNet) and Section 3.2 (excluding ImageNet)? It would be stronger to provide ImageNet results in Section 3.2.
4. The authors draw different conclusions about the usage of weight decay from a recent work (He et al, CVPR2019). The CVPR paper reports that training \gamma and \beta without weight decay on ResNet-50 yields significant performance improvement. However, this paper shows that training ResNet-50 with weight decay improves the performance. Please comment on the differences in the conclusions.
Reference: "Bag of tricks for image classification with convolutional neural networks." CVPR, 2019.
5. The authors only report ImageNet results of the proposed inference example weighing method. However, all the experiments in Section 4 are performed on three small datasets. It is necessary and important to provide ImageNet results to show the effectiveness of the other three techniques in Section 4.
6. Note that training deep models with non-i.i.d. minibatches is a typical case to evaluate normalization methods, e.g., Batch Renormalization. Specifically, examples in a minibatch are not sampled independently. What would happen if the authors apply the proposed techniques to the non-i.i.d. case?
7. Some closely related work should be discussed in the paper, such as
[1] "Decorrelated Batch Normalization." CVPR, 2018.
[2] "Double Forward Propagation for Memorized Batch Normalization." AAAI, 2018.
[3] "Differentiable Dynamic Normalization for Learning Deep Representation." ICML, 2019.
[4] "Iterative Normalization: Beyond Standardization towards Efficient Whitening." CVPR, 2019.
Minor issues:
1. In Section 1, the third contribution is not a complete sentence.
2. There are many typos in the paper.
(1) In Section 2, “Layer Normalization, which has found use in many natural language processing tasks.” Should “which has found use” be “which has been used”?
(2) In Section 3.1, “Batch Normalization has a disparity in function between training inference”. “between training inference” should be “between training and inference”.
(3) In Section 3.1, “we need only figure out …” should be “we only need to figure out …”


Current model-based reinforcement learning approaches use the model simply as a learned black-box simulator to augment the data for policy optimization or value function learning. In this paper, we show how to make more effective use of the model by exploiting its differentiability. We construct a policy optimization algorithm that uses the pathwise derivative of the learned model and policy across future timesteps. Instabilities of learning across many timesteps are prevented by using a terminal value function, learning the policy in an actor-critic fashion. Furthermore, we present a derivation on the monotonic improvement of our objective in terms of the gradient error in the model and value function. We show that our approach (i) is consistently more sample efficient than existing state-of-the-art model-based algorithms, (ii) matches the asymptotic performance of model-free algorithms, and (iii) scales to long horizons, a regime where typically past model-based approaches have struggled.

#rebuttal responses
Thanks for the clarification!  However, I will keep my original score for these reasons:
(1) Only 3 random seeds are used for each environment, which is not convincing as the variance of MAAC is large in some figures.
(2) Baselines are only trained with 10^5 steps and do not converge. Thus it not fair to say that MAAC matches the asymptotic performance of model-free algorithms.
#review
This paper constructs a model-based policy optimization algorithm (MAAC) that uses the pathwise derivative of the learned model and policy across future timesteps. The terminal value function is used to improve stability.  The theoretical guarantee of the error of the model-based gradient is presented. Experimental results show that MAAC outperforms SAC, STEVE, and MBPO on four environments in terms of the sample efficiency.
The experimental results are strong and I appreciate the plots of the gradient error in a simple task, shown in Figure 3. But I want to see the comparison of the final performance of each algorithm in these environments, and I doubt that the baselines do not converge.
However, the paper is badly written. First of all, the authors claim that the pathwise derivate method is applied to optimize the objective function. But the detail of the method is missing. Secondly, I can not follow the procedure of how Q function is learned in the MAAC algorithm. Thirdly, Theorem 4.1 gives a performance improvement bound of the new policy w.r.t J_pi without the entropy term. But the MAAC algorithm optimizes the objective with the policy entropy term.
Also, MAAC applies many techniques in other papers. The paper does not clearly show the advantage of each component:
(1) I want to see the experimental results of MAAC optimizing the objective without the entropy.
(2) The SVG(1) algorithm should be compared as a baseline as the SVG(1) also uses the gradient of the learned model to optimize the policy.
(3) The policy and the Q function are optimized on both the real samples and the generated samples. I want to see the ablation study or justification on whether training on real samples or both samples.
Questions:
1. How many independent runs are used in experiments?
2. Does the computation of the pathwise derivate method cost much time？ Is MAAC much slower than SAC?
I am happy to increase my score if the authors justify these questions.