Some fundamental limitations of deep learning have been exposed such as lacking generalizability and being vunerable to adversarial attack. Instead, researchers realize that causation is much more stable than association relationship in data. In this paper, we propose a new framework called do-calculus AutoEncoder(DO-AE) for deep representation learning that fully capture bivariate causal relationship in the images which allows us to intervene in images generation process. DO-AE consists of two key ingredients: causal relationship mining in images and intervention-enabling deep causal structured representation learning. The goal here is to learn deep representations that correspond to the concepts in the physical world as well as their causal structure. To verify the proposed method, we create a dataset named PHY2D, which contains abstract graphic description in accordance with the laws of physics. Our experiments demonstrate our method is able to correctly identify the bivariate causal relationship between concepts in images and the representation learned enables a do-calculus manipulation to images, which generates artificial images that might possibly break the physical law depending on where we intervene the causal system.

Thank you for your submission.
- What is the specific question/problem tackled by the paper?
The paper proposes a VAE architecture to learn causal relations and allow for interventions. The architecture requires knowledge of the causal graph, and the direction of the causal arrows are inferred by comparing the log-likelihoods of generated images. The architecture may also require knowledge that an arrow exists between two vertices. This relies on the principle that "low-capacity" neural networks can predict better along the causal arrows (with the cause as input and the effect as the output) than in the opposite direction (with the effect as input and the cause as the output).
The paper focuses on the graph (A, B) where one wants to understand whether A causes B, or B causes A. The paper also discusses intervening in this graph.
The paper uses a new dataset for evaluating the approach, based on simple Newtonian systems.
- Is the approach well motivated, including being well-placed in the literature?
I think the motivation is adequate, but the review of the literature glosses over related work (or the absence thereof) in predicting the direction of arrows in causal graphs. The comparison of the proposed dataset against existing ones is missing.
- Does the paper support the claims? This includes determining if results, whether theoretical or empirical, are correct and if they are scientifically rigorous.
The procedure for determining whether A causes B (or B causes A) is qualitative. The paper demonstrates that the performance gap between the correct and incorrect explanations is consistently distinguishable across multiple experiments.
Visual inspection of the generated images is also used for assessing the quality of the models.
Because the results are qualitative, the support for the claims is not as strong as it could be (with quantitative results).
- Summarize what the paper claims to do/contribute. Be positive and generous.
The paper has two main contributions:
* Evidence to the Independent Mechanism principle (in a setting different from Bengio et al.'s transfer setup).
* A new dataset for evaluating learning causal arrows (with accessible ground-truths).
I think these are interesting contributions.
- Is the paper clearly written?
The paper has a number of grammatical errors that should be fixed.
The explanation of how the latent interventions are made is important and should be included.
- Clearly state your decision (accept or reject) with one or two key reasons for this choice.
I vote for a weak accept.
- Provide supporting arguments for the reasons for the decision.
I trust that the writing issues will be addressed in due course, but I am also concerned about the fact that evaluations are qualitative. The qualitative results provide support for the contributions that could be strengthened.
The dataset is also an interesting contribution and it is a good idea to give it visibility. For this, though, it is important that the paper assess its strengths and limitations in comparison to alternative datasets.
- Provide additional feedback with the aim to improve the paper. Make it clear that these points are here to help, and not necessarily part of your decision assessment.
I am not convinced that mentioning Kolmogorov complexity is an efficient use of the space. I think the content could be improved by making the motivation section more concise and adding a few more experimental results or discussion.
Which discussions would be good to have? I think it should be noted that the intervention on effect should behave as demonstrated (creating implausible scenarios). Also some more development on the spring example: What is the right causal graph for it, and can the arrows in that graph be learned?
Quantitative results would also improve the paper. Maybe decide between A->B or A<-B based on a statistical test?
You give an example about elephant-grassland association. Please cite a source for that.
Suppose that both likelihoods for A->B and B<-A are about the same. How do you decide if your model is too rich, or if there's no relationship? (This is an important question to understand if the method requires knowledge that an (A,B) arrow exists or not.)
The panels in Figure 5 do not support the claim. The simple net gets better at the cause, but in some cases the rich representation does a better job at the effect.
I think the physics dataset is also a contribution, so its originality & impact should be discussed in comparison to related work. Why is this an adequate benchmark? How does it address limitations of other benchmarks that could be used to evaluate proposed solutions for the problem in question?
In summary, my suggestions for improving the paper are:
1) Make sure & demonstrate (by adequate discussion of related work) the originality of the contributions:
1.1) The method for detecting the direction of causal arrows.
1.2) The dataset as a benchmark for the problem being studied.
2) Report quantitative results across the dataset and maybe across multiple setups for each name/physical law, with good coverage. You may consider a test set where the parameters are within the sampling range of your training set, and also outside the sampling range (where success of the method would be even more interesting).


Quantum machine learning methods have the potential to facilitate learning using extremely large datasets. While the availability of data for training machine learning models is steadily increasing, oftentimes it is much easier to collect feature vectors that to obtain the corresponding labels. One of the approaches for addressing this issue is to use semi-supervised learning, which leverages not only the labeled samples, but also unlabeled feature vectors. Here, we present a quantum machine learning algorithm for training Semi-Supervised Kernel Support Vector Machines. The algorithm uses recent advances in quantum sample-based Hamiltonian simulation to extend the existing Quantum LS-SVM algorithm to handle the semi-supervised term in the loss, while maintaining the same quantum speedup as the Quantum LS-SVM.

This paper proposes to extend a quantum-computing based solution of least-squares support-vector-machine to include use of unlabeled samples.  The formulation is analogous to the classical-computing case, in which semi-supervised learning introduces an additional term in the system of equations, which the authors show how to compute in the quantum setting without degrading big-O complexity.
I would lean toward rejecting this paper,  primarily on account of how the contribution relates to the publication venue.  The primary contribution lies in the procedure for preparing and propagating the quantum mechanical states needed to compute on the additional term.  Although the application is machine learning, the technique itself is still rather afar from this topic and would not appear to be of general benefit to conference-goers outside of quantum computing.  The overwhelming majority of quantum machine learning references in this paper appear in physics journals (all but one, which was ICML 2019).  Most of this paper is background material, which yet remains inadequate to convey insights into design decisions in the details of their main contribution, the derivations in section 3.2.  (I have a background in physics but not quantum computing.)
Perhaps a paper organization more amenable to this venue would be to shift some of the lengthier equations into an appendix and use the space of the paper to discuss a more conceptual and contextual understanding of why this technique is desirable, at each step, relative to other possible quantum techniques.  For example, Figure 1 is not explained, and is not decipherable to someone outside the field, so doesn't itself add to the story.
Could be really good work, but the presentation doesn't quite come across.
EDIT:  See comments to do with paper revision, which significantly improved the presentation.


Understanding the representational power of Deep Neural Networks (DNNs) and how their structural properties (e.g., depth, width, type of activation unit) affect the functions they can compute, has been an important yet challenging question in deep learning and approximation theory. In a seminal paper, Telgarsky highlighted the benefits of depth by presenting a family of functions (based on simple triangular waves) for which DNNs achieve zero classification error, whereas shallow networks with fewer than exponentially many nodes incur constant error. Even though Telgarsky's work reveals the limitations of shallow neural networks, it doesn't inform us on why these functions are difficult to represent and in fact he states it as a tantalizing open question to characterize those functions that cannot be well-approximated by smaller depths. In this work, we point to a new connection between DNNs expressivity and Sharkovsky's Theorem from dynamical systems, that enables us to characterize the depth-width trade-offs of ReLU networks for representing functions based on the presence of a generalized notion of fixed points, called periodic points (a fixed point is a point of period 1). Motivated by our observation that the triangle waves used in Telgarsky's work contain points of period 3 -a period that is special in that it implies chaotic behaviour based on the celebrated result by Li-Yorke -we proceed to give general lower bounds for the width needed to represent periodic functions as a function of the depth. Technically, the crux of our approach is based on an eigenvalue analysis of the dynamical systems associated with such functions.

In tackling a curious construction by Telgarsky regarding a certain class of functions that can be represented by deep networks (but not shallow networks (unless those shallow networks have exponentially many units)), the authors derive depth-width tradeoff conditions for when relu networks are able to represent periodic functions using dynamical systems analysis.
This paper was a delight to read.  I particularly enjoyed the motivating examples, and the clean exposition of Sharkovsky's theorem.  This result seems to cleanly answer the open question originally posed by Telgarsky, and the proofs are cleanly written, and correct to my (admittedly not perfect) knowledge.  I strongly suggest acceptance.
Questions/comments:
1. Could the author speculate on how the introduction of a bias term might affect their lower bound?  Presumably, this breaks the cleanness of the characteristic polynomial for $A$, but perhaps there are limits where it's still tractable?  This analysis certainly isn't necessary for publishing--I'm simply curious.
2. Could the authors provide some guiding intuition for the sharpness of their lower bound? (possibly on a synthetic dataset?) . I'm particularly imagining a plot that literally shows "classification error" versus "depth" for some fixed task.  While this is certainly a strong theoretical result, it would be nice to be able to contextualize how this result actually shines for a "real" model (and would help me believe the result "in my gut" so to speak).


Graph matching aims to establishing node-wise correspondence between two graphs, which is a classic combinatorial problem and in general NP-complete. Until very recently, deep graph matching methods start to resort to deep networks to achieve unprecedented matching accuracy. Along this direction, this paper makes two complementary contributions which can also be reused as plugin in existing works: i) a novel node and edge embedding strategy which stimulates the multihead strategy in attention models and allows the information in each channel to be merged independently. In contrast, only node embedding is accounted in previous works; ii) a general masking mechanism over the loss function is devised to improve the smoothness of objective learning for graph matching. Using Hungarian algorithm, it dynamically constructs a structured and sparsely connected layer, taking into account the most contributing matching pairs as hard attention. Our approach performs competitively, and can also improve state-of-the-art methods as plugin, regarding with matching accuracy on three public benchmarks.

The authors proposed a new way to train graph siamese networks for the graph matching problem. The overall framework of this paper is somehow similar to [1] and [2], except for the final Hungarian attention module, which is the key contribution of this paper. In the current settings, the authors proved that using their Hungarian attention module, the performance can be improved. However, it would be better if the Hungarian attention can be applied to DGCNN in [1] and CMPNN in [2]. It would be good if the author can do an extra experiment to apply the Hungarian attention module to these two modules. Also, the authors may want to add some discussion about these two papers in the related works section (both papers do have published their codes).
In the current experiment settings, both visual and geometric feature is used. Is it possible for the module to only using geometric features as [1] and [2]?
[1] Wang, Yue, and Justin M. Solomon. "Deep Closest Point: Learning Representations for Point Cloud Registration.", ICCV 2019,
[2] Zhen Zhang, and Wee Sun Lee. "Deep Graphical Feature Learning for the Feature Matching Problem.", ICCV 2019


The consistency loss has played a key role in solving problems in recent studies on semi-supervised learning. Yet extant studies with the consistency loss are limited to its application to classification tasks; extant studies on semi-supervised semantic segmentation rely on pixel-wise classification, which does not reflect the structured nature of characteristics in prediction. We propose a structured consistency loss to address this limitation of extant studies. Structured consistency loss promotes consistency in inter-pixel similarity between teacher and student networks. Specifically, collaboration with CutMix optimizes the efficient performance of semi-supervised semantic segmentation with structured consistency loss by reducing computational burden dramatically. The superiority of proposed method is verified with the Cityscapes; The Cityscapes benchmark results with validation and with test data are 81.9 mIoU and 83.84 mIoU respectively. This ranks the first place on the pixel-level semantic labeling task of Cityscapes benchmark suite. To the best of our knowledge, we are the first to present the superiority of state-of-the-art semi-supervised learning in semantic segmentation.

- This paper introduces a structured consistency loss for semantic segmentation under the semi-supervised learning paradigm. Specifically, authors propose to include this term to force consistency between the predictions of two networks (a teacher and a student). To evaluate their method, authors employ images from the cityscapes dataset. Results show that the proposed method achieves competitive results compared to state-of-the-art architectures.
- Contributions are unclear. The only contribution seems to be the ‘first study that employs semi-supervised learning and achieves state-of-the-art performance in semantic segmentation using the Cityscapes’. First, this contribution is insufficient to be considered as a technical contribution. And second, I feel authors mislead the reader with this assertion.
- The paper is poorly written with many senseless sentences and weird word compositions (e.g., ‘our research designed the semi-supervised learning approach suitable for semantic segmentation’ or ‘In this study, we are aimed at bypassing the labeling cost problem with semi-supervised learning technique.’ or ‘It regularizes the inter-pixel relationships consistent, thereby allowing the network to learn more powerful generalization capabilities’ just to give few examples). This makes the reading of this work particularly difficult, as well as the understanding on some parts.
- Authors fail to compile relevant works, particularly in semi-supervised segmentation, leading to a very weak related work section. Particularly in semi-supervised segmentation, more than one hundred papers have been published since 2015 only in major conferences on vision (CVPR,ICCV,ECCV) and learning (ICLR, ICML, NeurIPS). None of them, however, are listed in the related work section. Authors should significantly improve this section.
- Which is the difference with French et al. (2019) and Liu et al. (2019)? Both seem to be using CutMix for semantic segmentation.
- Where the labeled images are used? Authors show that labeled images and their corresponding ground truth masks are employed in eq 1) and 2). Nevertheless, it is never explained in which moment these are employed. Are they used to pre-traine the student and teacher networks? Are they mixed with unsupervised images during training? Please clarify.
- Please add reference for the cross entropy loss with boundary label relaxation.
- Furthermore, this paper contains many grammatical errors.
- Even though the idea is interesting, I am inclined to reject this paper because there are many unclear details, the related work is insufficient and the paper needs a thorough proof-reading in english.


Recently, Neural Architecture Search (NAS) has been successfully applied to multiple artificial intelligence areas and shows better performance compared with hand-designed networks. However, the existing NAS methods only target a specific task. Most of them usually do well in searching an architecture for single task but are troublesome for multiple datasets or multiple tasks. Generally, the architecture for a new task is either searched from scratch, which is neither efficient nor flexible enough for practical application scenarios, or borrowed from the ones searched on other tasks, which might be not optimal. In order to tackle the transferability of NAS and conduct fast adaptation of neural architectures, we propose a novel Transferable Neural Architecture Search method based on meta-learning in this paper, which is termed as T-NAS. T-NAS learns a meta-architecture that is able to adapt to a new task quickly through a few gradient steps, which makes the transferred architecture suitable for the specific task. Extensive experiments show that T-NAS achieves state-of-the-art performance in few-shot learning and comparable performance in supervised learning but with 50x less searching cost, which demonstrates the effectiveness of our method.  2019). However, the existing NAS methods only target a specific task. Most of them usually do well in searching an architecture for single task but are troublesome for multiple datasets or multiple tasks. As shown in Figure 1, we get the architecture-0 on given dataset using a NAS method. Now what if there exist a new task? This drives us to ask: how to get a suitable architecture for a new task in NAS? Generally, there exist two naive solutions in handling multiple tasks. One of them (S1) is to search an architecture for a new task from scratch but it is inefficient and not flexible for practical application scenarios. Another solution (S2) is to borrow architecture from the ones searched on other tasks but it might be not optimal for the new task. Therefore, it is urgently needed to study the transferability of NAS for large-scale model deployment in practical application. It should be more desirable to learn a transferable architecture that can adapt to some new unseen tasks easily and quickly according to the previous knowledge.
To this end, we propose a novel Transferable Neural Architecture Search (T-NAS) method (the bottom of Figure 1  Finn et al. (2017), where a model learns the meta-weights that are able to adapt to a new task through a few gradient steps. Push it forward, it is also possible to find a good initial point of network architecture for NAS. Therefore, the T-NAS learns a meta-architecture (transferable 1 Under review as a conference paper at ICLR 2020 Dataset NAS Architecture-0

Summary:
Current neural architecture search (NAS) methods work in the mode of what is called S1 in this paper: given a dataset, search for a new architecture from scratch for that particular dataset. This paper proposes using MAML-style metalearning for learning meta-architectures across many meta-training tasks so that given a new test task (dataset) the meta-architecture is a few search steps away from a near-optimal one. Hence the name Transferable-NAS (T-NAS).
The main method pretty much follows what one would expect for a MAML-style approach. During meta-training a number of tasks (datasets) are used to learn a meta-architecture and corresponding meta-weights (Algorithm 1) and then during meta-testing time, given a new task (dataset) use any NAS search technique starting from the meta-architecture and corresponding meta-weights. (The authors use DARTS as the NAS technique which by itself is the same bilevel optimization as MAML, but for NAS. This sets up a 4-level optimization problem during meta-training time!).
Experiments on two settings are presented: Few-shot learning and the more traditional supervised learning in NAS literature.
Comments:
- I really like the premise of the paper. While there have been papers trying to leverage dataset-level features for NAS transfer via bayesian approaches, computing dataset level features is always a bit difficult. The nice part about the MAML is that the transfer is embedded into the representation itself. But I have a bunch of clarification questions which is quite possible is mostly due to my misunderstandings. So please bear with me:
1. Section 5.2.1: "On the Mini-imagenet
dataset, One{normal + reduction} cell is trained 10 epochs with 5000 independent tasks for each epoch and the initial channel is set as 16." What are the 5000 independent tasks? In 5.1 it is said that 64 training classes, 16 validation classes and 20 test classes are present. I took that to mean that tasks are different classes of images in the few-shot setting. Clearly that is not the case. What precisely is a task in the few-shot learning case?
2. Secton 5.3: "...we choose 10 tasks with 200-shot, 50-query, 10-way for each task...." Again, what precisely is a task in the supervised learning setting?
3. One curious question I had is how much does architecture search matter as opposed to just taking the meta-architecture found by Alg 1 and just pretraining it on large image datasets like ImageNet or bigger and then just finetuning on downstream tasks. Specifically if you look at Table 3 ResNet12 pretrained has really good performance (although its size is ~100 times that of the others in the table). Perhaps S2 or AutoMAML with pretraining is the best option?


Generative adversarial networks have achieved remarkable performance on various tasks but suffer from sensitivity to hyper-parameters, training instability, and mode collapse. We find that this is partly due to gradient given by non-robust discriminator containing uninformative adversarial noise, which can hinder generator from catching the pattern of real samples. Inspired by defense against adversarial samples, we introduce adversarial training of discriminator on real samples that does not exist in classic GANs framework to make adversarial training symmetric, which can balance min-max game and make discriminator more robust. Robust discriminator can give more informative gradient with less adversarial noise, which can stabilize training and accelerate convergence. We validate the proposed method on image generation tasks with varied network architectures quantitatively. Experiments show that FID score of generated samples is improved by 10% − 50% relative to the baseline with additional 25% computation cost. Training stability is improved and mode collapse is alleviated.

This paper presents an interesting idea based on introducing adversarial noise on real samples during GAN training. This novel approach may improve GAN training and have potentially large impact, but the paper in its current form is slightly below the standard of ICLR due to its lack of clarity.
While it is very interesting to apply adversarial noise in real data, this approach is not clearly motivated or explained. For example, at the beginning of page 2, why “As a consequence, training will become unstable when generated distribution approximates target distribution because the gradient given by non-robust discriminator around real samples contains more adversarial noise”? One may think that, on the contrary, the adversarial noise in generated samples would approximate that in real data when the generator distribution approximate the true data distribution. Similarly, after eq. 6, why “Consequently, gradient given by discriminator may vanish when discriminator becomes stronger than generator without capacity constrained.”? I would think the gradient would explode in this case given all the regularisation on gradients.
In addition, the term “non-robust discriminator” has been used several times in important places, but is not clearly defined. Properties used to justify the approach, such as “symmetric” and “balanced” need to be explained. For example, would it be possible to illustrate or measure the imbalance of discriminator?
The overhead of the algorithm needs to be stated more precisely. (after eq. 8) It is unclear to me that “backward propagation of Equation 5 with respect to x with negligible computation overhead”. Would this require back-prop through the entire discriminator, which can be very deep thus costly? It would be helpful to provide an estimation of runtime overhead supported by experiments.
In algorithm 1, why is it necessary to perform discriminator update twice? How about skipping step 4 (eq. 12)? I think this may better mirror the adversarial update for generated samples, which only involves updating generator parameters once.
In section 3.4, I am not sure it is similar to unrolled GAN when the perturbation is zero. Unrolled GAN required backprop through discriminator update, which I don’t think is the case here
Finally, the experimental results are confusing. In Table 1, why the reproduced results are already much better than those in the original papers? A solid baseline is necessary for any further assessment. Further more, since the DCGAN and ResNet baseline models did not use recent regularisation approaches such as spectral-norm, it is hard to assess whether the proposed method can work without those existing techniques. It would be helpful to use at least a SN-GAN baseline.
The analysis also need to be improved. It is hard to interpret the histograms and L1 norms in Figure 5 as related to “informative gradient” or “semantic information” as claimed in sectiion 4.4. Quantitative measurements such as correlation or mutual information may justify these claims.
Overall, I think the idea presented is certainly promising, but needs further development to be qualified for acceptance.
Nit:
The 1-line derivation of eq. 6 can be incorporated into the main text.
A transpose in eq.10 is missing.
Update:
I read the authors' rebuttal, which addressed many of my concerns and presented additional empirical results. The improvement over SN-GAN baseline is particularly convincing. I therefore believe the final camera-ready version can be a valuable contribution to the field, and would like to recommend accepting this paper.


Neural Architecture Search (NAS) has seen an explosion of research in the past few years. A variety of methods have been proposed to perform NAS, including reinforcement learning, Bayesian optimization with a Gaussian process model, evolutionary search, and gradient descent. In this work, we design a NAS algorithm that performs Bayesian optimization using a neural network model. We develop a path-based encoding scheme to featurize the neural architectures that are used to train the neural network model. This strategy is particularly effective for encoding architectures in cell-based search spaces. After training on just 200 random neural architectures, we are able to predict the validation accuracy of a new architecture to within one percent of its true accuracy on average, for popular search spaces. This may be of independent interest beyond Bayesian neural architecture search. We test our algorithm on the NASBench (Ying et al. 2019) and DARTS ) search spaces, and we show that our algorithm outperforms other NAS methods including evolutionary search, reinforcement learning, AlphaX , ASHA, DARTS, and TPE. Our algorithm is over 100x more efficient than random search, and 3.8x more efficient than the next-best algorithm on the NASBench dataset. As there have been problems with fair and reproducible experimental evauations in the field of NAS, we adhere to the recent NAS research checklist (Lindauer and Hutter 2019) to facilitate NAS research. In particular, our implementation has been made publicly available, including all details needed to fully reproduce our results.

The paper introduces a new encoding for cell structures to improve efficiency of neural architecture search methods.
As a second contribution the paper proposes Bayesian optimization with an ensemble of neural networks as probabilistic model for the objective function.
Both contributions combined show superior performance on the Nasbench101 benchmark as well as competitive performance on the DARTS search space.
While the paper identifies an important problem - encoding of architectures - I do not think the paper is ready for acceptance.
About the encodings:
First, using different encodings to enable better architecture search has been investigated by others before.
For example, Ying et al. also provided different encodings of the adjacency matrix for Nasbench101 besides the used binary encoding and it seems that different methods work well with different encodings.
Also in the work by Kandasamy et al. they presented an encoding for architecture, such that Bayesian optimization can be applied.
Second, the encoding described in the paper lacks some intuition.
 - How does enumerating all paths and encoding them as a binary vector convey more information than just using the adjacency matrix?
 - Lead isomorphic graphs, which for example occur in Nasbench101, to the same encoding?
 - It seems somewhat counter intuitive to use a large binary vector (more than 18000 dimensional vector for the DARTS space) as encoding for Bayesian optimization which is known to struggle with high dimensional input spaces.
About the Bayesian optimization strategy:
The proposed probabilistic model for Bayesian optimization seems straight forward and simple. Also here, previous work (Snoek et al, Springenberg et al., Perrone et al)  already proposed to use neural networks and ,in order to be more convincing, the paper should include a comparison to these methods.
Furthermore, the paper should clarify how the diversity in the neural network ensemble is enforced. Are the neural networks trained with different random initialization? How does it compare to the method proposed by Lakshminarayanan et al. which showed better performance for neural network ensembles based only on different random initialization?
Minor comments:
- In the Nasbench101 paper other Bayesian optimization strategies (e.g SMAC, BOHB, TPE) showed strong performance. The results would be more convincing if these methods are included in the comparison.
- Following the empirical protocol by Ying et al. the results would be easier to parse if the Figure 3  could report the log test regret on the y-axis. I am also missing a figure that shows the robustness of the method across independent runs.
- How are invalid architectures in the Nasbench101 (e.g architectures that violate the max edge constraint) treated in the experiments?
- I think the paper is missing the following references:
Simple and scalable predictive uncertainty estimation using deep ensembles
B Lakshminarayanan, A Pritzel, C Blundell
Advances in Neural Information Processing Systems, 6393-6395
Scalable hyperparameter transfer learning
V Perrone, R Jenatton, M Seeger, C Archambeau
Advances in Neural Information Processing Systems, 6845-6855
post rebuttal
------------------
I thank the authors for answering my questions regarding the path encoding and taking the time to improve the empirical evaluation of the paper. While I think that the paper has improved, I am afraid that the contributions of the paper are still not strong enough to reach the bar of acceptance because of the following reasons:
- The path encoding is an interesting approach and seems to improve upon just using the adjacency matrix directly, it doesn't scale and, hence, it remain somewhat unclear how valuable it is in practice.
- More importantly, I am not convinced that the proposed neural network model represents a sufficient contribution. After some discussion with the authors, they agree that existing BO methods based on neural networks could also be applied to this setting and even say that they may perform well with the path encoding. However, they are not include them in the comparison and only promise to add them for the final evaluation.  I am concerned that if it turns out that other methods perform as well or even better, it would dramatically lower the contribution of the paper.


Decentralized stochastic gradient descent (SGD), where parallel workers are connected to form a graph and communicate adjacently, has shown promising results both theoretically and empirically. In this paper we propose Moniqua, a technique that allows decentralized SGD to use quantized communication. We prove in theory that Moniqua communicates a provably bounded number of bits per iteration, while converging at the same asymptotic rate as the original algorithm does with full-precision communication. Moniqua improves upon prior works in that it (1) requires no additional memory, (2) applies to non-convex objectives, and (3) supports biased or linear quantizers. We demonstrate empirically that Moniqua converges faster with respect to wall clock time than other quantized decentralized algorithms. We also show that Moniqua is robust to very low bit-budgets, allowing less than 4-bits-per-parameter communication without affecting convergence when training VGG16 on CIFAR10.

This paper considers training of DL models in a decentralized setting. Previous work (Tang et al 2018b, Koloskova et al 2019, Tang et al 2019) introduced communication compression in order to reduce communication cost in decentralized SGD. This paper shows that the naive way to apply quantization to compress communication can fail (but, none of the previous works used this naive way). The paper proposes a novel decentralized SGD algorithm called MONIQUA which communicates only with compressed vectors. The paper theoretically proves that MONIQUA asymptotically converges with the same speed as D-PSGD (Decentralized SGD with full communications) for non-convex functions.
The main benefit of the proposed algorithm is that in contrast to the baselines, MONIQUA doesn’t require any additional memory and computation overhead connected to that.
The paper also allow to combine MONIQUA with asynchronous communications (AD-PSGD) and decentralized data (D^2), which is novel, previous baseline didn’t allow asynchronous communications. Paper experimentally validates MONIQUA and show that it converges faster than the baselines on early optimization stage, but omit comparison of final results.
My score is weak reject. The method is interesting and elegant, doesn’t require additional memory, theoretically convergent with a good speed and allows asynchronous communication. However, it looks a bit incremental. In contrast to the baselines, MONIQUA does not support arbitrary communication compression. Experimental comparison is shown only for the beginning of the optimization where the algorithm doesn’t achieve state of the art accuracy.
Major concerns:
1. Superior experimental results are shown only for (i) the constant stepsize schedule and (ii) only in the beginning of the optimization. The algorithm is unable to achieve s.o.t.a. test accuracy and has severe accuracy drop of 10-20%. It is unclear if MONIQUA is able to close the accuracy gap.
To get good test accuracy the exponentially decaying learning rate schedules are usually used when training ResNet on Cifar10. However, that is unclear if this learning rate schedule could be supported by the algorithm. I see that theoretically the same result would be not possible. Once the learning rate dropped, ||x_i - x_j|| has to be smaller than the new theta, which is not true.
2. In contrast to the baselines, MONIQUA doesn’t support arbitrary quantization. The quantization level is fixed. For example, for the ring 8 case (which was used in experiments), the number of bits required to converge is at least 5 (I calculated it using expression for B on page 5). This also questions if MONIQUA will be able to converge to good accuracy in extreme bit-budget.
Other concerns that should be addressed:
3. The spectral gap is usually not a constant: for example for the ring topology it decreases as 1/n^2. That should be clarified in the convergence rate and the number of bits required to communicate.
4. In the experiments, to ensure fair comparison, the tuning details of the averaging rate for DeepSqueeze and Choco baselines, and theta parameter for MONIQUA were not given.
5. Reproducibility of the experiments: Many experimental details are omitted:
   - What are the values of hyperparameters used in the plots?
   - How did you model asynchrony in experiments?
   - How did you choose the parameter \theta in experiments?
6. In (1), why x is \in [-1, 1]^d? What if this assumption doesn’t hold in (4)? The Theorem 1 is not clear then.
7. Why taking mod \theta is required? Cannot you assume that ||Q(x) - x|| <= theta delta? The whole analysis will stay true.
8. In Asynchronous communication section, why \tau_k are the same for all the workers? Cannot different workers have different delays?
9. Why in experiments D-PSGD doesn’t converge when the data are decentralized? This is not consistent with the theory, D-PSGD should converge just slower.
Questions:
10. Is there a way to compute theta it in practice beforehand?
Minor comments:
- x mod y is not defined when x is a float number.
- Intuition behind Moniqua: do you assume that worker 1 has only one neighbour (worker 2)? does m_1 is one dimensional or high dimensional?
- Algorithm 1, the notation \bar X_k might be confusing as in the proof big letters correspond to matrixes. Consider to change on \bar x_k.
- why some equations are numbered and some are not?
- In Asynchronous communication section: what are the assumptions on W_k?
- it would be helpful for the clarity to remind what is the stochastic rounding quantization in configuration of experiments section.
- which are the other algorithms you talk about in footnote 6?
- In wall-clock time evaluation section: how floored outputs could be represented as 16-bit integers? aren’t the vectors which you quantize always smaller than one?
- wall-clock time evaluation section: the statement that “algorithms diverge” might be confusing.
- Aggressive quantization add  ”.”
- why the number of epochs differ in different experiments? Fig. 3(a) and Fig. 4(a) ?


In contrast to fully connected networks, Convolutional Neural Networks (CNNs) achieve efficiency by learning weights associated with local filters with a finite spatial extent. An implication of this is that a filter may know what it is looking at, but not where it is positioned in the image. Information concerning absolute position is inherently useful, and it is reasonable to assume that deep CNNs may implicitly learn to encode this information if there is a means to do so. In this paper, we test this hypothesis revealing the surprising degree of absolute position information that is encoded in commonly used neural networks. A comprehensive set of experiments show the validity of this hypothesis and shed light on how and where this information is represented while offering clues to where positional information is derived from in deep CNNs.

This paper studies whether and how position information is encoded in CNNs. On top of VGG and ResNet, it constructs an additional PosENet to recover position information. By analyzing how well PosENet recovers position information, this paper provides several interesting findings: CNNs indeeds encode position information and zero-padding is surprisingly important here.
[Pros]
1. I enjoy reading this paper: probing CNNs is not easy, but it designs experiments in an intuitive way and rigorously performs ablation studies and analysis.
2. The observations and findings are interesting and helpful to the community.
[Cons]
1. A weakness of this paper is that it ignores the impact of training process while probing PosENet: In Table 1, VGG/ResNet perform much better than PosENet, but it could be because VGG/ResNet is easier to train (kind of fine-tuning PosENet only) than PosENet. Would be nice to show the training curve and train PosENet longer.
2. Zero-padding seems to play a surprisingly important role in encoding position information (Table 5), but it is still unclear why it is so important and how it helps.
Overall, I think this is a good paper.


Spiking Neural Networks (SNNs) operate with asynchronous discrete events (or spikes) which can potentially lead to higher energy-efficiency in neuromorphic hardware implementations. Many works have shown that an SNN for inference can be formed by copying the weights from a trained Artificial Neural Network (ANN) and setting the firing threshold for each layer as the maximum input received in that layer. These type of converted SNNs require a large number of time-steps to achieve competitive accuracy which diminishes the energy savings. The number of time-steps can be reduced by training SNNs with spike-based backpropagation from scratch, but that is computationally expensive and slow. To address these challenges, we present a computationally-efficient training technique for deep SNNs. We propose a hybrid training methodology: 1) take a converted SNN and use its weights and thresholds as an initialization step for spike-based backpropagation, and 2) perform incremental spike-timing dependent backpropagation (STDB) on this carefully initialized network to obtain an SNN that converges within few epochs and requires fewer time-steps for input processing. STDB is performed with a novel surrogate gradient function defined using neuron's spike time. The weight update is proportional to the difference in spike timing between the current time-step and the most recent time-step the neuron generated an output spike. The SNNs trained with our hybrid conversion-and-STDB training perform at 10X − 25X fewer number of time-steps and achieve similar accuracy compared to purely converted SNNs. The proposed training methodology converges in less than 20 epochs of spike-based backpropagation for most standard image classification datasets, thereby greatly reducing the training complexity compared to training SNNs from scratch. We perform experiments on CIFAR-10, CIFAR-100 and ImageNet datasets for both VGG and ResNet architectures. We achieve top-1 accuracy of 65.19% for ImageNet dataset on SNN with 250 time-steps, which is 10X faster compared to converted SNNs with similar accuracy.

This paper examines combining two approaches of obtaining a trained spikingneural network (SNN). The first approach of previous work is converting the weights of a trained artificial neural network (ANN) with a given architecture, to the weights and thresholds of a SNN, and the second approach uses a surrogate gradient to train an SNN with backpropagation. The true novelty of the paper seems to be in showing that combining the two approaches sequentially, trains a SNN that requiresfewer timesteps to determine an output which achieves state of the art performance. This is summarized by Table 1. However, it does not mention how many epochs it takes to train an SNN from scratch, nor compare this to the total training time (ANN training + SNN fine-tuning) of their approach. They also claim a novel spike-time based surrogate gradient function (eq. 11), but it is very practicallysimilar to the ones explored in the referenced Wu. et al 2018 (eq. 27 for instance), and these should be properly contrasted showing that this novel surrogate function is actually helpful (the performance/energy efficiency might only come from the hybrid approach). The authors argue for SOTA performance in Table 2, but the comparison to other work doesn’t clearly separate their performance from the otherlisted works; For example the accuracy gain against Lee et al.,2019 only comes from the architecture being VGG16 as opposed to VGG9, as can be seen from comparing with the VGG9 architecture from Table 1, furthermore they take the sameamount of timesteps, which is supposed to be the principle gain of this work.
Some small suggestions that are independent from the above:
1.The most similar or relevant version of equation (2) in previous work could be referenced nearby for context.
2.The last sentence of the first paragraph on p.4 “the outputs from each copy...” is confusing. Are you just meaning to describe BPTT?
3.Typos: sec7 4th line “neruons”, sec 2.2 “both the credit” (remove “the”)
---------------
Following the author response I have upgraded my rating.


Convolutional Neural Networks (CNN) have achieved state-of-the-art performance in different computer vision tasks, but at a price of being computationally and power intensive. At the same time, only a few attempts were made toward a deeper understanding of CNNs. In this work, we propose to use semantic pruning technique toward not only CNN optimization but also as a way toward getting some insight information on convolutional filters correlation and interference. We start with a pre-trained network and prune it until it behaves as a single class classifier for a selected class. Unlike the more traditional approaches which apply retraining to the pruned CNN, the proposed semantic pruning does not use retraining. Conducted experiments showed that a) for each class there is a pruning ration which allows removing filters with either an increase or no loss of classification accuracy, b) pruning can improve the interference between filters used for classification of different classes c) effect between classification accuracy and correlation between pruned filters groups specific for different classes.

Summary
---
(motivation)
This paper proposes a new approach to pruning activations from neural networks,
but uses it to understand neural nets better rather than trying to make them
more efficient. It prunes filters per-class 1) to measure how sensitive image
classes are to pruning and 2) to measure how similar classes are by comparing
the filters they prune.
(approach)
Using only images from class c, CNN pre-activations are aggregated across spatial
dimensions and examples. This gives an average feature vector for that class.
T percent of neurons are pruned. Features with lower pre-activations
(possibly negative with large magnitude) are pruned first with successively
higher activations pruned later.
No re-training is performed.
(experiments)
Experiments use AlexNet and 50 of the 1000 ImageNet challenge classes to show:
1. Pruning filters results in decreased accuracy with smaller decreases in accuracy as the first and last few filters are pruned.
2. More filters from middle layers are pruned than are those from early and late layers.
3. After some filters have been pruned, sometimes pruning can increase accuracy.
4. I could not understand section 3.2.
Strengths
---
The proposed pruning method is simple and efficient.
I like the broad goal of understanding CNNs. We could use more papers that just focus on analysis like this one.
I think the idea of class conditional pruning is novel.
Weaknesses
---
# Major Weaknesses
I don't see why these results are significant. I do not find these results very surprising (see next comment) and I do not see why the community will find them useful.
* In particular, consider the conclusion. Sentences 2, 3, 5, and 6 seem to all be observations about what happened in the experiments. The conclusion should re-iterate the results, but it should also say why they are important. How does the work relate to the goals of the community at large? Which goals? Will this enable important new capabilities? What general concepts did we learn from this that we didn't know before?
Some of the positions in the paper could be more carefully considered. There are alternate explanations for many of these phenomena.
* Pruning smallest activations doesn't make sense to me. Typically redundant or un"important" activations are pruned because doing so has negligible impact on accuracy. The smallest activations are pruned here. Should the smallest activations be redundant or unimportant in some way? Does this rely on ReLU activations following the pre-activations? I think these values are not necessarily redundant and could be quite important (e.g., as measured by some saliency explanation like Integrated Gradients), but I could be wrong. It would be useful to provide a baseline which removes highest instead of lowest activations.
* Much of the surprise about figure 5 seems to be because it is not monotonic but it was expected to be monotonic. I agree that these curves should generally go down as theta increases, but I don't see why that relationship should be strict. I would be surprised if activations were not in some way dependent on one another. Furthermore, why does that dependence have to be interference? Couldn't it also be that some activations are complementary (and thus ineffective when only one is present)?
* Does Network Wise Pruning (NWP) favor more layers than others? It may be that some filters have higher Accumulated Responses per filter simply because there are more feature maps in the previous layer (thus more things summed up) and not because of what information they capture or their relationships with other filters. Does this happen? This could be an alternative to the following conclusion: "This means that the encoding provided by the first and last layer seems to be the most crucial and the densest."
# Other General Weaknesses:
* AlexNet is a rather old architecture to use for this analysis, so I can't be confident these methods or behaviors will generalize to other architectures. Does it hold for more modern architectures?
# Missing details / Points of confusion:
* The paper says the correlation from Figure 6 should be expected to decrease as theta increases. Why should this be expected?
* What exactly does Figure 6 measure? I think it's the ratio of the size of the intersection rho_sigma to the size of the union of the same two sets. However, the text calls the metric "correlation." This should be made clearer.
# Minor presentation weaknesses
Some parts of the notation/explanation don't make sense to me:
* "Let Lambda=... be the number of object classes in the dataset." But Lambda is defined as a set, not a numeral.
* The number p needs more context/subscripts as it depends on class c and filter i.
* "θ = [0, . . . , 1]" This defines theta as a finite set, but I think it's meant to be a number in the interval range (0, 1).
* Why is lambda_c needed instead of just using the index c to specify a class? The variable lambda doesn't seem to be any different than a class index.
* "let Fσ and Fσ ̄ be the set of unpruned and pruned filters for a given γ..." I don't see how this works. According to eq. 3 gamma depends on a particular filter i and class c, so the only filter F could contain is the ith one. (Later it became clear that F was only mean to class conditional, not filter conditional.)
Figure 5: These plots shoul all have the same y axis range. This would make them comparable and allow readers to much more easily compare trends across classes. Similar steps should be taken so the same range is used any time theta is plotted on the x axis.
Figure 3: I find it hard to get an overall ordering of the approaches in this figure because there is so much variance from class to class. It effectively conveys the variance, but I'd also like to know what the means across classes are for each method so I can compare the proposed approaches more effectively.
"The results indicate that classification classes are asymmetrically represented by filters resulting in the fact that some object classes have their classification accuracy increased when pruned for."
* I'm not sure what it means to be asymmetrically represented by filters.
Suggestions
---
This analysis would have been more interesting with an existing pruning approach because we would already know that such an approach is good a removing unnecessary filters.
Final Evaluation
---
Quality: Experiments could have been cleaner, but they basically demonstrate the patterns the paper intended to show.
Clarity: I could understand most experiments at a high level, but I found it hard to understand the motivation and the rest of the experiments.
Significance: As explained above, I do not see why the paper is significant.
Originality: The experiments and the proposed class conditional pruning approach are somewhat novel.
The paper is somewhat novel, but do not find it very clear and I do not see why it is important, so I cannot reccomend it for acceptance.


Many variants of Graph Convolutional Networks (GCNs) for representation learning have been proposed recently and have achieved fruitful results in various domains. Among them, spectral-based GCNs are constructed via convolution theorem upon a theoretical foundation from the perspective of Graph Signal Processing (GSP). However, despite most of them implicitly act as low-pass filters that generate smooth representations for each node, there is limited development on the full usage of underlying information from low-frequency components. Here, we first introduce the octave convolution on graphs in spectral domain. Accordingly, we present Octave Graph Convolutional Network (OctGCN), a novel architecture that learns representations for different frequency components regarding weighted filters and graph wavelets bases. We empirically validate the importance of lowfrequency components in graph signals on semi-supervised node classification and demonstrate that our model achieves state-of-the-art performance in comparison with both spectral-based and spatial-based baselines.

This paper extends the previous graph wavelet neural network with separate computation for low frequency and high frequency part. The idea is to bring the octave convolution in vision to the graph domain. Experiments on three benchmark node classification tasks show comparable performances as previous methods.
Overall the paper is written in a coherent and self-contained way, where the paper clearly states the related work and the contribution of this newly proposed work. Also it is interesting to see that normalizing the diagonal filter, tying the weights for low/high frequency parts would make the generalization better. However, there are several major concerns with the paper:
1. The contribution is somewhat limited. The main component is based on GWNN. While the GWNN itself takes the full spectrum of basis already, the formulation in (8) and (9) should somehow capture the same information. I think the paper spends too much content on the reviews, while lacks the intuition or theoretical explanations of the proposed formulation.
2. Having marginal improvement on the three benchmarks is not that interesting. Also given the results are mixed with other baselines, I think more experiments (e.g., on large graphs, or graph-level supervised tasks, etc.) are necessary to demonstrate the empirical gain using the proposed formulation.
3. Regarding the experiments, is it true that d/n=0 and d/n=1 should have exactly the same results?
4. In Figure 2, I guess d/n=0 should be reduced to the GWNN. But it seems the performance is different than GWNN on citeseer and cora. Why there’s such inconsistency, or did I miss anything?


Reinforcement learning (RL) offers a principled way to achieve the optimal cumulative performance index in discrete-time nonlinear stochastic systems, which are modeled as Markov decision processes. Its integration with deep learning techniques has promoted the field of deep RL with an impressive performance in complicated continuous control tasks. However, from a control-theoretic perspective, the first and most important property of a system to be guaranteed is stability. Unfortunately, stability is rarely assured in RL and remains an open question. In this paper, we propose a stability guaranteed RL framework which simultaneously learns a Lyapunov function along with the controller or policy, both of which are parameterized by deep neural networks, by borrowing the concept of Lyapunov function from control theory. Our framework can not only offer comparable or superior control performance over the state-of-the-art RL algorithms, but also construct a Lyapunov function to validate the closed-loop stability. In the simulated experiments, our approach is evaluated on several well-known examples including classic CartPole balancing, 3-dimensional robot control and control of synthetic biology gene regulatory networks. Compared with RL algorithms without stability guarantee, our approach can enable the system to recover to the operating point when interfered by uncertainties such as unseen disturbances and system parametric variations to a certain extent. (Anonymous code is available to reproduce the experimental results 1 .)

######## Rebuttal Response:
Thanks for the thorough response.
Q2: The title still hasn’t changed on the current draft
Q4: To be more precise:
‘a novel data-based approach for analyzing the stability of the closed-loop system is proposed by constructing a Lyapunov function parameterized by deep neural network’ - this alone is not novel, you would need to specify how your method of doing this is new
‘a practical learning algorithm is designed to search the stability guaranteed controller’ - this is a natural consequence of contribution 1, some further justification is needed as to why this could be viewed as an interesting contribution (i.e. the Lagrangian approach, if this is novel)
‘ the learned controller is able to stabilize the system when interfered by uncertainties such as unseen disturbance and system parameters variations of certain extent’ - this is not a contribution, but an experimental result.
Q5: The review believes that model-free control and stability-guarantees are fundamentally orthogonal ideas, rather than just under-studied work as the authors have been suggesting in the script and rebuttal. Given that discrete-time Lyapunov stability is defined through expressions along the lines of L(f(x)) - L(x) < 0, for Lyapunov function L and closed-loop dynamics f, claiming that stability is being ‘analyzed’ without f is disingenuous. Instead, by making the value function a Lyapunov function, the goal is that the *converged* value function should produce a stable policy, and still, this is surely only assured within the space of samples. Moreover, the use of the discount factor \gamma, popular in MFRL, essentially acts as a time horizon, so I’m not convinced a Lyapunov function learned with a \gamma < 1 can be called stable in the pure infinite-horizon sense.
With this in mind, I think the work would benefit from a revised central claim: that the use of Lyapunov value functions (as an inductive bias) provides more *robust* model-free controllers. I believe this message highlights the value of this work for MFRL, without making false assertions. This, in particular, would highlight the fact that many MFRL algorithms are benchmarked on deterministic environments, and therefore incredible brittle as the experimental results suggest.
Q9: This remark was aimed at earlier in the paper, either the introduction or main section, rather than the experimental section. The fact that a value function can be viewed as a Lyapunov function makes sense but I’m not sure it is a well-known fact in the wider community. Basically, an introduction to the intersection of Lyapunov stability and optimal control would improve the paper.
Q:10 The fact that the clipping of the multiplier corresponds to unstable policies during learning demonstrates that this pitfall needs to be expressed explicitly. Whether the stability guarantees apply to the converged policy or also intermediate policies is not clear on the initial reading of the paper.
For me, this highlights another weakness in the paper. This initial theorems talk of L(s), which relates to the critic L_c by L(s) = E_{u\sim\pi(s)} [L_c(s,u)], however in the subsequent objectives (eg Eq 2), this marginalization never occurs, therefore I don’t feel like you can say Theorem 2 applies to your resultant algorithm.  Moreover, with the \alpha_3 c term in Equation 3, c should be c(s, a) with a marginalized, which it doesn’t appear to be, and the hyperparameter alpha_3 is never discussed nor tuning explained. Assuming this not done in the code, the experiments need to be re-evaluated with Theorem 2 properly enforced through a sample approximation of the marginalization.
Q11/Q12: Thank you for the Markov jump experiments. I’m not sure I understand why LAC is able to learn the task while SAC cannot. To me, this suggests perhaps a lack of hyperparameter tuning for SAC or further investigation. Moreover, there are typos in captions Fig 1, e and f.
A note of figures: Please ensure all axes should be labeled and should be of sufficient size. Many are too small and unreadable. Figure 2 looks like it could be 1 plot (though perhaps requires normalization).
Given that the strength of this method is the added robustness upon convergence, I think it would be valuable to focus less on time-domain results (Figure 3) (these can be added to the appendix for clarity), but instead show how each parameter/noise variation affects the mean and variance of the episodic return. I would expect that, while LAC provides significant robustness, it is still limited. The results don’t demonstrate this. It would also be interesting to know which hyperparameter controls this limit. I imagine there is a robustness/performance tradeoff.
In conclusion, while I appreciate the efforts the authors put into the rebuttal, the extended discussions made me rethink my rating and I have decreased my rating to reject. I believe fixing the issues highlighted above and redrafting the central message must be done before this paper is ready for publication.
######## Review:
This paper investigates the use of Lyanpunov theory as an inductive bias for improving the stability / robustness of policies in a model-free actor-critic reinforcement learning setting. Through viewing the Critic as a Lyapunov function, optimizing the policy with a Lyapunov-based constraint is meant to ensure the stability of the policy through a ‘cost stability’ metric.. Experimental results show that Lyapunov-based Soft-Actor Critic (LAC) is more robust than SAC on some linear and nonlinear environments.
The reviewer believes that the study of intersections between Control Theory and RL to be immensely valuable and the authors outline a principled formulation. However,  the implementation, experiments and general manuscript suggest that paper requires further work before it is conference-ready.
As the author understands it, the current state of the literature of Lyapunov methods for Deep Reinforcement Learning can be summarized as:
Richards et al, 2018, Classify stable region and learn neural Lyapunov function for a safe exploration strategy
Berkenkamp et al, 2018: Classify the stable region via GP, move there for exploration
Chow et al 2018 Constrained MDPs for discrete gridworld environments
Chow et al 2019 Constrained MDPs for continuous environments through a projection on the policy
This work: Actor-Critic constrained policy optimization with a lyapunov-based value function critic
In the introduction and the related work, too much emphasis is put on explaining stability and discussing methods like Model Predictive Control (MPC) which do not benefit the rest of the paper. Additionally, the three contributions listed do not seem particularly novel given the past literature.
The premise of the formulation also presents several unquestioned assumptions and design decisions:
Why model-free RL, as the authors also state that many samples are required to validate stability?
How does the requirement of stability inform the search strategy in this work? Especially as SAC uses a maximum entropy stochastic policy to aid exploration.
Do you really get ‘guarantees’ with sample-based methods? I would expect bounds based on the number of samples
The cost-based measure of stability seems open to abuse - i.e. for the half-cheetah environment only the centre-of-mass horizontal velocity in covered in the cost function, the stability of the embodiment (joint angles and velocities) are ignored. For the Fetch Reacher, a cost function in cartesian space ignores instabilities from kinematic singularities in joint space. I would image the cost function needs to be a measure on the entire dynamic state.
The notion of a Value function as a Lyapunov function is very interesting, and since it was the basis of the work, would have benefitted from more discussion, i.e. for which cost/reward function families the equivalence is valid for, and how it compares to other Lyapunov candidate functions.
With the RL formulation, the requirement of clipping with the lagrangians is suspicious, as it suggests the objective and/or its numerics are not well posed.
With the choice of experiments, they do not seem to question the central problem outlined by the paper. Rather than show environments SAC returns unstable trajectories during learning, the experiments aim to demonstrate instead a general robustness. The reviewer appreciates that stability is difficult to assess; however, while stability is heavily linked to robustness, a paper title promising stability guarantees should demonstrate some strong empirical evidence stability.
Additionally, the choice of environments do not seem to be ideal test beds for stability - i.e, the half-cheetah is stabilized via interactions with the ground. The reviewer would prefer to see simpler nonlinear environments, such as Markov Jump Processes / Switching Linear Dynamics, where SAC clearly demonstrates instability during learning which LAC is sufficiently regularized against. Additionally, while the `repressilator’ is an interesting application to the domain of bioengineering, its addition does not seem to be especially motivated by the central goal of the paper, so just adds to confuse the reader with unnecessary theoretical content.
Moreover, a brief literature review uncovered some relevant earlier work which was not cited:
Construction of neural network based Lyapunov functions, Petridis et al, 2006
Generation of Lyapunov functions by neural networks, Noroozi et al, 2008
Lyapunov Design for Safe Reinforcement Learning, Perkins et al, 2002
Some of the references also appear incorrectly formatted or incorrect, i.e. the reference for Spencer et al, 2018 should be the CoRL 2018 version rather than arxiv.
Also, the general use of grammar in the manuscript would benefit from another draft. In particular, the title could be improved, i.e.
    Model-free Control of Nonlinear Stochastic Systems with Stability Guarantees


We consider the problem of unconstrained minimization of a smooth objective function in R d in setting where only function evaluations are possible. We propose and analyze stochastic zeroth-order method with heavy ball momentum. In particular, we propose, SMTP, a momentum version of the stochastic three-point method (STP) Bergou et al. (2019). We show new complexity results for non-convex, convex and strongly convex functions. We test our method on a collection of learning to continuous control tasks on several MuJoCo Todorov et al. (2012)  environments with varying difficulty and compare against STP, other state-of-the-art derivative-free optimization algorithms and against policy gradient methods. SMTP significantly outperforms STP and all other methods that we considered in our numerical experiments. Our second contribution is SMTP with importance sampling which we call SMTP_IS. We provide convergence analysis of this method for non-convex, convex and strongly convex objectives.

The authors extend recent the recent stochastic three-point (STP) method to allow for Polyak-style momentum, as well as momentum with importance sampling. They provide a range of analysis that mostly extends existing STP results to the STP+momentum case. Most of these results are similar in spirit to stochastic gradient or subgradient results, as in the methods converge up to a ball around the solution, with radius depending on step-size, so you can get an epsilon solution by choosing a suitably small stepsize. The analysis covers non-convex cases (bounds are on the norm of the gradient) and the importance sampling case as well.
Overall, I think this is a strong paper, and a very interesting topic, and hence I support a "Weak accept". The numerical results look good, as the new method outperforms most of the compared methods, at least for the easier problems (SMTP beats competitor ARS in 3/5 trials; both methods are generally very similar, though SMTP does better on the easy Swimmer problem; the SMTP_IS results are more complicated).  The analysis is mostly good (non-trivial), and shows a broad understanding.
That said, I have some concerns.
(1) The analysis is nice in that it shows the methods work, but doesn't demonstrate benefit of their method over other methods
(2) Given that there are no results showing this method has better worst-case rates than other methods, we rely on experiments to see the actual benefit. In this case, more experiments is always better.
(3) I am quite skeptical of the importance sampling scheme. It's nice to include it, but I don't think it strengthens the paper too much. Empirically, the performance seems to help sometimes but not other times. Finding the individual Lipschitz constants seems tricky; this paper re-uses a scheme that iterates for a while, fits a function, and uses that to estimate the constants (it wasn't clear if this pre-processing was counted in the iteration count for experimental results). It's not clear how well that works to get an accurate estimate. Furthermore, to exploit the importance sampling, the directions must be sampled from a pre-determined basis, which seems restrictive.  This criticism is not just of the current paper but of other papers that use this approach.
-- The manuscript needs more proof reading, as there are mistakes in most paragraphs. There are a lot of problems with missing articles.  Phrases like "results for STP are shorts and clear" ("shorts" --> "short"), "which updates rule" [?? which-->with?? ], "hints [at] the update rule", "is far more superior" [-->"is far superior", since you can't be more superior], etc.
-- There is a confusion over how to use \cite, \citet and \citep in latex. Given the bibtex citation style, this makes it very hard to read in places
-- Literature review seems good and pretty thorough (mentions most key references through 2015, and a good selection of references since then).
-- Assumption 3.1 part 2 is stated in a funny way (it says, "there is a constant mu_D and a norm || ||_D such that ...").  You are free to choose the norm, and then find the constant (since all norms in finite dimensions are equivalent). That way, you can choose the norm that gives the tightest inequality.  I think you are aware of this, and it's just a wording issue.
-- Theorem 3.5 (Thm D.2) requires the mu_D^2 to be less than the condition number, which is weird. The easier the problem is, the tighter your assumptions are.  I suspect that this is because you use an inequality somewhere that simplifies things by bounding a term by the condition number. But as stated, this is a weak theorem.  It is also confusing because you have a mu_D which is not the strong convexity constant, but the actual strong convexity constant (mu) *does* depend on the norm D (cf eq 19; and this must be so, otherwise you can cheat and then the value of mu_D is meaningless). So both mu's are functions of the norm D. However, the Lipschitz constant L is *not* a function of D. So notation is confusing and makes interpreting the results harder.
-- sentences like "We achieve the state-of-the-art performance compared to *all* DFO based and policy gradient methods" are in appropriate (*italics* are mine). You mean to say that on the few examples you ran, based on a few DFO and policy gradient methods you tested, that the best of your two methods was better than the competitor methods on 4/5 problems.
-- I think a common-sense algorithm to compare to would be gradient descent (or heavy-ball) using finite differences to estimate the gradient.  In small dimensions this isn't such a bad idea.  I don't actually know what the dimensions of your test problems are (I looked in section 5 but didn't see it mentioned, other than reference to Ant-v1 and Humanoid-v1 being "high dimensional"; I think this is extremely relevant information. In small dimensions, traditional DFO and Bayesian optimization methods are competitive).
-- p 26/27, "Causchy-Schwartz" is spelled wrong, and usually this is called "Holder's inequality" when it's not the Euclidean norm.
-- Table 3, there is no space between the caption and the main text, so it's confusing
-- Eq (76) in appendix, the sum should go to d not n.
-- I think s^k may need to be independent of z^k for their tower property thing to work, otherwise it's not clear what's happening with the inequality prior to overset (30) on that last line of pg17. For example, if s^k were z^k measurable then that whole thing in the inner expectation would be a constant. This isn't a problem, it's fairly natural to assume that s^k is independent of z^k, I just didn't see the assumption anywhere.
-- Overall, comparing importance sampling results is hard, due to the different norms (this is mentioned in the paper, and there are inequalities between norms, but it's still hard to get a good result that shows importance sampling has better worst-case rates).
== AFTER READING REBUTTAL ==
I read the authors' response, and I am still slightly positive about the paper, though my major points were not addressed, but mostly deflected, e.g., referring to other papers that claim to show benefits of importance sampling. I think all the reviewers were curious how Bayesian optimization (BO) would perform. We understand numerical experiments are time consuming, but it's disappointing that you're not curious yourself whether your method outperforms BO. Your basic deflection seems to be that BO doesn't have provable guarantees, so because you do have guarantees, you don't need to compare with it.  Having one of the fastest methods among all methods with provable guarantees, but not necessarily the fastest method in general, sounds like a consolation prize to me.
The revision did not address some of the minor issues I mentioned, such as confusing "cite"/"cited"/"citep" issues in latex (which makes it hard to read).  My comment about Holder's inequality (vs Cauchy-Schwarz) applies not just to the $\|\cdot\|_1 \le \sqrt{d}\|\cdot\|_2$ bound, but also to the  $\|\cdot\|_2 \le \sqrt{d}\|\cdot\|_\infty$ bound in the next paragraph.
But despite a few items I'm being cranky about, I think it's still a solid paper, and it's still a weak accept.


Despite of the empirical success in various domains, it has been revealed that deep neural networks are vulnerable to maliciously perturbed input data that much degrade their performance. This is known as adversarial attacks. To counter adversarial attacks, adversarial training formulated as a form of robust optimization has been demonstrated to be effective. However, conducting adversarial training brings much computational overhead compared with standard training. In order to reduce the computational cost, we propose a simple yet effective modification to the commonly used projected gradient descent (PGD) adversarial training by increasing the number of adversarial training steps and decreasing the adversarial training step size gradually as training proceeds. We analyze the optimality of this annealing mechanism through the lens of optimal control theory, and we also prove the convergence of our proposed algorithm. Numerical experiments on standard datasets, such as MNIST and CIFAR10, show that our method can achieve similar or even better robustness with around 1/3 to 1/2 computation time compared with PGD.

The submission follows a recent line of work that formulates adversarial training (https://arxiv.org/abs/1412.6572) as a differentiable game and attempts to reduce the computational complexity of solving the inner maximization, corresponding to finding (an) adversarial example(s) for a given parameter setting. The authors propose a way to anneal the truncation of the inner iteration, and investigate a quantity to represent the suboptimality of the truncation.
Strengths:
- (From Table 3), the method is competitive with adversarial training (PGD; Madry et al. (2017)) as well as an improved method for adversarial robustness (FOSC; Wang et al. (2019)), while requiring half the computation time.
- The writing is very clear.
Weaknesses:
- Significance: The technique AMATA itself, is a  heuristic to gradually increase the length of the inner loop optimization in adversarial training. It is unrelated to the optimal control formulation.
- Novelty: Much of the motivation from the perspective of an optimal control formulation of adversarial training is similar to https://arxiv.org/abs/1803.01299, including the PMP and successive approximation components; Eq. (13) of the submission is the continuous version of Theorem 2 in this paper.
- Some experimental details are missing.
- Minor clarity issues (see below).
Questions for the authors:
- Could the authors better contrast their work in reference to prior work (https://arxiv.org/abs/1803.01299)?
- Could the authors clarify if they use random restarts for the PGD attacks? How was the number of iterations for the PGD attack (40) selected? Why was the iteration count of the PGD attack not varied?
Clarity issues:
- Please provide references for the optimal control formulation in 2.2.
- Some references to variables in the text are not made precise by identifying the variable in question (e.g.: "a set of optimal parameter choices"; "requires information on the entire training interval").
- The function of the hyperparameters \alpha, \gamma, K are not well discussed in Section 2.4.
- Reduce plot sizes and include more experimental details in the main text.


The variational autoencoder (VAE) has found success in modelling the manifold of natural images on certain datasets, allowing meaningful images to be generated while interpolating or extrapolating in the latent code space, but it is unclear whether similar capabilities are feasible for text considering its discrete nature. In this work, we investigate the reason why unsupervised learning of controllable representations fails for text. We find that traditional sequence VAEs can learn disentangled representations through their latent codes to some extent, but they often fail to properly decode when the latent factor is being manipulated, because the manipulated codes often land in holes or vacant regions in the aggregated posterior latent space, which the decoding network is not trained to process. Both as a validation of the explanation and as a fix to the problem, we propose to constrain the posterior mean to a learned probability simplex, and performs manipulation within this simplex. Our proposed method mitigates the latent vacancy problem and achieves the first success in unsupervised learning of controllable representations for text. Empirically, our method significantly outperforms unsupervised baselines and is competitive with strong supervised approaches on text style transfer. Furthermore, when switching the latent factor (e.g., topic) during a long sentence generation, our proposed framework can often complete the sentence in a seemingly natural way -a capability that has never been attempted by previous methods.

This paper presents a method for controlled text generation by using a new loss function (standard VAE loss with auxiliary losses added on). The method is tested on style transfer datasets: Yelp and Amazon. The central hypothesis is that when manipulating latent codes of a VAE, you can end up in low-density regions of the aggregated posterior. Such latent codes are rarely seen by the decoder so that quality of generation is low. To address this problem, they constrain the posterior mean to a learnt probability simplex and try to ensure that the simplex is densely filled. They do this by adding 2 regularizing losses to the VAE loss.
Some questions:-
- I had some trouble following section 4.2 where the derivation for L-reg is explained, this is the loss that constrains the posterior to a simplex. Is E = [e_1, ..., e_k] learnt or a hyperparameter? Most of the text seems to indicate that it is learnt, but you mention that alpha is a tunable hyperparameter, where alpha = e_i^2. Could you elaborate on how e_i^2 can be fixed to alpha is E is learnt?
- You state that mu^2 reaches a minimum at alpha/K. Could you please provide a proof? This could be placed in an Appendix.
- In section 6.3, is K still 3 or is it now >=4?
Some citations issues
- You're missing a couple key citations. I think your work should cite Hu et al.'s (2018) Toward Controlled Generation of Text.
- You should definitely cite Lample et al.'s (2019)  Multiple-Attribute Text Rewriting. The results in Lample et al. are definitely comparable, and often better than the results presented in this paper. While I think this paper makes an independent and useful contribution, some of the claims are overblown since the results from Lample et al. are not presented or discussed. For example, the Lample's model on Yelp achieves a higher accuracy and a lower perplexity than the CP-VAE model presented in this paper.
- Correction of citations: Kim et al. (2018), Adversarially regularized autoencoders, is in fact Zhao et al. (2018), you're just missing the first author.
Additionally, I'd like to see some more detail about the robustness of the method. Is the model robust to initialization and hyperparameter settings? Is posterior collapse always avoided? In Table 1 you show that without L-Reg the CP-VAE suffer with posterior collapse, but this would carry more weight if you showed some statistics on how often using L-Reg helps avoid posterior collapse. In general it looks like we're mostly presented with the best results, I'd like to know a little more about the mean and standard deviation of the model runs as well.
Overall, I think this paper makes a worthwhile contribution. I was unclear on a few parts of the derivation for the regularizing losses, but the intuition seems sensible. The presented results are impressive but have a few missing pieces: citing some key results (Lample et al. 2019) and including results about robustness. I think the questions I have, and any shortcomings this paper has, could be addressed in a camera-ready version.
Other bits,
- I do not understand figure 2. It's also never referred back to once we learn what v_p and v_n are. Some clarification here would be helpful.
- Section 5.1, line 3: achieve -> achieved


Learning with noisy labels has drawn a lot of attention. In this area, most of recent works only consider class-conditional noise, where the label noise is independent of its input features. This noise model may not be faithful to many real-world applications. Instead, few pioneer works have studied instance-dependent noise, but these methods are limited to strong assumptions on noise models. To alleviate this issue, we introduce confidence-scored instance-dependent noise (CSIDN), where each instance-label pair is associated with a confidence score. The confidence scores are sufficient to estimate the noise functions of each instance with minimal assumptions. Moreover, such scores can be easily and cheaply derived during the construction of the dataset through crowdsourcing or automatic annotation. To handle CSIDN, we design a benchmark algorithm termed instance-level forward correction. Empirical results on synthetic and real-world datasets demonstrate the utility of our proposed method.

--- Overall ---
I think the problem of incorporating information about labeling uncertainty (when such information is available) is an interesting and important problem; however, I think this paper contains a crucial misunderstanding of the definition of calibration (as defined in Guo et al. 2017), is missing some important parts of the literature, and does not provide adequate convergence guarantees for the proposed method.
--- Major comments ---
1. In section 3.1, the authors substitute the confidence score r_x for the conditional probability P(Y=y|\bar{Y}=y,X=x); however, even if the confidence scores are perfectly calibrated, these quantities are not necessarily equal (or even particularly close). As defined in Guo et al. (2017), a confidence score is well calibrated if the probability that a prediction is correct given the confidence score is equal to the confidence score. Using their notation: P(Y=\hat{Y}|\hat{P}=p) = p. Importantly, this is not conditioned on \hat{Y} or X. One way to see that these two quantities are not equal is to observe that there are many possible confidence score functions that satisfy the above definition whereas the conditional probability defined above is a unique function. In particular, if the confidence score function is a constant equal to the accuracy of the model, then the confidence score is well calibrated, but clearly not equal to the conditional probability above.
2. The other scenario considered by the authors is when there are multiple annotators; however, there is substantial literature on learning from multiple noisy annotations which the authors do not review. I suggest the authors start with "Modeling annotator expertise: Learning when everybody knows a bit of something" by Yan et al. (2010)  (and the citations therein) which also considers the instance dependent noise case.
3. Under what conditions does the proposed algorithm converge and to what does it converge to?
4. The proposed method relies on several assumptions that are scattered throughout the description of the algorithm. I highly recommend making these assumptions clear near the beginning of the paper. Specifically, my understanding is that the main assumptions are:
i. Confidence scores r_x are available for each instance and r_x \approx P(Y=y|\bar{Y}=y,X=x).
ii. Y _|_ X | Y\neq\bar{Y}, \bar{Y} = y
iii. Anchor points are available on some portion of the data.
Beyond the presentation, I find this to be a fairly strong set of assumptions, particularly the first assumption.
--- Minor comments ---
1. I really appreciated the synthetic example demonstrating the potential pitfalls of the small loss approach; however, I would spend a bit more time clearly explaining the small loss approach so that readers understand why it fails and how you are solving it's problems.
2. Also in the synthetic example, the authors state that covariate shift leads poor accuracy, however, I think this point would be stronger if demonstrated instead of just asserted.


Knowledge graphs (KGs) represent world's facts in structured forms. KG completion exploits the existing facts in a KG to discover new ones. Translation-based embedding model (TransE) is a prominent formulation to do KG completion. Despite the efficiency of TransE in memory and time, it is claimed that TransE suffers from several limitations in encoding relation patterns (such as symmetric, reflexive relations etc). To solve that, most of the attempts have circled around the revision of the TransE score function, resulting more complicated score functions (such as TransA/D/G/H/R etc). These attempts have totally disregarded effect of loss functions to this end. We show that loss functions are key factors in this regard and disregarding them results in conclusions which are inaccurate or even wrong. More concretely, we show that the claimed limitations are inaccurate, as the effect of the loss was ignored. In this regard, we pose theoretical investigations of the main limitations of TransE in the light of loss function. To the best of our knowledge, so far, this has not been comprehensively investigated. We show that by a proper selection of the loss function for training the TransE model, the main limitations are mitigated. That is achieved by setting upper-bound for the scores of positive samples, defining the region of truth (i.e. the region that a triple is considered positive by the model). Our theoretical proofs with experimental results fill the gap in understanding of the limitation of translation-based class of embedding models and confirm the importance of the selection of loss functions for training the models and on their performance.

The paper analyses the effect of different loss functions for TransE and argues that certain limitations of TransE can be mitigated by chosing more appropriate loss functions. Furthermore, the paper proposes TransComplEx -- an adaption of ideas from ComplEx/HolE  to TransE -- to mitigate issues that can not be overcome by a simply chosing a different loss.
Analyzing the behavior and short-comings of commonly-used models can be an important contribution to advance the state-of-the-art. This paper focuses on the performance of TransE, which is a popular representation learning approach for knowledge graph completion and as such fits well into ICLR.
Unfortunately, the current version of the paper seems to have issues regarding methodology and novelty.
Regarding the experimental evaluation: The paper compares the results of TransComplEx and the different loss functions to results that have previously been published in this field (directly, without retraining). However, it seems from Section 5 (Dataset), that this paper is using a modified dataset, as the TransE models are only trained on high-confidence triples. All prior work that I checked doesn't seem to do this, and hence the numbers are not comparable.
Even more serious: Following again Section 5 (Dataset), it seems that the paper imputes all missing triples in the training set for symmetric and transitive relations ("grounding"). Hence,  the models get to see _all_ true triples for these relation types and as such the models in this paper are trained on the test set.
Regarding novelty: The short-comings of TransE and improvements to the loss have been discussed quite extensively in prior work. Using complex representations in TransComplEx seems also a straightforward application of the insights of ComplEx/Hole. As such, the main novelty would lie in the experimental results which, unfortunately, seem problematic.


Off-policy reinforcement learning algorithms promise to be applicable in settings where only a fixed data-set (batch) of environment interactions is available and no new experience can be acquired. This property makes these algorithms appealing for real world problems such as robot control. In practice, however, standard off-policy algorithms fail in the batch setting for continuous control. In this paper, we propose a simple solution to this problem. It admits the use of data generated by arbitrary behavior policies and uses a learned prior -the advantage-weighted behavior model (ABM) -to bias the RL policy towards actions that have previously been executed and are likely to be successful on the new task. Our method can be seen as an extension of recent work on batch-RL that enables stable learning from conflicting data-sources. We find improvements on competitive baselines in a variety of RL tasks -including standard continuous control benchmarks and multitask learning for simulated and real-world robots. Videos are available at https: //sites.google.com/view/behavior-modelling-priors.

This paper proposes a method for “offline RL” (a.k.a “batch RL”), i.e. reinforcement learning from a given static dataset, with no option to perform on-policy data collection. Contrary to prior work (Fujimoto et al, Kumar et al, Agarwal et al) in this area which focuses on making Q-learning robust in the offline RL setting, the authors in this paper instead propose making the policy update step robust to the batch RL setting. They achieve this by constraining the policy that is being learned to stay close to a learned “prior policy” (using KL divergence as a distance metric). The authors provide two different ways of learning this prior policy  - the first one is a simple behavior model policy (which involves fitting a model using maximum likelihood on the entire dataset - effectively performing behavior cloning on the entire dataset), while the second is a more sophisticated “advantage behavior model”, which fits a model only on the “good” data (i.e. environment transitions with a positive advantage, where the advantage is estimated using the current policy). The authors extend the MPO and SVG algorithms to work with this constrained update step.
The authors provide experiments on the standard DM control suite tasks and some robotic tasks (both simulated and real world). For all experiments, the authors collect data by first running MPO in the standard RL setting (i.e. online data collection), and use the replay buffer from these successful RL runs to relearn a policy. The proposed method performs comparable to other methods in some settings (like all the experiments in the 2K episodes setting in Figure 2), and performs slightly better in some other settings (like Hopper and Quadruped in the 10K episodes setting). The gains are more significant in the non-standard robotics domains (Stack, Stack/Leave). The real world robotics results are not compared to any baseline (perhaps due to evaluation being expensive / time-consuming). However, I think the experimental results, especially Figures 7 and 8, point to a potential flaw in the proposed algorithm. In Figure 7 in the appendix, the learned behavior model (ABM) gets about the same performance as the proposed method (ABM+MPO) in pretty much all the four tasks. In one of the tasks (hopper), ABM+MPO does slightly better than ABM alone, but in this case plane MPO is better than ABM + MPO, indicating that this might just be a task on which MPO does well (for some unexplained reason). Similarly, in the robotics results in Figure 8, ABM alone does comparable to ABM+MPO in five of the seven tasks.
These experimental results imply to me that most of the performance actually comes from the learned prior (ABM), and whether performing MPO on top of it leads to any improvement or not is hard to predict^[1]. The ABM model itself is somewhat novel, and similar ideas under review at ICLR this year have shown that this alone is a useful algorithm for offline RL, see https://openreview.net/forum?id=H1gdF34FvS and https://openreview.net/forum?id=BJlnmgrFvS. However, the ABM model is not emphasized much in the paper. It is also unclear to my as to why the ABM/BM results were placed in the Appendix for Figure 3, while similar results for experiments in Figure 1 were provided using dotted lines.
For the paper to be accepted for publication, I think it needs to make a stronger argument (experimentally, at least) about the proposed algorithm being superior to ABM. If this is not really the case, then I think it would require a substantial rewrite to emphasize that ABM is where most of the model performance comes from (as seen in concurrent work listed above).
[1]: A slight caveat here: in the proposed Algorithm 1, the ABM model is learned online along with the policy, but I believe it could be learned without the policy - for example, by using MC returns as in https://openreview.net/forum?id=H1gdF34FvS.
----------------------
Edit: The author response has convinced me to bump my rating to a weak accept.