Neural network (NN) trojaning attack is an emerging and important attack that can broadly damage the system deployed with NN models. Different from adversarial attacks, it hides malicious functionality in the weight parameters of NN models. Existing studies have explored NN trojaning attacks in some small datasets for specific domains, with limited numbers of fixed target classes. In this paper, we propose a more powerful trojaning attack method for large models, which outperforms existing studies in capability, generality, and stealthiness. First, the attack is programmable that the malicious misclassification target is not fixed and can be generated on demand even after the victim's deployment. Second, our trojaning attack is not limited in a small domain; one trojaned model on a large-scale dataset can affect applications of different domains that reuses its general features. Third, our trojan shows no biased behavior for different target classes, which makes it more difficult to defend.

This paper proposed a "learnable" trojan by training a neural network that takes a sample (e.g. an image) as an input and generates a programmable trigger pattern as an output. The authors argue that the proposed method can support dynamic and out scope target classes, which are particularly applicable to backdoor attacks in the transfer learning setting. The authors conducted experiments on large-scale models (ImageNet) in two settings: (1) outsourced training attack;  (2) transfer learning attack.
Although the idea of making a backdoor attack more robust (e.g. more transferrable) and programmable is interesting, I don't think the current results fully substantiate the claimed benefits. Below are my concerns:
1. Lack of performance comparison: in the outsourced training attack, the attack success rate is quite low ( the best top-1 attack rate is ~50% on VGG). In the existing literature, the backdoor attack success rate can be made nearly 100%. This makes me wonder whether the low attack success rate only occurs in the proposed attacking method. Since there are no results from existing attacks, there is no way to evaluate how good the proposed attack is.
2. How about small dataset/model? I understand that the authors want to emphasize the scalability of their attack to large models like ImageNet. However, there are no comparisons on ImageNet (my comment 1). The authors are suggested to compare performance on standard datasets in backdoor attack literature (e.g. CIFAR-10, traffic sign)
3. It was unclear how "dynamic" the proposed method can be. Based on the attack formulation in equation (3), in order to make the attack "dynamic" in terms of changing different target classes, the attackers need to train a neural trojan network for every target class, which does not seem to be dynamic to me. Can the authors further justify the advantage of the dynamic feature in the proposed attack? And I have concerns about how many target classes an attacker can "dynamically" change. Some experiments showing the number of target classes vs attack performance and clean data accuracy will be very helpful.
4. The defense argument against detection methods is weak. Unless the authors can show the proposed attack has the ability to simultaneously backdoor all possible target classes, simply arguing the attack is dynamic and thus can evade detection is not convincing, not to mention in the backdoor setting, attacker should make the first move before the defender takes action.
*** Post-rebuttal comments
I thank the authors for the clarification. However, I feel my comments have not been fully addressed, especially on the part on justifying 50% attack success rate on VGG should be considered as significant in the considered setup. Without any valid comparisons, I find it difficult to assess the contributions. In addition, the authors did not add new empirical evidence regarding my questions but mainly re-iterated the applicability of the proposed method, so I will remain my review rating.
***


In this paper, we propose a novel regularization method, RotationOut, for neural networks. Different from Dropout that handles each neuron/channel independently, RotationOut regards its input layer as an entire vector and introduces regularization by randomly rotating the vector. RotationOut can also be used in convolutional layers and recurrent layers with small modifications. We further use a noise analysis method to interpret the difference between RotationOut and Dropout in co-adaptation reduction. Using this method, we also show how to use RotationOut/Dropout together with Batch Normalization. Extensive experiments in vision and language tasks are conducted to show the effectiveness of the proposed method. Codes are available at https://github.com/ RotationOut/RotationOut.

Paper Summary: This paper proposes a novel regularization method for training neural networks. The high-level motivation is to add noise (and thus regularize) neurons in an inter-dependent fashion, unlike existing methods such as DropOut where each neuron is treated independently. The authors evaluate their approach on image and speech classification, and object recognition benchmarks. They also discuss how different regularization schemes might help reduce neuron co-adaptation.
High-level comments: I find the proposed method interesting, and I think the paper is well-written. In particular, I like the exposition of their approach, as well as the common framing of different regularization schemes (in Section 3).
My chief concerns with the paper are:
1. The improvement over state-of-the-art, across tasks, seem marginal and largely is within the reported statistical error margins. From a practical standpoint, I think that alternatives to popular techniques like DropOut should either offer significant benefits empirically, or be computationally more feasible/simpler to implement. RotationOut does not seem to offer major benefits along either of these axes.
2. Even though I find the theoretical comparison of co-adaptation reduction in different regularization methods interesting, its significance is unclear. Based on my understanding, there is no concrete evidence that co-adaptation hampers training or generalization. Moreover, the authors also do not demonstrate that RotationOut actually decreases co-adaptation in a meaningful way---they propose a specific definition for co-adaptation but do not look at this quantity experimentally. There are a number of other proxies for mutual information in the community (such as CCA from arXiv:1806.05759) that the authors should also evaluate.
Given that improvements over prior art are small, I think the paper would have more significance if the authors demonstrate that reducing co-adaptation is important---from the perspective of training/generalization/interpretability and that RotationOut significantly helps with this compared to state-of-the-art approaches.
Other comments:
i. The authors mention that they fix rotate all feature vectors with the same direction but different angles: is the performance actually worse if the directions are also randomized?
ii. In general, I feel the authors should substantiate the various claims they make in Section 3 with experimental evidence. For instance, the authors mention that zero-centered Dropout-a might be more compatible with BatchNorm, but provide no experimental evidence to support this claim.
iii. How did the authors pick the hyperparameters and learning rate schedules for their experiments? These numbers do not seem standard (for instance the learning rate schedule)---did the authors grid over hyperparameters for each of the approaches? I think this is extremely important in papers which are proposing a new approach to establish a rigorous comparison to baselines.
iv. For several of the experiments, the authors report that they use RotationOut only for a few residual blocks. Do the remaining residual blocks have DropOut, or is no regularization applied to these? In the comparisons between different regularization schemes, are the regularizers applied to the same layers? For instance, in the comparison in Table 2, are all the methods applied only to Res3 and Res4?
v. For the results on COCO, is the same regularization method used even for ImageNet pre-training? For instance, does the “RetinaNet, RotationOut + ImageNet” row apply RotationOut even during the ImageNet pre-training? It would be nice to see the results of using “RetinaNet, keep prob = 0.9, block size = 5 + ImageNet” given that “RetinaNet, keep prob = 0.9, block size = 5” performs the best among prior approaches.
Overall, I find the idea of the paper interesting, I am not convinced of its significance. In particular, I think that the improvement over state-of-the-art is marginal and it is not clear that this method actually reduces co-adaptation between neurons in practice, or more broadly, that co-adaptation is a relevant quantity in deep network training. Thus, I am recommending rejection.


Expanding Non Markovian Reward Decision Processes (NMRDP) into Markov Decision Processes (MDP) enables the use of state of the art Reinforcement Learning (RL) techniques to identify optimal policies. In this paper an approach to exploring NMRDPs and expanding them into MDPs, without the prior knowledge of the reward structure, is proposed. The non Markovianity of the reward function is disentangled under the assumption that sets of similar and dissimilar trajectory batches can be sampled. More precisely, within the same batch, measuring the similarity between any couple of trajectories is permitted, although comparing trajectories from different batches is not possible. A modified version of the triplet loss is optimised to construct a representation of the trajectories under which rewards become Markovian.

** Summary
The paper studies a specific class of Non-Markovian Reward Decision Processes (NMRDPs). In general, in NMRDP the dynamics is Markovian but the reward function may depend on the entire trajectory. The authors consider a sub-case where the trajectory can be mapped to a specific "task" and the reward function can be formalized as a mapping between the state-task pair to the reward. This greatly simplifies the problem that can be mapped onto an augmented MDP and solved using standard RL tools. The authors focus on the representation learning problem of the mapping between trajectories to an embedding of the task itself. In particular, they consider contrastive learning to find a representation that discriminate between trajectories associated to the same task and trajectories coming from different tasks. The resulting LSTM-based architecture is then evaluated on a grid-world synthetic problem and on GPS trajectories from tourists in the city of Salzburg.
** Overall evaluation
My main concerns about the paper is that the exact objective of the setting and the representation learning is not that clear and the empirical validation is limited to a qualitative assessment of the representation learning with no down-stream task. More in detail.
1- The assumption of task-dependent reward functions is very sensible. Yet I wonder to which extent it overlaps with literature in multi-task and meta-learning, or more in general with embedding of time-series (the trajectory in your case). It would be useful to frame/compare at the conceptual level the similarities and differences of the proposed setting with those fields.
2- At training time, the PK dataset is somehow supervised, as it is possible to know which trajectories are associated to the same task. At test time, the learned f_theta recursively maps trajectories to tasks. As such, it seems like it could effectively detect changes into the task itself by tracking how the trajectory evolves (when the task function is unknown). This aspect is never really evaluated in the empirical section, but it would be one of the most interesting uses of the learned representation.
3- If the task function T is known, it means that standard RL techniques can be used to solve MDP N. The actual advantage of using a specific function to embed the task to a space in Re^d is never really explained in the paper. Does it make solving N simpler or more effective than using a simple encoding for the task? No evaluation is available in this sense.
4- The empirical evaluation is limited to qualitative analysis of the representation learned in the problem. Yet, there is no clear support that the representation is good/useful in a more quantitative way (e.g., by actually solving the RL part or in identifying quickly the current task).
Some more specific questions/comments
1- Some of the notation is a bit redundant. For instance, phi is mapping from H to Re^d, while f is a mapping from trajectories to Re^d, but in the end they are doing the same thing, as H itself can be obtained as a mapping from trajectories to tasks through T.
2- "Using RL techniques, finding an optimal policy pi* in the equivalent MDP \hat M is possible and by extension pi* is optimal for N". This passage is not fully clear, if phi is introducing some form of approximation, then pi* may no longer be optimal for the original MDP N. On the other hand, if phi is not approximating but "just" changing the representation from H to Re^d, then it is not clear what is the interest of it.
3- The assumption that similar trajectories can be identified is somehow strong. It would be good to have a more thorough support for it.
4- It is not fully clear what L_BH^local is indeed a local loss. It seems like you are simply using the mapping from the whole trajectory. Is this why it is called local?
5- While I appreciate that the introduction is sketching many different scenarios to support the models studied in the paper, in the end they mostly lack of depth and they rather give a confusing impression instead of clarifying in a compelling way what is the problem studied in the paper. I suggest you rather pick one single scenario with a good level of detail to provide a more solid support to the paper.
** Minor comments
1- You often use "he" to refer to the agent. It would be better to use "it" or "she".


Identifying anomalous samples from highly complex and unstructured data is a crucial but challenging task in a variety of intelligent systems. In this paper, we present a novel deep anomaly detection framework named AnoDM (standing for Anomaly detection based on unsupervised Disentangled representation learning and Manifold learning). The disentanglement learning is currently implemented by β-VAE for automatically discovering interpretable factorized latent representations in a completely unsupervised manner. The manifold learning is realized by t-SNE for projecting the latent representations to a 2D map. We define a new anomaly score function by combining β-VAE's reconstruction error in the raw feature space and local density estimation in the t-SNE space. AnoDM was evaluated on both image and time-series data and achieved better results than models that use just one of the two measures and other deep learning methods.

This paper presents a novel deep anomaly detection model. It combines two existing models: B-VAE and t-SNE. The B-VAE is trained unsupervised and learns an encoder and decoder which provide both an embedding and a reconstruction. Using t-SNE to reduce its dimensionality, the embedding is projected into a 2 dimensional space. An anomaly score function is defined that combines the reconstruction error and the distance in t-SNE space to the K nearest neighbor(s). Experiments are conducted with several image datasets (MNIST,FMNIST,CIFAR10,SmallNORB) and one timeseries dataset (Arrhythmia). For the image sets, the B-VAE model is implemented with a CNN, while for timeseries, a TCN is used. Comparisons are conducted showing the approach to be beat other SOT unsupervised methods, AnoGAN and ADGAN, by 63% and 22% respectively for MNIST and 8% and 2% for FMNIST (in terms of error reduction). For CIFAR-10 and FMNIST it is even demonstrated to beat a supervised SOT method CapsNET. Another experiment shows that t_SNE dramatically improves the performance over B-VAE alone. For the timeseries, the approach is not compared to other SOT approaches as the authors only provide an experiment showing that TCN beats CNN and LSTM for the implementation of the B-VAE. In addition the authors study the effect of the various parameters of the system, in particular the effect of the B in B-VAE and of alpha, the mixing factor between reconstruction error and kNN distance in t_SNE. 3D plots give a good idea on how to select optimal values for the various datasets. The impact of B is also shown on the t-SNE map for MNIST. Finally an ablation studies compares on MNIST the performance of the approach with t-SNE alone, reconstruction alone, and latent distance. On average over 4 digits taken as anomaly, the proposed approach dramatically outperforms the others.
PROS:
* The proposed approach improves over the SOT of competitive recent methods for anomaly detection on four image datasets.
* The authors make an effort to abstract the approach into a framework where other deep learning models and dimensionality reduction techniques can be used. They illustrate this by using a TCN instead of a CNN for the timeseries example.
* The parameter studies and ablation studies are informative and answer many of the questions i had as i read the paper.
* The paper is relatively clearly written (at least sufficiently to easily understand the technical details).
CONS:
* The novelty of the paper is limited as it is mostly a combination of 2 existing methods.
* The timeseries dataset is not compared to SOT methods (although the authors claim SOT in the conclusion).
* A pseudo-code algorithm is not provided, making it unlikely someone can reproduce the method.
* There are many typos an grammatical errors
* The paper could have been shortened. 10 pages is too long.
Overall, because of the good performance and thoughtful ablation studies, and despite the limited novelty, I think the paper makes a good contribution to anomaly detection.


When using distributed training to speed up stochastic gradient descent, learning rates must adapt to new scales in order to maintain training effectiveness. Re-tuning these parameters is resource intensive, while fixed scaling rules often degrade model quality. We propose AdaScale SGD, a practical and principled algorithm that is approximately scale invariant. By continually adapting to the gradient's variance, AdaScale often trains at a wide range of scales with nearly identical results. We describe this invariance formally through AdaScale's convergence bounds. As the batch size increases, the bounds maintain final objective values, while smoothly transitioning away from linear speed-ups. In empirical comparisons, AdaScale trains well beyond the batch size limits of popular "linear learning rate scaling" rules. This includes large-scale training without model degradation for machine translation, image classification, object detection, and speech recognition tasks. The algorithm introduces negligible computational overhead and no tuning parameters, making AdaScale an attractive choice for large-scale training.

This paper proposes a novel rule for scaling the learning rate, called the gain ratio, for when the effective batch size (induced by synchronous distributed SGD) is increased. This rule for the gain ratio uses estimates of the trace of the covariance and the norm of the true gradient to determine the appropriate steplength. This results in a method with a R-linear convergence guarantee to a neighborhood that is not dependent on $S$ (which is called scale-invariant). The algorithm additionally tracks the sum of the gain ratios in order to determine the "effective" number of iterations taken, and cut the steplength appropriately.
Strengths:
The gain ratio proposed in this paper is intuitive. I particularly like how the algorithm estimates the mean and variance information in an unbiased manner to determine an appropriate scaling for the steplength. The method is able to attain a R-linear rate of convergence and appears to perform well in practice for a wide variety of applications. The gain ratio is simple to estimate within a distributed framework.
Weaknesses:
I found some of the terms in the paper to be unclear or ill-defined. The original use of the term "scale" was unclear to me. Does this refer to the number of nodes in the distributed implementation? What is its relationship to batch size?
I found the definition of scale invariance in this paper to also be unclear on first read. The claim is that the algorithm is scale invariant if its final model does not depend on $S$. What is the "final model"? As currently defined, the current analysis does not guarantee that the algorithm will reach the same final model (assuming that $f(w, x) = \ell(h(w, x), y)$, i.e. a composition of a loss function and model), as the PL condition only ensures that one reaches a global minimum, which may not be unique. In fact, the analysis only guarantees convergence to a neighborhood. The description within the analysis appears to imply that scale-invariance is a property of the algorithm attached to its convergence property. Is this the case?
The definition of scale invariance is also already used in optimization to mean algorithms that are not modified when the objective is multiplied by a constant or an affine transformation. This adds to the lack of clarity, and I would suggest the authors use a different term for this kind of invariance (batch size invariant, or something like that?).
Is the theoretical comparison between SGD and AdaScale fair? Note that one can prove a stronger convergence result with SGD because one can actually attain a Q-linear rate of convergence to a neighborhood (for a proof, see for example, Bottou, et al. (2018)). In particular, one should have something like (in the paper's notation):
$$\mathbb{E}[F(w_T) - F^*] \leq (1 - \gamma)^T [F(w_0) - F^*] + \Delta.$$
This means that one can actually guarantee a fixed ratio of decrease in expectation to a neighborhood, whereas AdaScale converges linearly but not with a fixed ratio.
Some other small questions regarding the theory and experiments:
- Is there a reason why batch normalization was not tried for the CIFAR-10 experiments?
- Is it possible for $r_{t - 1} \gamma > 1$?
- Why was it necessary to estimate $\sigma^2 (w_t)$ and $\|\nabla F(w_t)\|^2$ by both aggregating at the current iteration and exponential averaging? What happens if exponential averaging is removed?
- What are the limitations of this method? How large of a batch size can one use with AdaScale before the algorithm breaks down (if at all)?
Additional Comments:
The algorithm is quite reminiscient of the steplength prescribed in Bollapragada, et al. (2018), which consider the steplength:
$$( 1 + \frac{\sigma^2(w_t)}{\|\nabla F(w_t)\|^2})^{-1}.$$
This gain ratio prescribed in this paper is the ratio between this quantity for two different batch sizes. Is there a clear explanation for why the relationship between these two quantities would arise?
This method could also be used for determining an appropriate scaling of the steplength in the sequential setting, when a larger batch size is used. Has this been considered?
Despite the concerns regarding the clarity in writing and the rigor in the theory of the paper, I think that the algorithmic idea proposed in this paper is interesting, novel, and practical. Because of the lack of clarity and rigor, I have given this paper a weak reject, but I would be happy to accept the paper if my concerns above were addressed in the final manuscript.


Deep neural networks (DNNs) are vulnerable to adversarial examples crafted by imperceptible perturbations. A range of defense techniques have been proposed to improve DNN robustness to adversarial examples, among which adversarial training has been demonstrated to be the most effective. Adversarial training is often formulated as a min-max optimization problem, with the inner maximization for generating adversarial examples. However, there exists a simple, yet easily overlooked fact that adversarial examples are only defined on correctly classified (natural) examples, but inevitably, some (natural) examples will be misclassified during training. In this paper, we investigate the distinctive influence of misclassified and correctly classified examples on the final robustness of adversarial training. Specifically, we find that misclassified examples indeed have a significant impact on the final robustness. More surprisingly, we find that different maximization techniques on misclassified examples may have a negligible influence on the final robustness, while different minimization techniques are crucial. Motivated by the above discovery, we propose a new defense algorithm called Misclassification Aware adveRsarial Training (MART), which explicitly differentiates the misclassified and correctly classified examples during the training. We also propose a semi-supervised extension of MART, which can leverage the unlabeled data to further improve the robustness. Experimental results show that MART and its variant could significantly improve the state-of-the-art adversarial robustness.

The paper essentially presented a viewpoint, i.e. misclassified examples may have a significant impact on the final robustness.
The authors conducted a series of qualitative experiments to verify
1) Misclassified examples have more impact on the final robustness than correctly classified examples.
2) For misclassified examples, different maximization techniques may have a negligible influence, but minimization techniques play a critical role on the final robustness.
3) The authors proposed a new defense algorithm which focus on generating adversarial examples from misclassified examples during the training. The algorithm was shown to improve the final robustness by revisiting these adversarial misclassified examples.
Generally speaking, the idea, though somewhat straightforward and less elegant, is reasonably presented and also well-motivated. Empirical validations seemed to support the idea and indicated that the proposed approach could improve the adversarial robustness.
There are several major concerns with the paper as follows:
1.	It is good and reasonable to put more emphasis on the misclassification examples, since it is likely that the region of the “perturbation” largely overlaps with the region of the misclassified examples.
2.	However, this may be dealt with in a more elegant or systematic way. In the viewpoint of the reviewer, actually a different emphasis can be imposed on each different data point including both the correctly-classified and mis-classified samples. In another word, a more systematical extension or a metric may actually be developed emphasizing more on mis-classified samples. It is suspected that different samples within the set of mis-classified samples (or even in the set correctly-classified samples) could also have a different influence. It is highly likely that a more elegant and mathematic way can be designed for this purpose.
3.	Further to point 2, in particular, the regulation w.r.t the correctly classified examples may not be ignored. It is interesting to consider differently the correctly-classified examples due to the trade-off between robustness and standard accuracy. This can also be seen in another submission of ICLR2020 (titled Sensible adversarial learning”) which actually discards the mis-classified samples. I would like to see some additional comments, clarification, or discussions on this point.
4.	I am curious to know if outliers would be over-emphasized by the proposed idea. Some discussion or even some illustrations on a synthetic case would be interesting. Will the existence of outliers affect the robustness?
5.	In Table 2, the best and the last results on FGSM are totally identically, I wonder if this is a wrong copy, which should be further clarified and discussed.
6.	 In the Unlabeled Data experiment part, it is better to compare the results with both UAT++ and RST at the same time rather than separately.
7.	Equation (4) shows that the two parts on the right side are added together, is that right? This is inconsistent with Equation (7) and the following description.
8.	In 3.2, the first paragraph, “max-margin adversarial training (MMA) (Ding et al., 2019)” appears to be wrong. The correct reference is “GavinWeiguang Ding, Yash Sharma, Kry Yik Chau Lui, and Ruitong Huang. Max-margin adversarial (mma) training: Direct input space margin maximization through adversarial training. arXiv preprint arXiv:1812.02637, 2018.”.
==================
I have read carefully the response from the authors. The newly added results appear satisfactory to me and I  am generally happy with the clarifications. I have then  adjusted my rating accordingly.


Learning to cooperate is crucial for many practical large-scale multi-agent applications. In this work, we consider an important collaborative task, in which agents learn to efficiently communicate with each other under a multi-agent reinforcement learning (MARL) setting. Despite the fact that there has been a number of existing works along this line, achieving global cooperation at scale is still challenging. In particular, most of the existing algorithms suffer from issues such as scalability and high communication complexity, in the sense that when the agent population is large, it can be difficult to extract effective information for highperformance MARL. In contrast, the proposed algorithmic framework, termed Learning Structured Communication (LSC), is not only scalable but also learns efficiently. The key idea is to allow the agents to dynamically learn a hierarchical communication structure, while under such a structure the graph neural network (GNN) is used to efficiently extract useful information to be exchanged between the neighboring agents. A number of new techniques are proposed to tightly integrate the communication structure learning, GNN optimization and MARL tasks. Extensive experiments are performed to demonstrate that, the proposed LSC framework enjoys high communication efficiency, scalability, and global cooperation capability.
For instance, the agent "G" finds the target (red square), then it will be possible to get a higher weight "4" and become the central. Right: The importance weight generation step and network construction step will be repeated iteratively. After communication and action procedures, agents will generate their new communication importance weights, and determine to keep or change their roles respectively. Further, the structured communication network will be re-established.
To address the above two issues, we propose a novel structured communication-based algorithm, called learning structured communication (LSC). Our LSC combines a structured communication network module and a communication-based policy module, which aims to establish a scalable hierarchically structured network and information exchange scheme for large scale MARL. In particular, a hierarchically structured communication network (Figure 1(d)) is dynamically learned based on local partial observations of agents. In the hierarchically structured network, all agents are grouped into clusters, where global collaboration can be achieved via intra-group and inter-group communications. In contrast to the other three types in Figure 1, the proposed hierarchical communication network is more flexible and scalable, with fewer resources needed to achieve long-range and global collaboration.
The procedure to establish such a hierarchically structured communication network is shown in Figure 2. To better utilize the relationship between agents given the hierarchically structured communication network and obtain more effective information extraction, graph neural network (GNN) (Scarselli et al., 2008) is employed. In GNN, each communication step involves information embedding and aggregation. Benefiting from the unordered aggregation power and the dynamic graph adaptability of GNN, the proposed LSC algorithm can extract valuable information effectively. The GNN-based information extraction procedure is depicted in Figure 3. This paper is devoted to the learning of communication structure among agents. To our knowledge, this is the first work of hierarchical structured learning to communication for MARL. It allows to learn communication structure adaptively instead of using predefined forms. Specifically: i) To improve scalability for a large number of agents, a hierarchical structure is devised that divides the agents into higher-level central agents and sub-level normal ones. As such, the communication network is sparsified. While it still allows for more effective global cooperation via message passing among the central agents, compared with the star/tree structures. ii) For effective communication and global cooperation, the message representation learning is deeply integrated into the information aggregating and permeating through the network, via graph neural network (GNN), which is a natural combination with the hierarchical communication structure. iii) Extensive experiments on both MAgent and StarCraft2 show our approach achieves state-of-theart scalability and effectiveness on large-scale MARL problems.

This paper proposes a method of learning a hierarchical communication graph for improving collaborative multi-agent reinforcement learning, particularly with large numbers of agents. The method is compared to a suitable range of baseline approaches across two complex environments. The initial results presented seem promising, but further work is needed to ensure the results are reproducible and repeatable.
To enable reproducibility, please include details of all hyperparameters used for all approaches in both domains. These should include justification of how the hyperparameters were tuned. Without understanding how these values were set I cannot support acceptance.
To ensure the results are repeatable, repeated runs of training should be completed and the variation in performance quantified in the results. These repeats may have already been performed as Table 3 and Figure 5 discuss average results, but if not they must be completed before the work can be published due to the known issues with high variance in performance that commonly occur in deep RL.
I would also argue against the justification of excluding ATOC from the StarCraft II experiments as its performance in MAgent with 25 agents is comparable to the other baseline methods that were tested. However, this is lower priority than the issues above provided there is no significant change in the relative performance of methods when the variance across multiple runs is documented in all existing experiments.
Minor Comments:
The following are suggestions for improvements if the paper is accepted or for future submissions.
In Section 2, centralised critic methods are grouped as "communication-free" however I don't think this is the best term to explain this approach as each agent has to communicate both its observations and actions to a centralised node (e.g. COMA) or all other agents (e.g. MADDPG). I also think this section should include coverage of other methods of utilizing graph neural networks in multi-agent reinforcement learning - e.g. "Deep Multi-Agent Reinforcement Learning with Relevance Graphs." Malysheva et al. Deep RL Workshop @ NeurIPS 2018 and "Relational Forward Models for Multi-Agent Learning" Tacchetti et al. ICLR 2019.
In Section 3, the acronyms CBRP and HCOMM are used on page 4 before they are introduced in full on page 5 for CBRP and never for HCOMM. HCOMM is also not used in Figure 4 or in the text description of the method. I believe it is the module described in Section 3.2 but this should be made clearer.
Many claims in the paper are worded too strongly and should be revised. In Section 2, it is claimed that DQN "is one of the few RL algorithms applicable for large-scale MARL" - However, there are now many successful applications of deep RL to multi-agent systems (some of which are cited earlier in this same section) that use a variety of algorithms other than DQN. It is also claimed in this section that "DQN has excellent sample efficiency" despite the sample efficiency of deep RL being a known issue, open research question and a barrier to its widespread use in practice.
In Section 3.2 the authors conclude "Overall, our LSC algorithm has advantage in the communication efficiency" despite in the same section noting two cases where ATOC has better efficiency (N_msg and N_b-r). I would suggest removing this sentence entirely as the paragraph above already contains a balanced account of the relative merits of each approach.
On pages 8 and 9 the authors make references to guarantees and in the Appendix to proofs that are not supported by theory only empirical results. Without supporting theory these words should be avoided.
The writing is also often informal to the detriment of presenting important information clearly. Notably, on page 3 "due to explosive growing number of agents" and on page 4 "The overall task of the MARL problem can be solved by properly objective function modeling." The second of these, particularly the word "solved" is also related to the issue above of using the words guarantee and prove.
Finally, the paper would benefit from a thorough grammar check. I note the following issues as simple changes that can be made to improve the readability of the paper:
- In the abstract, the sentence "but also communication high-qualitative" does not parse. Perhaps this could be shortened to the brackets following? i.e. "is not only scalable but also learns efficiently."
- Page 2, "or employing the LSTM to" -> "or use the LSTM to"
- Page 2, "still hinder the" -> "still hinders the"
- Page 2, "the communication structure need be jointly" -> needs to be jointly
- Page 6, "policy module motioned below" -> discussed below
- Page 10, "the map size are is 1920x1200" -> the map size is 1920x1200
- Page 10, "We do not compare with ATOC because its poor performance in MAgent" -> because of its
- Page 10, "we will to improve" -> we will improve
- Page 10, "practical constrains" -> practical constraints


To acquire a new skill, humans learn better and faster if a tutor, based on their current knowledge level, informs them of how much attention they should pay to particular content or practice problems. Similarly, a machine learning model could potentially be trained better with a scorer that "adapts" to its current learning state and estimates the importance of each training data instance. Training such an adaptive scorer efficiently is a challenging problem; in order to precisely quantify the effect of a data instance at a given time during the training, it is typically necessary to first complete the entire training process. To efficiently optimize data usage, we propose a reinforcement learning approach called Differentiable Data Selection (DDS). In DDS, we formulate a scorer network as a learnable function of the training data, which can be efficiently updated along with the main model being trained. Specifically, DDS updates the scorer with an intuitive reward signal: it should up-weigh the data that has a similar gradient with a dev set upon which we would finally like to perform well. Without significant computing overhead, DDS delivers strong and consistent improvements over several strong baselines on two very different tasks of machine translation and image classification.

The paper proposes an iterative method that jointly trains the model and a scorer network that places a non-uniform distribution over data sets.  The paper proposes a gradient method to learn the scorer network based on reinforcement learning, which is novel as to what the reviewer knows.
There are several concerns/questions:
1) The paper doesn’t define the D_{dev} clearly. How is D_{dev} chosen? Is it a subset of D_{train}?
2) In section 2.1, why “smaller development set D_{dev} is much closer to the P_{test}(X,Y)”? P_{test}(X,Y) is supposed to be not observed during training?
3) In Eq (5), if D_{dev} is s subset of D_{train}, if \theta* is the minimal of J, it means the gradient
at  \theta* is 0. To calculate the gradient of J with respect to \psi, by chain rule, it need to calculate gradient to \theta* first then \theta* to \psi. If gradient of \theta* is 0, the product is also 0? So the \psi will not be updated if D_{dev}  is sufficiently similar to D_{train} ?
4) In Section 2.3, it omits the second order Hessian term. How does that influence the performance?
5) it mentions “without significant computing overhead“ in abstract, which is not demonstrated elsewhere.
6) In the experiments, table 1, it seems the major improvement comes from retrain and TCS rather than DDS? In figure 3, it is better to show the weights of an image without DDS and comparing that with DDS.
7) The paper contains many typos such as Eqn.11 is not defined in the main paper, the “Eqn ??” Appears in the appendix, “tha minimizes” etc.
In general, the idea of the paper is natural and the results seem promising. I am looking forward to the reply to my questions/concerns.
#############
I have read the author's feedback. I think the clarity of both methodology and experiment does not reach the acceptance level and would maintain my current rating.


Sparsely available data points cause a numerical error on finite differences which hinder to modeling the dynamics of physical systems. The discretization error becomes even larger when the sparse data are irregularly distributed so that the data defined on an unstructured grid. It is hard to build deep learning models to handle physics-governing observations on the unstructured grid. In this paper, we propose a novel architecture named Physics-aware Difference Graph Networks (PA-DGN) that exploit neighboring information to learn finite differences inspired by physics equations. PA-DGN further leverages data-driven end-to-end learning to discover underlying dynamical relations between the spatial and temporal differences in given observations. We demonstrate the superiority of PA-DGN in the approximation of directional derivatives and the prediction of graph signals on the synthetic data and the real-world climate observations from land-based weather stations.

This paper proposes a method to reduce numerical error when predicting sequences governed by physical dynamics. The idea is that most physics simulators use finite difference differential operators, and the paper adds trainable parameters to them (the derivative and Laplacian operators). The added parameters are computed by a GNN. The paper concats the original graph feature and the output of the differential operators, and inputs them to a recurrent GNN to obtain the final prediction.
I think the idea is interesting. Incorporating modulated derivative and Laplacian operators into physical simulators is novel and well justified. It could strengthen the argument is there is more justification of why this particular parameterization is selected.
I think the experimental evaluation is somewhat adequate. There are a good selection of baselines including both manually designed iterators and GNNs. In particular, the weather prediction experiment show improved performance over several baselines. I am not familiar with this task or its state-of-the-art performance, but I am convinced that the proposed approach is superior compared to the claimed baselines (RGN, GRU).
I have several confusions or concerns about the synthetic experiments
1. In the synthetic experiments, is the evaluation task different from the training task? It is unclear from the description how well the learned parameters generalize. Does the method generalize to a. New functions/dynamics b. New graphs with similar properties (e.g. another graph draw from the same distribution) c. New graph with different properties (e.g. more or less sparse)?
2. One short-coming of the synthetic experiment is the lack of error bars, or analysis of statistical significance. I think some of the improvements are not large enough to be statistically convincing without additional analysis. It seems necessary to experiment on multiple random problems (e.g. with random meshing, dynamics parameters).
Minor comments:
A related idea is “Learning Neural PDE Solvers with Guarantees” which modulates the finite difference iterative solver with deep networks, but the objective is solving PDEs with known dynamics instead of prediction with unknown dynamics. Conversely, the method the authors proposed seem also useful for speeding up PDE solvers.
I think there is an error in the type definition of f and F in section 2.1. The two claimed types contradict each other.


In this work, we develop the theory for analyzing the generalization performance of recurrent neural networks. We first present a new generalization bound for recurrent neural networks based on matrix 1-norm and Fisher-Rao norm. The definition of Fisher-Rao norm relies on a structural lemma about the gradient of RNNs. This new generalization bound assumes that the covariance matrix of the input data is positive definite, which might limit its use in practice. To address this issue, we propose to add random noise to the input data and prove a generalization bound for training with random noise, which is an extension of the former one. Compared with existing results, our generalization bounds have no explicit dependency on the size of networks. We also discover that Fisher-Rao norm for RNNs can be interpreted as a measure of gradient, and incorporating this gradient measure not only can tighten the bound, but allows us to build a relationship between generalization and trainability. Based on the bound, we analyze the effect of covariance of features on generalization of RNNs theoretically and discuss how weight decay and gradient clipping in the training can help improve generalization.

This is an interesting and very well-written paper. I read the paper carefully but I don’t have sufficient expertise to determine whether all the proof steps are correct.
This paper builds on recent works giving generalization bounds on RNNs.
The primary contribution is that they are able to bound the generalization error for RNNs without a dependence on the network size parameters d and m.
Their proof is roughly as follows:
1. They decompose the RNN + ReLU activation into the sum of a linear network and difference terms. This step is key because it lets you treat each term independently when estimating the Rademacher complexity.
2. The linear network term can be bounded directly with their Fisher-Rao norm.
3. They make a second decomposition: the difference term can be written as a sum of simpler terms
4. For the simpler terms, their Rademacher complexity can be bounded independently using matrix-1 norm. Using the matrix-1 norm instead of the spectral norm means their bounds won’t depend on the network size parameters.
5. Then they combine these bounds to give the Rademacher complexity bounds for RNNs.
6. Lastly, they combine the Rademacher complexity bound with Kuznetsov et al 2015’s multiclass margin bound to give the generalization bound.
The downside to their generalization bound is that it requires the covariance matrix of the input data must be positive definitive, and it explodes when the smallest eigenvalue is close to zero.
Their second contribution attempts to address these downsides. They prove another generalization bound for RNNs when training with random noise, which has the effect of increasing the term containing the smallest eigenvalue of the input covariance matrix.
They remark on several empirical phenomena that are consistent with their results:
- Correlation of features in the input data makes it harder for RNNs to generalize
- Weight decay could help by decreasing the relevant gradient terms in their bounds
- Gradient clipping could help when the smallest eigenvalue of the input covariance is very small
Their third contribution is a single experiment. This contribution is fairly weak and the practical value of their theoretical work would be much more convincing if they were to put more effort into this section.
- They use IMDB data set (50k movie reviews + binary sentiment classification task)
- They add Gaussian noise to the input data with four different values.
- They plot the generalization error, which is the difference between the test error without noise and the training error with noise
Specific comments to improve their experiment section:
- I’m confused by the following sentence: “generalization errors … for different combinations of L and \sigma_epsilon are shown in Figure 1.” However, in Figure 1 I only see different values of sigma. I don’t see anything about using various values for L. This should be clarified.
- Figure 1 draws linear interpolation between data points - there’s no evidence for those interpolations. It should be reported as a scatter plot, preferably with error bars.
- If space limitations prevent reporting the experiment results more rigorously, I would prefer to see the experiment results reported in an appendix, with a brief comment on their significance in the main paper.


Prediction of future states of the environment and interacting agents is a key competence required for autonomous agents to operate successfully in the real world. Prior work for structured sequence prediction based on latent variable models imposes priors with limited expressiveness or are difficult to optimize e.g. determining the number of Gaussian mixture components which makes it challenging to fully capture the multi-modality of the distribution of the future states. In this work, we introduce Conditional Flow Variational Autoencoders (CF-VAE) using our novel conditional normalizing flow based prior to capture complex multi-modal conditional distributions for effective structured sequence prediction. Moreover, we propose two novel regularization schemes which stabilizes training and deals with posterior collapse for stable training and better fit to the target data distribution. Our experiments on three multi-modal structured sequence prediction datasets -MNIST Sequences, Stanford Drone and HighD -show that the proposed method obtains state of art results across different evaluation metrics.

The paper proposes a combination of conditional VAE wtih normalising flows priors and posterior regularisation strategies to capture the diversity of multi-modal trajectories of complex motion patterns. The paper argues that more flexible priors over the latent space can provide posteriors that more closely resemble the trajectories observed in the training data. To this end, the paper presents a derivation of the evidence lower bound for VAEs with normalising flows and discusses the effect of fixing the variance of the posterior to reduce instability during training. Additionally, it shows that conditioning the regularisation on whether or not the dataset contains a dominating mode leads to more diversity and captures minor modes more effectively. Experiments are reported on sequence datasets of handwritten digits, and two datasets with trajectories of vehicles in traffic.
A central point the paper makes is the importance of prior distributions for the latent space in VAEs such that it can capture diverse modes of trajectories. It is well known that more flexible priors such as MoG lead to better generative power as shown in Tomczak and Welling, 2018. The paper focuses on an extension of the work by Ziegler and Rush, 2019 which proposes normalising flows as priors to capture sequences, conditioned on the initial part of the trajectory. This extension is relatively simple, but does address the specifics of the problem well.
In general, the paper is well written and clear. The main innovation, in my opinion, is the combination of several ideas applied to the problem of sequence prediction. While none of the ideas, in isolation, are significantly new, the combination can be useful to this particular problem. However, I would like feedback from the authors on the following two main points below which are the main weaknesses of the paper:
1) Posterior regularisation: The posterior regularisation strategies, while intuitive, are very ad-hoc and somewhat contrary to the Bayesian framework. It is difficult to see how the variance in pR and the "dominant mode" detector in cR can be estimated automatically. Within a Bayesian framework it would be much more natural to place a prior distribution over the variance and marginalise it out within the variational inference procedure. For the other regularisation (cR), how is the dominant mode detected?
2) Experiments: A major concern reported throughout the paper is the instability of training and the risk for overfitting. I do not think the experiments demonstrate how stable and robust the method is to different initialisations, seeds, training data shuffles, etc. I strongly suggest the authors to run cross validation experiments and report the mean and standard deviation for all methods being compared. Also, how sensitive are the results to different values of C? How to decide whether to use cR or not when don't have access to the ground truth?


We propose a framework for defining a Boolean algebra over the space of tasks. This allows us to formulate new tasks in terms of the negation, disjunction and conjunction of a set of base tasks. We then show that by learning goal-oriented value functions and restricting the transition dynamics of the tasks, an agent can solve these new tasks with no further learning. We prove that by composing these value functions in specific ways, we immediately recover the optimal policies for all tasks expressible under the Boolean algebra. We verify our approach in two domains-including a high-dimensional video game environment requiring function approximation-where an agent first learns a set of base skills, and then composes them to solve a super-exponential number of new tasks.

This paper proposes a new framework for defining Boolean algebra over the space of tasks in goal conditioned reinforcement learning and thereby achieving composition of tasks, defined by boolean operators, in zero-shot. The paper proves that with some assumptions made about a family of MDP’s, one can build Boolean algebra over the optimal Q-functions of the individual MDP and these Q-functions are equipped with all the mathematical operations that come with the Boolean algebra (e.g negation, conjunction). The paper verify their theoretical results by experiments in both the 4-room domain with standard Q-learning and in a simple video game domain with high-dimensional observation space and DQN. The proofs of all the theoretical results seem sound and the experiments support the theory. I enjoyed reading this paper as the paper is generally well written and the idea is quite neat.
That being said, I have a few concerns and questions about the paper that I would like the authors to respond to so I am leaning towards rejecting this paper at the moment. However, I will raise my score if the revision addresses my concerns or provide additional empirical evidence. My concerns are the following:
    1. My biggest concern is whether boolean algebra is the right abstraction/primitive for task level composition. Thus far, the most important application of boolean algebra has been in designing logic circuits where the individual components are quite simple. In the proposed framework, it seems that all of base tasks are required to be a well defined task which are already quite complex, so the utilities of composing them seems limited. For example, in the video game domain the author proposed, a very reasonable base task would be “collect white objects” -- this task when composed with the task “collect blue objects” is meaningless. This seems to be true for a large number of the MDP’s in the super-exponential composition. Furthermore, [1] also considers task level composition with sparse reward but I think these compositions cannot be expressed by boolean algebra. One of the most important appeal of RL is its generality so It would be great if the author can discuss the limitations of the proposed framework and provide an complex/real-world scenarios where composing these already complex base tasks are useful. Just writing would suffice as I understand setting up new environments can be difficult in short notice (Of course, actual experiments would be even better).
    2. Does the maze not change in the environment setup? (It would be nice if source code is provided) If that is the case I would like to see additional experiments on different mazes (i.e. different placement of walls and objects). In my opinion, if there is only a single maze, then the only thing that changes is the location of the agent which makes the task pretty easy and do not show the full benefit of function approximators. I think it’d strengthen the results if the framework generalizes to multiple and possibly unseen mazes.
    3. In the current formulation, a policy is discouraged to visit goals that are not in its current goal sets (receives lowest reward). While this could be just a proof artifact, it can have some performance implications. For example, in the 4 room domain, if I place a goal in the left corridor, then the agent in the bottom left room will need to take a longer route to reach top left (bottom left -> bottom right -> top right -> top left) instead of the shorter route (bottom left -> top left). From this perspective, it seems some non-trivial efforts need to be put into designing these "basis" tasks. I am curious about the discussion on this as well.
    4. Haarnoja et al. 2018 and other works on composing Q values can be applied to high-dimensional continuous control using actor-critic style algorithms and relies on the maximum entropy principle. Can the method proposed in this paper be used with actor-critic style? Is the max-entropy principle applicable here as well? Discussion would be great and experiments would be even better.
Out of all my concerns, 1 matters the most and I am willing to raise my score to weakly accept if it’s properly addressed. If, in addition, the authors could adequately address 2-4 I will raise my score to accept.
=======================================================================
Minor comments that did not affect my decision:
    - In definition 1, it would be nice to define r_min and r_max and g \ne s \in \mathcal{G} is also somewhat confusing.
    - In definition 2, \pi_g is never defined
Reference:
[1] Language as an Abstraction for Hierarchical Deep Reinforcement Learning, Jiang et al. 2019


The success of reinforcement learning in the real world has been limited to instrumented laboratory scenarios, often requiring arduous human supervision to enable continuous learning. In this work, we discuss the required elements of a robotic system that can continually and autonomously improve with data collected in the real world, and propose a particular instantiation of such a system. Subsequently, we investigate a number of challenges of learning without instrumentation -including the lack of episodic resets, state estimation, and hand-engineered rewards -and propose simple, scalable solutions to these challenges. We demonstrate the efficacy of our proposed system on dexterous robotic manipulation tasks in simulation and the real world, and also provide an insightful analysis and ablation study of the challenges associated with this learning paradigm.

The paper takes seriously the question of having a robotic system learning continuously without manual reset nor state or reward engineering. The authors propose a first approach using vison-based SAC, shown visual goals and VICE, and show that it does not provide a satisfactory solution. Then they add a random pertubation controller which brings the robot or simulated system away from the goal and a VAE to encode a compressed state, and show that it works better.
The paper is a nice read, it contains useful messages thus I'm slightly in favor of accepting it, but I may easily change my mind as it suffers from serious weaknesses.
First, and most importantly, the experimental study is very short, the authors have chosen to spend much more space on careful writing of the problem they are investigating.
To mention a few experimental weaknesses, in Section 6.2 the authors could have performed much more detailed ablation studies and stress in more details the impact of using the VAE alone versus using the random pertubation controller alone, they could say more about the goals they show to the system, etc. There is some information in Figure 7, but this information is not exploited in a detailed way. Furthermore, Figure 7 is far to small, it is hard to say from the legend which system is which.
About Fig.8, we just have a qualitative description, the authors claim that without instrumenting they cannot provide a quantitative study, which I don't find convincing: you may instrument for the sake of science (to measure the value of what you are doing, even if the real-world system won't use this instrumentation).
So the authors have chosen to spend more space on the positionning than on the empirical study, which may speak in favor of sending this paper to a journal or magazine rather than a technical conference. But there is an issue about the positionning too: the authors fail to mention a huge body of literature trying to address very close or just similar questions. Namely, their concern is one the central leitmotives of Developmental Robotics and some of its "subfields", such as Lifelong learning, Open-ended learning, Continual learning etc.  The merit of the paper in this respect is to focus on a specific question and provide concrete results on this question, but this work should be positionned with respect to the broader approaches mentioned above. The authors will easily find plenty of references in these domains, I don't want to give my favorite selection here.
Knowing more about the literature mentioned above, the authors could reconsider their framework from a multitask learning perspective: instead of a random perturbation controller, the agent may learn various controllers to bring the system into various goal states (using e.g. goal-conditioned policies), and switching from goal to goal to prevent the system fro keeping stuck close to some goal.
More local points:
In the middle of page 5, it is said that the system does not learn properly just because it is stuck at the goal. This information comes late, and makes the global message weaker.
in Fig. 4, I would like to know what is the threshold for success.


Learning home appliance usage is important for understanding people's activities and optimizing energy consumption. The problem is modeled as an event detection task, where the objective is to learn when a user turns an appliance on, and which appliance it is (microwave, hair dryer, etc.). Ideally, we would like to solve the problem in an unsupervised way so that the method can be applied to new homes and new appliances without any labels. To this end, we introduce a new deep learning model that takes input from two home sensors: 1) a smart electricity meter that outputs the total energy consumed by the home as a function of time, and 2) a motion sensor that outputs the locations of the residents over time. The model learns the distribution of the residents' locations conditioned on the home energy signal. We show that this cross-modal prediction task allows us to detect when a particular appliance is used, and the location of the appliance in the home, all in a self-supervised manner, without any labeled data.

# Review ICLR20, Self-Supervised Learning of Appliance Usage
This review is for the originally uploaded version of this article. Comments from other reviewers and revisions have deliberately not been taken into account. After publishing this review, this reviewer will participate in the forum discussion and help the authors improve the paper.
## Overall
**Summary**
The authors introduce a new method for classifying which appliance was turned on by looking at the change in total household electricity consumption and tracking people's coarse positions in the house.
**Overall Opinion**
The paper is concise and interesting, but for now, I have to reject it because the same major things aren't clear to me:
- Why are you doing this, to begin with, i.e. "... to optimize energy consumption"?. What does energy consumption have to do with classifying which appliance was turned on? There might be the rare edge case of a fridge door left open accidentally but that can't be the main argument here, can it?
- Now, assuming that there is a benefit to this, why can't we use the smart plugs all around? You already use them to provide ground truth. But I imagine they are cheap and easy enough to distribute at large scale, whereas the location sensor was custom-built by you if I understood correctly and probably has to be calibrated for every house.
Other than that, the paper is pretty good. Here are some minor...
## Specific comments and questions
- I got slightly disappointed when I picked up the paper because I was in a robotics mindset and assumed "Self-supervised learning of appliance usage" was some new way for a robot to analyze appliances and push buttons. I'd add a "human" in the title for clarity, e.g. "Self-supervised Learning of Human Appliance Usage" or "Self-supervised Learning of Appliance Usage from Human Location Data". But that's just my opinion, no pressure on this one.
### Abstract
- Please never repeat sentences from the abstract in the introduction or vice versa, good as they might be.
### Intro & Rel. Work
all good
### Problem Formulation
- Please always provide a proper caption for your figure so that the caption contains enough information for the reader to understand what's going on in the figure without having to search for it in the main text. This applies to Fig.1 and even more so to Fig.2 where I'd recommend explaining in broad strokes the information flow.
- Fig. 1 (a) Why is there higher power usage between 02-06 compared to 14-18 o'clock? Is that a heater as the outside temperatures drop overnight? (c/d) I don't think that's a good way of visualizing this. I don't get anything out of diagram (c) and I'd prefer it if you could encode the time aspect in the diagram (d) as color/alpha. E.g. blue agent's oldest position is almost imperceptibly white, growing bluer and darker as the data points get closer to the final step in the recording.
### Model
- Doesn't the softwax require to know the number of appliances a priori?
- How do you deal with $\alpha$ and $P_t$ when a person is at the edge of the visible space? Does this cause problems during training or did this never occur? I could imagine, since the location is noisy, a person could jump in and out of the visible area and cause training instability.
### Datasets
all good
### Results
- What's "Door / leaving" in Table 2? Which appliance is this?
- I think the standard at ICLR is to have the table and figure captions both below the respective objects
- The last 3 sections, 6.3 to 6.5, read a bit rushed and might need a rewrite for clarity.
- Fig. 5 is nice. But shouldn't the signal add up to explain the entire scene? There's a bit of blur next to the couch. Is this a training set vs test set artifact?
### Conclusion
- This needs to be developed a bit more: what are some downsides to your method? Where does it NOT work or what makes it prone to error? And what are future directions based on this work? What's worth researching further or have you solved appliance use detection for good?
---
I'm sure there are some good answers for my main concerns outlined at the beginning and after that, I'd be happy to adjust my rating. The rest are minor issues. Like I mentioned, save for page 8, the paper is well-written and focused.


We investigate new methods for training collaborative filtering models based on actor-critic reinforcement learning, to more directly maximize ranking-based objective functions. Specifically, we train a critic network to approximate ranking-based metrics, and then update the actor network to directly optimize against the learned metrics. In contrast to traditional learning-to-rank methods that require re-running the optimization procedure for new lists, our critic-based method amortizes the scoring process with a neural network, and can directly provide the (approximate) ranking scores for new lists. We demonstrate the actor-critic's ability to significantly improve the performance of a variety of prediction models, and achieve better or comparable performance to a variety of strong baselines on three large-scale datasets.

The work presents a two-level architecture for building an efficient ranking solution for standard collaborative filtering task in recommender systems. The core of this solution is the actor-critic approach, where the critic (motivated by ideas from RL) tries to directly approximate a ranking-based metric using as an input prediction from the actor (VAE framework) and the ground truth knowledge from an Oracle. Both actor and critic require a pre-training step, after which the critic is set to propagate adjustments to actor parameters. Experiments demonstrate the advantage of such approach over competing methods.
The work presents an interesting idea of feature-based critic for learning a better ranking. The choice of features is surprisingly effective considering how simple they are – just basic statistics. On the other hand, the motivating example with NLL vs NDCG is not convincing. Comparing to much simpler linear SOTA methods, the improvement over multinomial VAE is not impressive. This is probably due to VAE being not a real SOTA in the first place, however other possible candidates for actor were not demonstrated. Anyway, I believe that this is an important direction for research and can be considered for acceptance if the authors adjust the text according to the comments below.
Main arguments
The text is well structured, easy to read. Key concepts are explained carefully with the appropriate level of details. Open-source repository with the source code is definitely a plus. However, there’s a number of inaccuracies, which should be addressed.
In the example demonstrating VAE's pitfalls, it is stated that prediction A is intuitively better than prediction B. I’d argue with that intuition. For me, B looks more consistent as there’s a smaller gap between predictions for positive targets. In A one of the predictions is much lower, whcih seems more like an outlier. Moreover, shouldn’t NDCG be 1 in both cases? The ordering of positive examples is the same in both cases. Please, consider addiing more explanations to clarify your intuition or replace the example with something less contradicting.
The  statement “making RaCT scalable to large-scale datasets” is arguable from the technical viewpoint. As far as I know, there’s still no good support for sparse calculations in NN frameworks. When you take user mini batch you have to convert it to a dense matrix with explicitly stored zeros. It’s not a problem for the datasets tested in the paper (with <100K items). However, consider 100M items. What will be the size of user batch that will fit GPU memory? Will it be still large enough and effective from the learning viewpoint (smaller batches are likely to reduce generalization)? As it will mostly contain zeroes (few know interactions vs ~100M unknown), efficiency of calculations with this batch will also be low comparing to operations in sparse format. In that sense the datasets used in the paper are not really large scale.
Another false statement: “This is much more efﬁcient than most traditional latent factor collaborative ﬁltering models …, where a time-consuming optimization procedure is typically performed to obtain the latent factor for a user who is not present in the training data”. All factorization models admit folding-in procedure, which doesn’t depend on the size of the problem (i.e., the number of users or items) and depends only on the rank of decomposition (the number of latent variables). This is among the reasons why MF is so popular in industrial applications.
Furthermore, SVD-based models (I’m talking about the real SVD, not MF models like SVD++) are absolute winners here as they do not even require running any additional optimization steps. Due to the orthogonality property, the solution for folding-in has an analytical form in the case of SVD, e.g., given a low-rank item embeddings matrix V with orthonormal columns, the folding-in solution (vector of predicted item scores for a user) is simply given by an orthogonal projector
r = VV^T p,
for any vector of user preferences p, be it a known user or a new one (i.e., not present in the training data). It is basically a simple encoder-decoder architecture with a single linear layer. Training SVD-based models can also very efficient and can be scaled to almost billion-size problems, e.g. https://github.com/criteo/Spark-RSVD. Moreover, one can significantly improve standard PureSVD model [Cremonesi, Koren, Turrin 2010] by a simple scaling trick as it is shown in the EigenRec model [Nikolakopoulos 2019], you can also see more details on this in the blog https://www.eigentheories.com/blog/to-svd-or-not-to-svd. I also find it interesting that the scaling trick in SVD partially intersects with the idea of importance of user behavior statistics in the critic model,  as it also depends on how many clicks a user has made.
I ran my own experiments with VAE and SVD in google colab (I can provide a link to the notebook if needed). With a proper tuning the SVD-based solution gives NDCG=0.415, which is within 3% difference of the Mult-VAE’s best result on the ML-20M dataset. It takes around 200s to train the SVD model of rank 600 on the CPU in the google colab. For comparison, one epoch of VAE training takes 26s in the same google colab environment (which I believe is using Tesla K80 GPU). The same NDCG level is achieved by Mult-VAE after approximately 20 epochs, which amounts to more than 500s of training time, twice longer than SVD. Speaking about practicality, even a student will have a decent CPU nowadays, but not everyone will have Tesla K80. SVD merely requires doing `from scipy.sparse.linalg import svds` in most cases. Moreover, there are even better solutions than SVD (in terms of accuracy), still linear, not even based on MF, e.g., RecWalk [Nikolakopoulos and Karypis 2019] and even better one - Personalized Diffusions model [Nikolakopoulos, Berberidis, Karypis, Giannakis 2019] - that are shown to significantly outperform VAE models.
The general conclusion is that even though you improve VAE, you are actually starting from not the best model and there’s much more room for improvement (demonstrated by much simpler models) than you show in your work. Please consider this fact and adjust wording in your text accordingly. Especially in the statements like “VAEs signiﬁcantly outperform many state-of-the-art methods” or “the proposed RaCT also signiﬁcantly outperforms other state-of-the-art methods, including VAE, CDAE, WMF and SLIM” as it is really about outperforming strong baselines not real SOTA.
In addition to that, recent studies show that many modern neural networks-based approaches underperform even simple KNN-based models, see work by [Dacrema, Cremonesi, Jannach 2019] on “A Worrying Analysis of Recent Neural Recommendation Approaches”. It includes VAE, DAE and NeuCF that you reference in your paper.
In spite of that, the statement “[LVM]… may yield suboptimal performance, especially for large datasets (He et al., 2017)” is at best questionable and requires more investigation as the referenced here paper is about NeuCF. This also makes me doubt in the claims, where you talk about approximation of the real ranking measure by the critic. Examples with RecWalk and PersDiff show that even without optimizing any ranking metric you can get a decent quality in terms of NDCG. Even though I find the general explanation in the text intuitive and appealing, I believe, more rigorous research should be done before we can have something reliable to say about ranking metric approximation. I’d suggest being more careful with such statement in the text. Even in your own experiments a simple EASE model is sometimes better, even though it has nothing to do with approximating any elaborate ranking metric (it's closer to the nearest neighbours search).
Other comments
1)	“performance comes at a computational cost, as inference requires multiplication with an unfactored M ×M matrix” is arguable to a certain extent. In the case of SLIM there’s a sparse matrix with zero diagonal, so once it’s computed and stored in a sparse format, it’s not really a problem to do calculations with it (the training is expensive, though).
2)	 “which is argued to be a close proxy to ranking loss” a reference required here.
3)	The two-level architectures are very popular in general. At the first level, a compact list of candidates is generated by a quick to compute algorithm like MF,  then the second-level algorithm based on decision-trees or multiarmed bandits (MAB) is trained on a much smaller output space. Comparing with such approaches (especially with MAB as they are widely used in industry, probably the second popular approach after MF) would make the work more complete.
4)	“Previous work on learning-to-rank has been explored this question…” something is not right here.
5)	Unfortunately, a lot of prior studies do not report confidence intervals, and this paper is not an exception. It makes it hard to reason about statistical significance of the results and about generalization capabilities of the approach.
References:
Cremonesi, Paolo, Yehuda Koren, and Roberto Turrin. "Performance of recommender algorithms on top-n recommendation tasks." In Proceedings of the fourth ACM conference on Recommender systems, pp. 39-46. ACM, 2010.
Nikolakopoulos, Athanasios N., Vassilis Kalantzis, Efstratios Gallopoulos, and John D. Garofalakis. "EigenRec: generalizing PureSVD for effective and efficient top-N recommendations." Knowledge and Information Systems 58, no. 1 (2019): 59-81.
Nikolakopoulos, Athanasios N., and George Karypis. "Recwalk: Nearly uncoupled random walks for top-n recommendation." In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, pp. 150-158. ACM, 2019.
Nikolakopoulos, Athanasios N., Dimitris Berberidis, George Karypis, and Georgios B. Giannakis. "Personalized diffusions for top-n recommendation." In Proceedings of the 13th ACM Conference on Recommender Systems, pp. 260-268. ACM, 2019.
Dacrema, Maurizio Ferrari, Paolo Cremonesi, and Dietmar Jannach. "Are we really making much progress? A worrying analysis of recent neural recommendation approaches." In Proceedings of the 13th ACM Conference on Recommender Systems, pp. 101-109. ACM, 2019.


To flexibly and efficiently reason about temporal sequences, abstract representations that compactly represent the important information in the sequence are needed. One way of constructing such representations is by focusing on the important events in a sequence. In this paper, we propose a model that learns both to discover such key events (or keyframes) as well as to represent the sequence in terms of them. We do so using a hierarchical Keyframe-Inpainter (KEYIN) model that first generates keyframes and their temporal placement and then inpaints the sequences between keyframes. We propose a fully differentiable formulation for efficiently learning the keyframe placement. We show that KEYIN finds informative keyframes in several datasets with diverse dynamics. When evaluated on a planning task, KEYIN outperforms other recent proposals for learning hierarchical representations.

This paper introduces a variational objective to train a model which can jointly select keyframes of a video and generate the intervening frames to produce a resultant video. The model is provided an initial set of frames as context. At training time the model always learns to produce N*J frames, where N is the number of keyframes and J is a fixed number of frames to generate for each keyframe. The authors compare their method for selecting informative keyframes on a number of baselines and show an improvement over these baselines. The problem is interesting and well-motivated, but I have some concerns with the proposed approach and experiments. As such, I am a weak reject.
comments / questions:
- Equation 3 lacks context. Initially, when looking at the authors' objective it seems that the inner expectation should be taken with respect to the joint time indices for the current and next keyframe. Only later after equation 4 do they mention that they always predict a fixed number of frames J.
- The need for normalizing over the first T timesteps in equation 4 seems quite messy. Is it guaranteed that all of the needed keyframes will actually be within the first T timesteps? How does this work in practice?
- Many important details of the inference procedure are relegated to the appendix. For example, there are no details for extracting which of the 60 keyframes that were trained for a sequence (due to the fixed length sequences) should be selected at test time. Looking at the appendix, it is clear that the approach requires an extensive planning algorithm at inference time, which seems like an important component.
- The authors prominently highlight that their method is fully differentiable, yet they train in two stages while freezing weights. Why isn't the model trained end-to-end? The stated reason for doing so is that this "simple" two-stage procedure improves optimization. What exactly happens if you don't do this two stage training process? Does it fail to learn? Some experimental numbers would be nice to see.
- The authors do not compare their method to any strong keyframe prediction baselines. Considering there is existing work in keyframe prediction, it seems important to highlight the difference between other competing models, rather than relying on simple baselines. Why don't they use self-information/surprisal as a baseline i.e., by training an autoregressive model on the frames and then picking the N frames with the largest -log(p)? This is a metric that has been investigated frequently and has better interpretability than defining a new measure of surprise. Note that Kipf et al. (2019) uses this notion of surprisal as well.
- Sauer et al. (BMVC 2019) should likely be cited as it does very similar keyframe analysis. Also, as the ICML 2019 conference had already concluded by the ICLR submission deadline, is it really fair to state the work with Kipf et al. (2019) was conducted in parallel?
- Why does the model trained to learn a fixed number of timesteps for the intermediate frames? Did they investigate jointly predicting the indices for the current and next timesteps? It seems like it would greatly simplify their inference scheme if they did this. If they tried that approach and it failed, maybe that should be mentioned in the paper (with an explanation as to why it fails).
- In the literature review, when discussing hierarchical temporal structure, the authors state: "However, these models rely on autoregressive techniques for text generation and are not applicable to structured data, such as videos." Autoregressive techniques have been investigated in relation to videos; in fact, the authors later describe papers that have used autoregressive techniques for modeling videos.


We consider off-policy policy evaluation when the trajectory data are generated by multiple behavior policies. Recent work has shown the key role played by the state or state-action stationary distribution corrections in the infinite horizon context for off-policy policy evaluation. We propose estimated mixture policy (EMP), a partially policy-agnostic methods to accurately estimate those quantities. With careful analysis, we show that EMP gives rise to estimates with reduced variance for estimating the state stationary distribution correction while it also offers a useful induction bias for estimating the state-action stationary distribution correction. In extensive experiments with both continuous and discrete environments, we demonstrate that our algorithm offers significantly improved accuracy compared to the state-of-the-art methods.

*Synopsis*:
  The main contribution of this paper is the development of estimated mixture policy (EMP), which takes ideas from the new off-policy policy evaluation infinite horizon estimators (i.e. Liu) and from a recent development in more traditional importance weight approaches using regression importance sampling (i.e. Hannah). This new method provides a nice extension of Liu's algorithm to many policies, and to when the policy is unknown. The paper provides some nice analysis inspired by Hannah. Finally, they provide empirical results.
  *Review*:
  While I think the method has potential interest to the community, I found the empirical results lacking (particularly in missing competitors). I also have some concerns about the theory as there seems to be many typos and consistency issues making much of it hard to follow. Overall, I think this paper is not quite up for publication, but if the authors address my consistency/missing proofs issues and provide a comparison to DualDice I will increase my score.
  1. I don't find the reason provided for not including DualDICE compelling and think it is an important competitor here, as there are many similarities between the two methods.
     - It would also be interesting to reproduce the results provided by Liu et al with the model based approach, and the on-policy oracle. I don't think these are as pressing as DualDice but still interesting.
  2. There are many consistency issues with respect to notation, and some odd notation choices as compared to the rest of the literature:
     - What is script \epsilon in the equation in 2.1? It looks like it should be an expectation over d_\pi, but I've never seen this notation before.
     - The indicies of sums and sets often change between one based and zero based indexing. This should be unified (preferably to one based). For example, section 2.1 \pi_j(j=0,1,2,...,m) and m=1 for one policy doesn't work. Either \pi_j(j=0,1,2,...m-1) or m=0. This occurs throughout the proofs in the appendix as well.
     - What is script \Epsilon_\theta? Do you mean script F_\theta? And then what does it mean for \theta_0 \in E_1? There seems to be many definitions missing.
  2.5. There are also some issues with the presentation of the theory over the consistency issues already mentioned.
     - The assumptions and conditions for the theorems presented in the main text should be clearly specified in the theory statement.
     - The proof to theorem 2 (i.e. in the appendix) should be provided.
     - The proof of proposition 4 seems to be missing as well.
  *Questions*
  Q1 What are the properties of SADL? Why was this used instead of DualDICE in the comparison?
  Q2 How would BCH do if we used the mixture policy as the behavior policy in the multiple behavior policy case? How would it compare to your method? This could be an interesting comparison, just to test if the lower MSE argument holds up in the multiple behavior policy case.
  Q3 What is the meaning of partially-policy-agnostic? It is unclear to me how it is different from the policy-agnostic approaches. If all that is different is you are estimating the behavior to use in the usual infinite-horizon approach, should this be in a separate category from policy-agnostic approaches? (I would say probably not, but I think you could make a case for it).
  Q4 "Then, a single (s,a,s') tuple simply follows the marginal distribution...". Is this trivially true?
  *Minor comments not taken into account in the review*
  - section 2.1 "target policy \pi via a pre-collected..." -> remove "a"
  - The layout of the related works section is a bit hard to follow.
  - "Recently, Nachum et. al. (2019) proposes DualDice": proposes->proposed
  - "by their estimated values in two folds": do you mean in two ways? This is unclear.
  - Section 3.1: "notation abusion" -> notation abuse
  - Equation right after equation 1: "d_\pi_0(s)\pi(a|s)" -> "d_\pi_0(s)\pi_0(a|s)"
  - "The derivation of kernel method are put in..." -> "The derivation of the kernel method is put in..."
  - "we need introduce more notation" -> "we need to introduce more notation"
  - Section 4: "detailed description on the data sample": "on"->"of"
  - "In this light by pooling the data together...." These two sentences should be put together.
  - I would like if your theorems were restated in the appendix, for ease of reading.
-----------
Post Rebuttal
I'd like to thank the author for their thorough response! Given the major additions to the paper including empirical comparisons and clarity for the theory I've decided to update my score to reflect my new feelings (i.e. to a 6). I think this paper is well worth accepting in its current form.


Well-performing deep learning models have enormous impact, but getting them to perform well is complicated, as the model architecture must be chosen and a number of hyperparameters tuned. This requires experimentation, which is timeconsuming and costly. We propose to address the problem of hyperparameter tuning by learning to forecast the training behaviour of deep learning architectures. Concretely, we introduce a forecasting model that, given a hyperparameter schedule (e.g., learning rate, weight decay) and a history of training observations (such as loss and accuracy), predicts how the training will continue. Naturally, forecasting is much faster and less expensive than running actual deep learning experiments. The main question we study is whether the forecasting model is good enough to be of use -can it indeed replace real experiments? We answer this affirmatively in two ways. For one, we show that the forecasted curves are close to real ones. On the practical side, we apply our forecaster to learn hyperparameter tuning policies. We experiment on a version of ResNet on CIFAR10 and on Transformer in a language modeling task. The policies learned using our forecaster match or exceed the ones learned in real experiments and in one case even the default schedules discovered by researchers. We study the learning rate schedules created using the forecaster are find that they are not only effective, but also lead to interesting insights.

The paper investigates the possibility of learning a model to predict the training behaviour of deep learning architectures from hyperparameter information and a history of training observations. The model can then be used by researchers or a reinforcement learning agent to make better hyperparameter choices. The paper first adapts the Transformer model to be suitable to this prediction task by introducing a discretization scheme that prevents the transformer decoder's predictions from collapsing to a single curve. Next, the problem is formalized as a partially-observable MDP with a discrete action set, and PPO and SimPLe are introduced. The proposed model-based method is compared against a human and a model-free baseline training a Wide ResNet on CIFAR-10. The model-based method achieves better validation error than the other baselines that use actual data. Next, the method is compared against a human and a model-free baseline training Transformer models on the Penn Treebank dataset. While the human achieves the best performance at the end of the run, the proposed method appears to learn more quickly than the others and finishes with performance comparable to the model-free baseline.
Currently I lean towards accepting this paper for publication, despite a few issues. It asks an interesting question: can we learn a model of the training dynamics to avoid actually having to do the training? This could potentially prevent a lot of unnecessary computation and also lead to better-performing models. It then shows some experimental evidence suggesting that this is possible.
Most importantly, I would like to see a measure of variance/uncertainty like confidence intervals included in the results; otherwise it's impossible to assess whether the results are likely to be significant or not. Other questions:
1. In the PTB experiment, it looks like the human only adapts the learning rate and leaves the rest of the hyperparameters alone. Why was this policy used as the baseline? It seems extremely basic and unlikely to truly lead to optimal performance.
2. Why were more baselines from the related work not included? I understand the experiments are a proof of concept, but it would be nice to get a feeling for what some of the other methods do.
3. How do PPO and SimPLe handle partial observability? Is it principled to apply them to partially-observable environments?
4. Why not use continuous actions with a parameterized policy (e.g. Gaussian)?
5. Is it reasonable to assume that the learning dynamics of all deep learning architectures are similar enough that a model trained on one set of deep learning architectures and problems will generalize to new architectures and problems?


We propose a deep network that can be trained to tackle image reconstruction and classification problems that involve detection of multiple object instances, without any supervision regarding their whereabouts. The network learns to extract the most significant K patches, and feeds these patches to a task-specific networke.g., auto-encoder or classifier -to solve a domain specific problem. The challenge in training such a network is the non-differentiable top-K selection process. To address this issue, we lift the training optimization problem by treating the result of top-K selection as a slack variable, resulting in a simple, yet effective, multi-stage training. Our method is able to learn to detect recurring structures in the training dataset by learning to reconstruct images. It can also learn to localize structures when only knowledge on the occurrence of the object is provided, and in doing so it outperforms the state-of-the-art.

The article introduces the novel MIST architecture which tries to solve the problem of multiple-instance classification and image generation from multiple objects. It employs two submodels, where the first generates a heatmap of intereting region and the second model is a task-specific model that works on image-patches, for example a classifier or an autoencoder. Both models are connected by a patch-extraction routine. The main contribution of this paper is to provide a way to propagate errors through this non-differentiable patch-extraction scheme. This is done by introducing slack-variables.
------------------
The paper is overall relatively easy to follow and the results are very good. However, it suffers from the fact that it does not differentiate between model-architecture and the overall approach. While the main contribution is described in Section 4, the paper spends a lot of space beforehand to introduce the task-dependent models as well as the heatmap architecture - things that i can imagine will vary a lot in different applications. The real important part is how to train the model and this is unfortunately only half described. A good deal of abstraction from the network architecture would have made the paper a lot better.  Further, I think that the loss-function for the classification task does not work in the general case.
On my first read-through, i completely misunderstood Section 4. Here is an unsorted list of issues i had with this:
- since E_K is not truly invertible, writing the approximate inverse as E_K^{-1} is misleading.
- It might help to stress that you treat {x_k} as continuous and the sampling as differentiable.
- In (7) it would be better to explicitly write E_K^{-1}(x_k) instead of introducing \bar{h}. The line below is not clear.
- It is also misleading, because the choice of \bar{h}=E_K^{-1}(x_k) is not the minimizer of (5) given that all other variables are fixed. You can see this by observing that assuming that when {x_k}=E_K(H(I)) holds, we can choose all other pixels
to be exactly the value returned by H(I).
- I am not sure where the alternating part comes from because this usually involves taking your solution from (7) and feeding it into (6).
- I am pretty sure that in (6)+(7), as well as lines 3+6 of the algorithm, you actually don't want to optimize for tau or eta from scratch but only perform a single SGD step. I think that is what you are doing, but right now it is written as "find a complete new model for each batch".
- Since this is performed batch-wise: is x_k a variable kept between iterations or do you use H(I) for an initial estimate of x_k for the batch?
Regarding the classification objective:
-since (3) uses the MSE of the mean class-label and the mean-prediction, a dataset where all objects always appear with the exact same amount will not work since than for each image the mean label is identical.
- Therefore, the MNIST-easy dataset should be unsolvable for the proposed architecture since every digit occurs exactly once.