Pairwise Choice Markov Chains (PCMC) have been recently introduced to overcome limitations of choice models based on traditional axioms unable to express empirical observations from modern behavior economics like context effects occurring when a choice between two options is altered by adding a third alternative. The inference approach that estimates the transition rates between each possible pair of alternatives via maximum likelihood suffers when the examples of each alternative are scarce and is inappropriate when new alternatives can be observed at test time. In this work, we propose an amortized inference approach for PCMC by embedding its definition into a neural network that represents transition rates as a function of the alternatives' and individual's features. We apply our construction to the complex case of airline itinerary booking where singletons are common (due to varying prices and individual-specific itineraries), and context effects and behaviors strongly dependent on market segments are observed. Experiments show our network significantly outperforming, in terms of prediction accuracy and logarithmic loss, feature engineered standard and latent class Multinomial Logit models as well as recent machine learning approaches.

1.The goal of the paper is to connect flexible choice modeling with a modern approach to ML architecture to make said choice modeling scalable, tractable, and practical.
2. The approach of the paper is well motivated intuitively, but could more explicitly show that PCMC-Net is needed to fix inferential problems with PCMC and that e.g. SGD and some regularization + the linear parameterization suggested by the original PCMC authors isn't scalable in itself.
3. The approximation theorem is useful and clean, and the empirical results are intriguing. While consideration of more datasets would improve the results, the metrics and baselines considered demonstrate a considerable empirical case for this method.
My "weak accept" decision is closer to "accept" than "weak reject." (Edit 11/25/19: I raised my score in to accept in conjunction with the author's improvements in the open discussion phase)
Improvement areas(all relatively minor):
- While I personally enjoy the choice axioms focused on by the PCMC model and this paper, stochastic transitivity, IIA, and regularity are probably more important to emphasize than Contractibility. Because the properties of UE and contractibility were not used, it may be more appropriate to use this space to introduce more of the literature on neural-nets-as-feature-embeddings stuff.
- This paper could be improved by generalizing to a few other choice models- in particular the CDM (https://arxiv.org/abs/1902.03266) may be a good candidate for your method. This is more a suggestion for future work if you expand this promising initial result.
- Hyper-parameter tuning: I noticed that several of your hyper parameters were set to extremal values for the ranges you considered. If you tuned the other algorithms' hyper parameters the same way, it could be the case that the relative performance is explained by the appropriateness of those ranges. Would be interesting to have a more in-depth treatment of this, but I do understand that it's a lot of work.
Specific Notes:
Theorem 1 is nice, and the proof is clean, but doesn't explicitly note that a PCMC model jointly specifies a family of distributions \pi_S for each S \in 2^U obtained by subsetting a single rate matrix Q indexed by U. It's clear that PCMC-Net will still approximate under this definition, as \hat q_ij  approximates each q_ij because \hat q_ij doesn't depend on S. While the more explicit statement is true with the same logic in the theorem, the notational choice to have "X_i" represent the "i-th" element in S is confusing at first, as e.g. X_1 is a different feature vector for S = {2,3} and S={1,3}. I don't see this issue as disqualifying, but it took me a while to realize that there wasn't more than a notational abuse problem when I returned to the definitions where the indexing depended on the set S under consideration.
Typos/small concerns:
-Above equation (1), the number of parameters in Q_S is |S|(|S|-1) rather than (|S|-1)^2, as each of the |S| alternatives has a transition rate to the other |S|-1 alternatives.
-Below equation (3), I think you mean j \in S rather than 1<= j <= |S|, as S may not be {1,2,...,|S|}. Later I noticed that you always index S with {1,\dots,|S|}, but using i \in S in combination with 1<=j<=|S| was a bit confusing.
-X_i as the i-th element of S is a bit of an abuse of notation, as it surpasses dependence on S
-In Figure 1, you show X_0 in a vector that is referred to as "S." It is my understanding that X_0 represents user features. As the user is not in the set, this is confusing. The use of a vertical ellipsis to connect \rho(X_0) to \rho(X_1) is also confusion, as \rho(X_1) is input into the Cartesian product while X_0 is input into the direct sum.
Overall, nice job! Really enjoyed the paper and approach, good to see connections made between these literatures so that progress in discrete choice can be used at scale.


In this paper, we introduce a new task, Contextual Text Style Transfer, to translate a sentence within a paragraph context into the desired style (e.g., informal to formal, offensive to non-offensive). Two new datasets, Enron-Context and RedditContext, are introduced for this new task, focusing on formality and offensiveness, respectively. Two key challenges exist in contextual text style transfer: 1) how to preserve the semantic meaning of the target sentence and its consistency with the surrounding context when generating an alternative sentence with a specific style; 2) how to deal with the lack of labeled parallel data. To address these challenges, we propose a Context-Aware Style Transfer (CAST) model, which leverages both parallel and non-parallel data for joint model training. For parallel training data, CAST uses two separate encoders to encode each input sentence and its surrounding context, respectively. The encoded feature vector, together with the target style information, are then used to generate the target sentence. A classifier is further used to ensure contextual consistency of the generated sentence. In order to leverage massive non-parallel corpus and to enhance sentence encoder and decoder training, additional self-reconstruction and back-translation losses are introduced. Experimental results on Enron-Context and Reddit-Context demonstrate the effectiveness of the proposed model over state-of-the-art style transfer methods, across style accuracy, content preservation, and contextual consistency metrics.

The authors propose the task of contextual text style transfer: transferring the style of one text into another (i.e., informal to formal, or offensive to non-offensive), when the text is present within some larger, provided context. The authors propose a model (CAST) which takes advantage of the additional context to perform the style transfer. CAST outperforms previous style transfer models according to several automatic metrics, as well as human evaluation.
Overall, I am not convinced that the context is useful in the proposed tasks (Reddit-context and Enron-context), for 2 reasons:
* In the examples shown in Table 4, it seems possible to complete the proposed contextual style transfer tasks without the additional context (i.e., "Do y’all interface with C/P" -> "Do you all interface with C/P" for formal->informal transfer).
* CAST performs better across several metrics (including human evaluation), but I am not sure that the gain is from using the context in the manner that the authors suggest (increasing the generated sentence's coherence with the context). In particular, it is hard for me to see how some style transferred sentences can appear much more coherent than others, given that the the required changes are pretty local (i.e., word substitution). On the other hand, style transfer models seem to have some difficulty in maintaining semantic context, so it seems that some of the gains in e.g. BLEU could be from CAST using the context simply as a bag of words to draw from in generating a style-transferred sentence. Instead of including context, the issue of semantic correctness could be fixed in other, simpler ways, however, such as making a more extractive single-sentence style transfer model. I think some further analysis is necessary to conclude that the context makes the generations more coherent, rather than helping the model in some other way.
Other general concerns/questions for the authors:
* The baselines do not appear to have learned even the single-sentence style transfer task that well. It appears that the authors used the default hyper parameters for other style transfer models (Hybrid Seq2Seq and ControlGen), but wouldn't it be better to choose hyper parameters for those models based on the evaluation criteria (as is done for CAST)? I know that some style transfer models can be quite sensitive to hyperparameters, so I would be surprised if the models worked optimally out of the box. I also feel that a strong unsupervised machine translation model like XLM (or the related style transfer model from "Multiple-attribute text style transfer") could do better than the provided baselines, as these models seem to work notably better than their predecessors.
* I have a general concern that CAST is quite complicated and has a lot of moving parts and parameters to tune. I don't think there are ablations on all components of the model (i.e., the style classifier). I also wouldn't be surprised if a simple state-of-the-art unsupervised or semi-supervised machine translation model like XLM outperformed CAST, without using any context. To add the use of the context, it seems possible to make a simple modification to an existing state-of-the-art transfer/translation model like XLM (adding the context to the input)
* Why are the style and coherence classifiers fixed during training? Wouldn't it be better to update the classifiers while training the rest of the model (so they doesn't get exploited)?
* Which of the evaluated models/baselines are trained using the style and/or coherence classifiers directly used for evaluation? For models that are not trained to optimize the style and/or coherence classifier predictions, I am not sure it is fair to compare those models to CAST on the style/coherence classifier accuracy, since CAST has been directly trained to perform well on that metric.
* Are the evaluation classifiers identical to those used to train CAST? At the least, it seems better to use a different style/coherence classifier (i.e., trained with a different random seed or other hyperparameters) at test time that the one used during training (i.e., in case CAST has overfit the style/coherence of generations to the classifiers used during training).
* In Eqn. 5, why is there a tanh activation immediately preceding the softmax? Wouldn't the tanh limit the range of the output softmax probabilities?
The proposed datasets do seem useful to the community, and the empirical results do support that CAST performs better than baselines - it would just be helpful to understand in what way the gains are coming from the additional context (to know if the task is an interesting task).


In most practical settings and theoretical analyses, one assumes that a model can be trained until convergence. However, the growing complexity of machine learning datasets and models may violate such assumptions. Indeed, current approaches for hyper-parameter tuning and neural architecture search tend to be limited by practical resource constraints. Therefore, we introduce a formal setting for studying training under the non-asymptotic, resource-constrained regime, i.e., budgeted training. We analyze the following problem: "given a dataset, algorithm, and fixed resource budget, what is the best achievable performance?" We focus on the number of optimization iterations as the representative resource. Under such a setting, we show that it is critical to adjust the learning rate schedule according to the given budget. Among budget-aware learning schedules, we find simple linear decay to be both robust and high-performing. We support our claim through extensive experiments with state-of-the-art models on ImageNet (image classification), Kinetics (video classification), MS COCO (object detection and instance segmentation), and Cityscapes (semantic segmentation). We also analyze our results and find that the key to a good schedule is budgeted convergence, a phenomenon whereby the gradient vanishes at the end of each allowed budget. We also revisit existing approaches for fast convergence and show that budgetaware learning schedules readily outperform such approaches under (the practical but under-explored) budgeted training setting.

Pros:
The paper is clearly written. It provides an interesting perspective for training neural networks under resource constraints. The problem setting is novel. The proposed solution is simply decaying learning rate linearly from the initial value to zero during training, which is parameter free.
Cons:
- As the authors are advocating using linear scaling schedule, I would like to see whether it has some clear advantages over other schedules, but it is not quite clear. For example, we can still see step based schedule has better performance in 2 of the 4 tasks in Table 2. Poly and Cosine schedule is also better in some of the budgets in Figure 2.
- The comparison in Figure 2 and Table 2 is not very convincing without considering the variance of different trails. It is not clear whether the advantage is caused by learning rate schedule or randomness. It is better to report the mean and variance for multiple trials. Ideally, it would be better to performance significance test.
- I would like to see other lr schedules in Table 2. As shown in Figure 2, step based schedule is not the top3 schedules for CIFAR-10.
- As shown in Table 3, the proposed method has to wait until the end of training to get the best performing model, while step based schedule can find the best model around 90% training. The author argues that the proposed method does not need to perform validation test for each checkpoint and reduce the computation cost,  however, on the other side, this means early stopping is not able to use for linear scaling based schedule, which could be very useful when the training budget is large enough and evaluation is cheap.
- My major concern for this work is a lack of deeper understanding about the reason why linear LR schedule works better, if any. It would be stronger with such understandings. The authors try to provide an explanation from the relationship between learning rate and gradient magnitudes, but no clear conclusion is given. As noted in [1,2,3], when weight decay is used in training and BN layers are used, the weight magnitude is also decreasing, so is the gradient norms. But the weight norms or gradient norms does not mean too much due to scale invariance. I would like to see when no weight decay is used and whether there is any correlation between the learning rate and gradient norms.
- What is the lr decay unit for linear schedule? Is it decaying per epoch or per mini-batch? If epoch based lr decay is used, it is essentially step-based lr decay with many steps. Then when the number of epochs is three, the step decay method (lr decays at epoch 1 and 2) and the linear decay method are actually almost equivalent.
- I would expect the convergence to be related with number of iterations. When the number of iterations is not long enough, neither linear lr decay or step based decay will work. It would be better if the author can investigate when linear schedule starts to outperforms step based decay in terms of epochs or iterations. Since different batch size results in different number of iterations, I would expect the difference between two schedules for small batch size at the early stage of training would be less in comparison with large batch training, especially when the number of iterations is enough.
- Different initial learning rate may also results in different behaviour. We often see some learning curves with larger initial learning rate converges faster at the beginning but yields similar generalization error at the end of training. On the other hand, the author only compared different schedules with single initial learning rate. Image when the initial learning rate is small, there would not be too much difference for different schedules. Actually the linear decay schedule changes may simply find a good learning rate during training as long as the initial learning rate is larger than the optimal one.
[1] Dinh et al, Sharp minima can generalize for deep nets, ICML 2017
[2] Li et al, Visualizing the Loss Landscape of Neural Nets, NIPS 2018
[3] van Laarhoven, L2 regularization versus batch and weight normalization, NIPS 2017
----- update after rebuttal ------
The authors' rebuttal addressed some of my concerns.  I think early stopping is still an important feature to save compute, especially for HPO, which could limit the usage of the proposed method (cannot stop earlier). If the authors believe practitioners are used to train model in full budget without early stopping to guarantee the best performance, then why would people try budged training with worse performance? I hope the authors could make the limitation clear. My major concern about the reason for why linear scaling schedule is better is still not clear, which makes the contribution kind of weak. Nevertheless,  the problem setting and the observations could be beneficial to the community for further discussion, So I raised my score to weak accept.
minor: I see exactly the same variance values for step and linear methods int the top rows of Table 12, is this a mistake?


Consider a world in which events occur that involve various entities. Learning how to predict future events from patterns of past events becomes more difficult as we consider more types of events. Many of the patterns detected in the dataset by an ordinary LSTM will be spurious since the number of potential pairwise correlations, for example, grows quadratically with the number of events. We propose a type of factorial LSTM architecture where different blocks of LSTM cells are responsible for capturing different aspects of the world state. We use Datalog rules to specify how to derive the LSTM structure from a database of facts about the entities in the world. This is analogous to how a probabilistic relational model (Getoor & Taskar, 2007) specifies a recipe for deriving a graphical model structure from a database. In both cases, the goal is to obtain useful inductive biases by encoding informed independence assumptions into the model. We specifically consider the neural Hawkes process, which uses an LSTM to modulate the rate of instantaneous events in continuous time. In both synthetic and real-world domains, we show that we obtain better generalization by using appropriate factorial designs specified by simple Datalog programs.

Summary:

The paper proposed to use Datalog rules to specify the design of the LSTM architecture for event data in continuous time. The LSTM module will be used to model the rate of the events. By incorporating Datalog rules, the paper aims to encode informed inductive biases into the model.
Comments:
    After reading the entire paper, I think the main idea of this paper is to use sparse and structured weight matrices (called structural zeros” in the paper) to substitute the dense weight matrices in the original LSTM, and to split the hidden state into blocks where each block refer to a different world’s state.
         How to design the structured weight matrices and how to define the node blocks, it is informed by the  Datalog rules. This design, however, will lead to a huge weight matrix and a very  long hidden state once the types of events and number of entities grow. The proposed model will face a severe scalability issue. From this point of view, only “structural zeros” weight matrices are not enough for an elegant model.

        How to smartly share parameters and how to control the number of parameters will be an interesting direction to explore. This submission touches on this a bit but not in a principled way. For example, in Eq. 18(a) and 18(b), the embedding vectors for the grounded predicate is a summation of the embedding vectors of the entities and the predicates. This final embedding is empirically validated or is based on some permutation invariant property? This needs more clarification or some references.

        2.      The presentation needs to be polished. The current writing is not easy to follow. Especially for section 3. The architecture design needs to be clarified more. When I read this part, I felt a little difficult to map the Datalog rules to your model.
        3        Since you are learning the vector embeddings for event types and entities, what are the advantages of this compared to the marked point process model, where the event types and entities are treated as discrete markers and are a much more parsimonious model. The Datalog rules can also be defined on the marker level by introducing a structured dependency structure over the markers. What are the potential benefits of learning the embeddings? The explanation is missing in this paper.
       4     Lack of references. The proposed neural-symbolic architecture shares some similarities to the following papers:
          (1) End-to-End Differentiable Proving
          (2) DeepProbLog: Neural Probabilistic Logic Programming
          (3) Neural Logic Machines.

        What are your contributions and differences in terms of the neural-symbolic architecture design?

       As for introducing logic rules to guide event predication, this is not a new topic. Here is a list of references:

         (1) PEL-CNF: Probabilistic event logic conjunctive normal form for video interpretation.
          (2) A general framework for recognizing complex events in Markov logic.
          (3) Learning Bayesian networks for clinical time series analysis.
          (4) Logical Hierarchical Hidden Markov Models for Modeling User Activities.
          (5) Slice Normalized Dynamic Markov Logic Networks.
       5.         Lack of strong baselines. The paper only did a small-scale experiment study. It only compares a neural Hawkes process model. The experimental evaluation also needs stronger baselines. Specifically, methods that can handle continuous-time (e.g. marked point process) or probabilistic logic methods that can discretize time (as mentioned in the above references). The baselines are not quite strong and appear a bit arbitrary in the paper.


Common fairness definitions in machine learning focus on balancing various notions of disparity and utility. In this work we study fairness in the context of risk disparity among sub-populations. We introduce the framework of Pareto-optimal fairness, where the goal of reducing risk disparity gaps is secondary only to the principle of not doing unnecessary harm, a concept that is especially applicable to high-stakes domains such as healthcare. We provide analysis and methodology to obtain maximally-fair no-unnecessary-harm classifiers on finite datasets. We argue that even in domains where fairness at cost is required, no-unnecesary-harm fairness can prove to be the optimal first step. This same methodology can also be applied to any unbalanced classification task, where we want to dynamically equalize the misclassification risks across outcomes without degrading overall performance any more than strictly necessary. We test the proposed methodology on real case-studies of predicting income, ICU patient mortality, classifying skin lesions from images, and assessing credit risk, demonstrating how the proposed framework compares favorably to other traditional approaches.

This paper considers the notion of "no-harm" group fairness, i.e. trying to reduce the risk gap between minority and majority groups without excessive reduction in performance on the majority groups. Authors formalize the problem by defining a Pareto fair classifier, i.e. one that minimizes the risk gaps between groups and belongs to the family of Pareto classifiers containing the classifier minimizing the empirical risk. Authors suggest an optimization procedure for finding the Pareto fair classifier and demonstrate its performance on multiple datasets.
Pros:
I think that studying "no-harm" classifiers is an important topic given the alarming tendency of some of the recent group fairness approaches to achieve fairness by essentially driving down the performance on the majority group without improving on the minority group. Decision making in medical applications is one of the prominent examples where "no-harm" is absolutely needed, as authors suggested. The mathematical formulation of the problem around the notion of Pareto optimality also seems reasonable.
Cons:
My concerns are related to counter-intuitive experimental results and lack of clarity in parts of the presentation.
Figure 1 seems important for understanding the ideas in the paper, but is not explained in much detail. Analogous to it Figure 2 is lacking important details. In the upper left plot, what are the decision boundaries of the baselines? What are the baselines risks in the center top figure, particularly for the equal risk classifier? It is hard to see from the right figure if the proposed classifier achieves the "no-harm" fairness over the equal risk classifier - numerical summary in a table could help. Finally, why is it necessary to use a neural network (which seems to be the case based on the supplement A.3) for the toy problem? I would recommend working through a toy example in more detail using a linear classifier to verify the correctness of the proposed technique and improve the overall clarity. Further, absence of a toy problem with linear classifier is alarming given there is not much discussion of the algorithm and its convergence properties.
Regarding the real data experiments, none seem to showcase the "no-harm" versus "zero-gap" fairness tradeoff motivating this paper.
MIMIC-III results seem to contradict the main story of the paper. The minority group appears to be "D/A/NW", then there should be a "harming" group fairness classifier achieving close to 0 discrimination at the cost of lowering performance on other subgroups. The "no-harm" classifier should then achieve a similar or slightly lower performance on "D/A/NW", but much better results on other sub-groups. Despite, the "no-harm" classifiers seems to outperform other approaches on "D/A/NW" by a good margin. Next, it appears that "Naive+Zafar" (it also would be helpful to have a brief discussion of the baselines considered) approach was not configured to eliminate A/S disparity as suggested by poor results on the D/A/NW and D/A/W, while it performs very well on other subpopulations.
Skin Lesion classification experiment departs from the problem of fairness and considers the problem of classification with unbalanced classes instead. Results are again counterintuitive. "Rebalanced Naive" only mildly improves over the "Naive" approach, while proposed algorithm seems to achieve a quite significant mean accuracy improvement. This again does not show the motivating "harm" vs "zero gap" tradeoff, but it could be interesting as the imbalanced classification is an important problem by itself. Could you please compare to more advanced imbalanced classification algorithms?
Results on the Adult and German Credit datasets are very similar across competing methods.
Acknowledgement and references to the individual fairness line of works are missing when presenting the problem of fairness in machine learning.
Font size in tables and Figure 2 legends is too small.
Typo in the last sentence on page 8: "have highly impactful" -> "are highly impactful".


To select effective actions in complex environments, intelligent agents need to generalize from past experience. World models can represent knowledge about the environment to facilitate such generalization. While learning world models from high-dimensional sensory inputs is becoming feasible through deep learning, there are many potential ways for deriving behaviors from them. We present Dreamer, a reinforcement learning agent that solves long-horizon tasks purely by latent imagination. We efficiently learn behaviors by backpropagating analytic gradients of learned state values through trajectories imagined in the compact state space of a learned world model. On 20 challenging visual control tasks, Dreamer exceeds existing approaches in data-efficiency, computation time, and final performance.

Paper summary.
The paper proposes Dreamer, a model-based RL method for high-dimensional inputs such as images. The main novelty in Dreamer is to learn a policy function from latent representation-and-transition models in an end-to-end manner. Specifically, Dreamer is an actor-critic method that learns an optimal policy by backpropagating re-parameterized gradients through a value function, a latent transition model, and a latent representation model. This is unlike existing methods which use model-free or planning methods on simulated trajectories to learn the optimal policy. Meanwhile, Dreamer learns the remaining components, namely a value function, a latent transition model, and a latent representation model, based on existing methods (the world models and PlaNet). Experiments on a large set of continuous control tasks show that Dreamer outperforms existing model-based and model-free methods.
Comments.
Efficiently learning a policy from visual inputs is an important research direction in RL. This paper takes a step in this direction by improving existing model-based methods (the world models and PlaNet) using the actor-critic approach. I am leaning towards weak accepting the paper.
I am reluctant to give a higher score due to its incremental contribution. Specifically, the policy update in Dreamer resembles that of SVG (Heess et al., 2015), which also backpropagates re-parameterized gradients through a value function and a transition model. The main difference between Dreamer and SVG is that Dreamer incorporates a latent representation model. From this viewpoint, the actor-critic component in Dreamer is an incremental contribution. Since the latent models are learned based on existing techniques, the paper presents an incremental contribution.
Besides the above comments, I have these additional comments.
- Effectiveness on very long horizon trajectories:
Simulating long-horizon trajectories with a probabilistic model is known to be unsuitable for model-based RL due to accumulated errors. This is an open issue in model-based RL. The paper attempts to solve this issue by backpropagating policy gradients through the transition model, which is known to be more robust against model errors (see e.g., PILCO (Deisenroth et al., 2011)). However, the issue still exists in Dreamer, since there seems to be an upper limit of effective horizon length (perhaps around 40, according to Figure 4). This horizon length is still short compared to the entire horizon length of many MDPs (e.g., 1000). I think this point should be discussed in the paper. That is, the issue still exists, and Dreamer is less effective with very long horizon.
- Inapplicability to discrete controls:
One restriction of re-parameterized gradients is that the technique is not applicable to discrete random variables. This restriction exists in Dreamer, and the method cannot be applied to discrete control tasks unless approximation techniques such as Gumbel-softmax are used. Still, such approximations would make learning more challenging, especially with long-horizon backpropagation. This restriction should be noted in the paper.
- There is no mention about variance of policy gradient estimates. Dreamer does not use any variance reduction technique, so the gradient estimates could have very large variance.
- q_theta was introduced in Eq. (8) before it is defined in Eq. (11). Also, I suggest moving Section 4 to be right after Section 2, since Section 4 presents existing techniques similarly to Section 2, while Section 3 presents the main contribution.
Update after authors' response.
I read the response. The paper is more clear after authors' clarification. Though, I still think the contribution is incremental, since back-propagating gradients through values and dynamics has been studied in prior works (albeit with less empirical successes compared to Dreamer). Nonetheless, I am keen to acceptance. I would increase the rating from 6 to 7, but I will keep the rating of 6 since the rating of 7 is not possible.


There has recently been some success in the use of reinforcement learning methods for single robot combinatorial optimization problems. In this paper, we develop the first learning-based method for multi robot/machine planning problems with combinatorial nature. One real-world concern is the capability to achieve transferability to an unseen number of robots and tasks. The method developed here, for the first time, enables such transferability. Our method is comprised of three components. First, we illustrate how to represent a robot planning problem as an extension of probabilistic graphical models (PGMs) which we refer to as random PGMs. We develop a mean-field inference method for such random PGMs and use it for Q-function inference. Second, we show that transferability can be achieved by carefully encoding the problem state via a two-step sequential process. Third, we resolve the computational scalability issue of fitted Q-iteration. This is achieved by employing an auction-based heuristic as a substitute for the max operation in the Bellman equation. The auction is enabled by the transferability achieved. Our method is applicable to discrete-time, discrete space problems such as multirobot reward collection (MRRC). For such problems, with determinsitic assumptoins, we scalably achieve 97% optimality with transferability. This optimality is maintained under stochastic contexts. By extending our method to a continuous time, continuous space formulation, the approach is the first learning-based method for machine scheduling problems. Our method scalably achieves comparable performance to popular metaheuristics when applied to identical parallel machine scheduling (IPMS) problems in a deterministic context.

The authors study a combinatorial multi-robot scheduling problem (in fact the robot part is a bit inflated, since the experiments only involve agents in a simulated discrete state-space maze) using a method that builds upon recent advances from [Dai et al. (2017)]. The main contribution is to consider each of the steps taken by Dai et al. to solve combinatorial problems on graphs, and adapt them to the considered scheduling problem.
Not being an expert in RL, my assessment should be discounted. However, I am not sure I follow properly the main idea of the paper. The point of Dai et al. was to use RL to solve a wide family of combinatorial problems. Now, the authors claim to build upon these ideas to solve... what looks essentially like a far more standard RL problem, and not necessarily a combinatorial optimization problem. The main insight by Dai et al. was to highlight the fact that combinatorial problems are usually solved (or approximated) without "warm starts", i.e. they do not consider distributions on problem instances to learn from. The problem considered by the authors is, quite on the contrary, a typical RL problem where information is extracted from the problem's structure (here a maze). Therefore, I feel there is something of a fundamental contradiction going on at a fairly high-level, in the sense that the paper "uses RL to solve a subset of combinatorial problems that were studied by RL before". The absence of other baselines in experiments make this even more suspicious. Therefore I believe the paper's presentation could be greatly improved if it were better "located" within the RL literature (which is almost non-existent in the very brief bibliographic section) and that the authors were able to show that  their proposals are original, within an RL context.
minor points:
* the comment "While learning-based methods are generally believed to suffer exponentially increasing training requirements as problem size (number of robots and tasks) increases, our method’s training requirement is empirically shown not to scale while maintaining near-optimal performance" --> this is too loose a statement. Provide more evidence or references.


Batch normalization (BN) is often used in an attempt to stabilize and accelerate training in deep neural networks. In many cases it indeed decreases the number of parameter updates required to achieve low training error. However, it also reduces robustness to small adversarial input perturbations and common corruptions by double-digit percentages, as we show on five standard datasets. Furthermore, we find that substituting weight decay for BN is sufficient to nullify a relationship between adversarial vulnerability and the input dimension. A recent mean-field analysis found that BN induces gradient explosion when used on multiple layers, but this does not fully explain the vulnerability we observe, given that it occurs for a single BN layer. We argue that the main cause is the tilting of the decision boundary with respect to the nearest-centroid classifier along input dimensions of low variance. As a result, the numerical stability constant of BN acts as an important meta-parameter that can be tuned to recover some robustness at the cost of standard test accuracy. We explain this mechanism explicitly for linear models but find that it still holds for nonlinear models.

This paper identifies an important weakness of batch normalization: it increases adversarial vulnerability. It is very well written and the claims are theoretically sound. In the experiments, the authors demonstrated a significant difference in robustness between networks with or without batch normalization layers, in varies settings against both random input noise and adversarial noise. This weakness of batch norm was explained due to the "decision boundary tilting" effect caused by the normalization. Overall, this paper has done solid work to reveal an interesting phenomenon. If it is true, this finding will impact almost all DNN models.
My concern is that this phenomenon is just another effect of "gradient masking " (as pointed out by Athalye, et al.). Batch norm is a well-known technique to avoid overfitting, without batch norm the network can be easily trained to be saturated with almost zero gradients, demonstrating a false signal of "robustness" to noise. The random noise and real-world corruption experiments are definitely helpful to clear this doubt, but only partially. My concern remains because of two obvious signs of  gradient masking:
1. The accuracy on PGD-li (epsilon=0.031) attacks are suspiciously too high (20% - 40% Table 3/4). For this level of attack, the acc should be nearly zero. This is likely caused by the gradient masking effect, considering the cifar-10 networks were trained for longer time with larger learning rate (150 epochs, fixed lr 0.01). Training on MNIST is much easier to get zero gradients.
2. The weight decay discussion is not helpful at all, on the contrary, it confirms my concern on the gradient masking effect. In Table 8, the robustness was increased ~40% by just using large weight decay. This is not the "real robustness", and can be easily evaded by adaptive attack (see Athalye's paper).
With the above two concerns in mind, I doubt the phenomenon revealed in this paper is just "one can easily train a saturated model without batch norm" or equivalently "it's hard to train a saturated model with batch norm". It is hard to say if this is a bad thing for batch norm.
I am quite surprised that the authors ignore this completely. Here are a few things that can be done to rule out the possibility of gradient masking. The masked gradient can be identified by: 1) One-step attacks perform better than iterative attacks; 2) Unbounded attacks do not reach 100% success., etc (see Section 3.1 of Athalye's paper).
1. Including FGSM in the experiments and show the same trends as PGD-li.
2. Show two networks have similar gradient norms.
3. Apply cw-l2 attack, and show batch norm has forced large perturbation.
Two other suggestions:
1. Summarize the different angles/steps taken to verify the phenomenon, somewhere before the experiments.
2. Cannot see why the input dimension discussion contribute to explanations of the batch norm weakness.
============
My rating stays the same after rebuttal.
My original concerns are like the other reviewers: why BN, not other techniques such as structure of DNNs MLP vs CNN vs ResNet, activation functions, weight decay, learning rates, softmax etc. My initial suspect was that it is caused by gradient masking likely caused by the l2 weight regularization, so asked the authors to look at the gradient norms and run some testes to rule this out. Yes, the weight norm is directly related to the Lipschitz continuity of the function represented by the network, but it often becomes more complicated on complex nonlinear neural networks.
According to the new experiment results, the vulnerability is indeed not an effect of gradient masking, thanks for the clarification. However, the new results also indicate that the finding is susceptible to both weight decay and learning rate: in Figure 16 (a): "Un PGD" < "BN PGD" before learning rate decay, andFigure 17 (a) vs (b), doubling the weight decay penalty to 1e-3 also increases the vulnerability of BN. Overall, I believe the phenomenon exists, but the reasons behind requires more explanations, at least not just the batch norm.


Computing equilibrium states for many-body systems, such as molecules, is a long-standing challenge. In the absence of methods for generating statistically independent samples, great computational effort is invested in simulating these systems using, for example, Markov chain Monte Carlo. We present a probabilistic model that generates such samples for molecules from their graph representations. Our model learns a low-dimensional manifold that preserves the geometry of local atomic neighborhoods through a principled learning representation that is based on Euclidean distance geometry. In a new benchmark for molecular conformation generation, we show experimentally that our generative model achieves state-ofthe-art accuracy. Finally, we show how to use our model as a proposal distribution in an importance sampling scheme to compute molecular properties.

In this paper, a new method for estimating the equilibrium measure of molecular compounds is proposed and tested on a new (enriched) version of an existing data set (CONF17 is derived from the pre-existing ISO17, itself extracted from QM9).
The method takes as input a collection of graph structures of a given stoichiometry mixture, and learns the corresponding conformations.
This results in a generative model, taking as input a graph structure (or actually a single conformation instance, thus setting the stoichiometry AND the isomer/graph structure), to generate (in a one-shot fashion) other conformations of that isomer.
The distribution of conformations sampled in this way compare well with those of an extensive training set (CONF17, set of isomers and conformations of a single compound), and in particular they perform better than 2 alternative methods published recently (2015 and 2019, the former performing globally much better than the latter!).
Concretely, the model/method/algorithm consists in a combination of existing network architectures.  The main original contribution is to map the Cartesian coordinates of atoms into a pairwise distance space, which is naturally translationally and rotationally invariant.  Some of the second and third neighbors distances in the graph structure are considered, such that the conformation is completely set (this would not be the case if considering only 1st nearest neighbors distances).
The precise combination of architectures is explained in section 2.2. This explanation is a little fast for me (but I am new to this field).
In that section, it could be nice to precise that the job of the CVAE is to learn (among others) the parameters mu,sigma of q_phi(z|d,G), using a batch of conformations of a given isomer (and then generalize to other isomers, each time guessing the correct mu,sigma parameters, even for a new isomer).
This is not obvious for the inexperienced reader (I hope I "guessed" correctly).
The "new dataset" shared in this work is indeed useful for reproducing results and for later comparison with other works which aim at sampling conformations.
However, it derives directly from ISO17, which is itself an extraction from QM9. ISO17 lists the Cartesian coordinates of 127 isomers of C7,H10,O2 (each having ~3000 conformations).  The "new" CONF17 is novel only in that it provides these conformations in distance-matrix space, i.e. it sets which pairs of atoms need to have their distances computed.  This is very useful because this choice of pairs is not unique, and so setting it once is necessary for comparison.  I would not stress too much that this is a new dataset: at first it actually sounds like authors chose a dataset such that their method outperforms others'!  Which turns out not to be the case (they chose C7,H10,O2, but this doesn't look like fine-tuning).  Indeed, their sharing of this augmented version of ISO17 is absolutely necessary: I would simply put it differently, stating from the beginning that they share an augmented version of a pre-existing data set.
About the experimental protocol/methodology:
the work carried out seems quite impressive. Indeed, the paper presents a rather extensive comparison of the new model with 2 other ones, on a large data set.
However sometimes the presentation rushes a bit towards results, in the sense that it is not easy to figure out over which ensemble the mean/median/max/min has been computed. This is related to the intrinsic complexity of estimating accuracy, given the encapsulation (compound-> isomer-> conformations-> each bond-> distribution over bond properties).  I think however it could be made clearer, and this would benefit the paper greatly.
A deeper comment: in the field of conformation sampling, there are two issues. One is to correctly sample states in each local minimum of the free energy. This is addressed in the paper. The second issue is to hunt for all possible local minima of the isomer, i.e. sample even unlikely but crucial conformations. This is not addressed here.  For computing average energy, dipole moment etc, the most likely conformations will dominate, most of the time.  However given the topic I think it would be fair to mention this point, and explain why in this specific case, the second issue is not crucial.
Because of this, also, I think
Overall, the paper is well written and is very well situated within the literature (including the very recent literature).
It states clearly its contributions and results, and provides detailed presentation of experimental results.
I think it contributes to the field and provides an interesting new way of dealing with the problem of molecular conformations sampling.  The results obtained are much better than those of the compared ML method (RDkit).
However, some claims (in introduction or in section 3) are stated more strongly than they are supported by direct experimental evidence.
- ''... the first work on sampling molecular conformations for molecules of arbitrary size and shape based on deep learning.''
However, the experiments have been performed only on isomers of  C7,H10,O2.
Furthermore, there is no discussion on the computational cost of the method at all, so its extendibility to larger shapes is not obvious.
- '' We create a new, challenging benchmark dataset CONF 17 for conformation generation, which is made publicly available.''
I made remarks on this claim above.''
- '' We develop a rigorous experimental approach for evaluating and comparing the accuracy of conformation generation methods based on the mean maximum deviation distance metric.''
However the paper's methodology doesn't address the issue of ''hunting for new local minima''.
The data set could be enriched also with a labeling of ''which conformation is in which local minimum of free energy''
- ''It is a fast alternative to resource-intensive approaches based on MCMC or MD.''
As there was no mention of speed of computation in the paper (although we can guess it is faster), this is unsupported. However it is easy to remedy, simply give an order of magnitude of sampling time (for thousands of conformations of 1 isomer).
- ''Our principled representation based on pair-wise distances does not restrict our approach to any type of system (e.g., proteins) or any particular graph structure. In addition, it is extendable in a systematic fashion.''
This suggests that the method would also work for dense packings. However this is not shown in the experiments.
Probably I misunderstood the claim, and it only refers to the ability to tackle general molecular compounds, not just linear ones?  However proteins are not always linear, but can present cycles, and other branch-style patterns, so the claim is not very clear to me.
Because some of the important claims are only weakly supported, and because of the relatively small novelty of the method, I lean on rejecting the paper.
Additional comments (for improving the paper):
There is no discussion of hyper-parameter tuning, or any detail of the intermediate GNN that constitute the encoder and decoder in the CVAE.
So it was either not done at all, and then it should be mentioned; or it was done and it should be discussed quickly.
In particular, in section 2.2 there is no discussion on the impact of the latent space' dimension, here set to Nv, i.e. 7+2+10=19 (correct?).  I guess there is a balance between accuracy and generalizability, a short discussion on this point would be interesting.
The paper should clarify the hierarchy of molecular configurations:
for a given composition (%C, %O, %H, etc) (here, fixed throughout the paper at C7,H10,O2)
there are several structures possible (graphs or isomers),
for a given graph (isomer), there can be a couple of chiral variants (mirroring of (pieces of) the molecule,
for each chiral variant, there are conformations, which correspond to rather weak variations of the Cartesian coordinates, due to thermal fluctuations. Some of these variations can be important, e.g. when a bond can freely rotate. Some correspond to bending or torsion, etc, of the molecule.
The algorithm presented in the paper attempts to sample appropriately the equilibrium distribution of conformations for each given graph structure (at fixed chirality, if I understood correctly).
Add a comment on why (in your opinion) the distributions are not exactly correctly sampled, which after all is the goal (since the initial graph structure is given as input here).
Additional question, for curiosity. In section 2.3: only a single molecule output is taken from each distance matrix.
Could you comment on the possibility to sample several Cartesian coordinates, for each distance matrix ?  It seems like several solutions should (often) exist.
Sampling continuously from that distribution would surely provide a more continuous (rich) output (am I correct) ?


Deep neural networks are known to be annotation-hungry. Numerous efforts have been devoted to reduce the annotation cost when learning with deep networks. Two prominent directions include learning with noisy labels and semi-supervised learning by exploiting unlabeled data. In this work, we propose DivideMix, a novel framework for learning with noisy labels by leveraging semi-supervised learning techniques. In particular, DivideMix models the per-sample loss distribution with a mixture model to dynamically divide the training data into a labeled set with clean samples and an unlabeled set with noisy samples, and trains the model on both the labeled and unlabeled data in a semi-supervised manner. To avoid confirmation bias, we simultaneously train two diverged networks where each network uses the dataset division from the other network. During the semi-supervised training phase, we improve the MixMatch strategy by performing label co-refinement and label coguessing on labeled and unlabeled samples, respectively. Experiments on multiple benchmark datasets demonstrate substantial improvements over state-of-the-art methods. Source code to reproduce all results will be released.

This paper proposes a DivideMix framework for learning with noisy labels, where they first Co-Divide the training data into a labeled clean set and an unlabeled noisy set by modeling the per-sample loss distribution with GMM and using the small loss trick, then they exploit MixMatch to train the model on those labeled and unlabeled data in a semi-supervised manner. Experiments and comparisons with SOTA are provided, together with an ablation study.
Pros:
-The paper bridges the area of learning with noisy labels with semi-supervised learning and proposes an interesting method to learn with noisy labels in a semi-supervised manner. The treatment proceeds from analyzing good unsupervised loss functions, improving the MixMatch and implementing thorough experiments.
-The paper is clear and flows smoothly. The treatment is thorough, proceeding from designing algorithms, implementing experiments and an ablation study.
-The impact of the method is a clear asset. Most existing label noise works focus on certain noise models, but this paper is more general and proposes an interesting method to learn with noisy labels in a semi-supervised manner, and the experimental results are promising.
-The effort made on designing the confidence penalty for asymmetric noise is interesting.
Remarks:
-Sec 3.1: it seems that \tau is an important hyperparameter for dividing the noisy data into labeled and unlabeled sets, but in Sec 4 I only see that \tao is set to be 0.5 or 0.6 for CIFAR, what about other experiments? I’m also wondering if \tao needs some decay during training? For example, it may be 0.5 at early training epochs, but may be smaller at last, since deep NNs are gradually fitting noisy features during training.
-Algorithm: the proposed method needs to train two networks simultaneously. During each epoch, it firstly divides the noisy data by modeling the per-sample loss distribution with GMM, and then do MixMatch with label co-refinement and co-guessing. I’m a bit concerned about efficiency. So how about the computation time?
-Experiments: more details on experimental protocol may be needed: what kind of hyperparameter tuning was done? How many repeated runs? It would be helpful to report the means and standard deviations based on repeated samplings.
Overall take: this paper proposes a thorough treatment of learning with noisy labels in a semi-supervised manner, designing the algorithm and testing it empirically, which is an interesting and important contribution. My only concern is about the novelty since the small loss trick in label noise and the MixMatch approach in SSL are already explored by many recent studies, but to the best of my knowledge, this paper is the first to unify them to solve label noise problems.


A plethora of methods attempting to explain predictions of black-box models have been proposed by the Explainable Artificial Intelligence (XAI) community. Yet, measuring the quality of the generated explanations is largely unexplored, making quantitative comparisons non-trivial. In this work, we propose a suite of multifaceted metrics that enables us to objectively compare explainers based on the correctness, consistency, as well as the confidence of the generated explanations. These metrics are computationally inexpensive, do not require model-retraining and can be used across different data modalities. We evaluate them on common explainers such as Grad-CAM, SmoothGrad, LIME and Integrated Gradients. Our experiments show that the proposed metrics reflect qualitative observations reported in earlier works.

See post-rebuttal updates below!
Summary
---
(motivation)
There are lots of heat map/saliency/visual explanation approaches that try to deep image classifiers more interpretable.
It's hard to tell which ones are good, so we need better ways of evaluating explanations.
This paper proposes 3 such explanation evaluation metrics, correctness, consistency, and confidence.
(approach - correctness)
An explanation is correct if it highlights enough of an image for a classifier to tell the correct class with only the highlight parts of the image.
The default way to evaluate on only highlighted portions is to set the non-highlighted bckground to black/grey.
Instead, this method finds images with the same ground truth class which the classifier scored the lowest of all such images, forming a low-confidence baseline.
It copies the background from one of these images instead of using a black/grey background
to try and put the masked image back into the distribution of images from the ground truth class.
This style of masking is used to compute correctness.
(approach - consistency)
An explanation is consistent if it is invariance w.r.t. a number of mostly semantically invariant transformations.
These include small affine transformations, horizontal flips, vertical flips, and adding noise.
(approach - confidence)
An explanation is confident if the masked images it produces still have high condidence under the classifier.
Masked images are produced as for correctness, by copying a distractor from the same class into the background.
(experiments)
The experiments compare existing explanations (LIME, Grad-CAM, Integrated Gradients, SmoothGrad) using the proposed metrics.
1. Correctness: Classifiers have higher accuracy on explanation-masked images than on images they were least confident on (the ones used to fill in the background).
2. Grad-CAM is most correct, followed by SmoothGrad, Integrated Gradients, and LIME.
3. Consistency: Grad-CAM explanations are most resilient to the proposed transformations with Integrated Gradients, SmoothGrad, and LIME being successively less invariant.
4. Confidence: Explanation-masked images have higher scores for their ground truth class than the low-confidence baseline images.
5. Hyperparameter variations in the correcness/confidence metrics mostly preserve the ranking of methods, though the absolute values of performance do change substantially.
(conclusion)
The paper concludes that Grad-CAM is usually the best of the methods tested according to the new metrics and that LIME is the worst.
Strengths
---
I really like the related work section. It could be a valuable resource going forward.
I like the research direction of this paper very much. I think that enumerating a suite of complementary benchmarks is a good way to measure explanation quality because we can only come up with benchmarks that capture a small part of what we want so far.
Weaknesses
---
I see some major conceptual flaws with these metrics:
* In section 3.1 it seems like the first reasons that normal masking failed is not solved by the proposed approach. The generated images are still out of distribution because the "foreground" and the "background" don't match.
* I'm concerned about the low-confidence distractor images used in the background. They are from the same ground truth class as the high confidence images they are pasted into the background of, correct? The correctness metric is supposed to capture whether or not an explanation highlights all the class-relevant content in an image and no more. However, information that the explanation did not highlight (the background) can inform the classifier of the ground truth class because the background came from an image of that class (even if a low confidence one). This is especially true because the relevant objects might be in differrent positions in the two images. Thus it could be that the explanation did not highlight informative content but the classifier still gets the corresponding masked image correct because of the background. How often does this happen?
* Consistency is supposed to measure "the ability of the explainer to capture the relevant components" under semantically invariant transformations.
The reported metric is mimized when the explanation is the same before and after a variety of transformations.
If this were the case then at least one of them must be wrong in the sense that it would not have captured some relevant components
(unless perhaps it just highlighted everything and was thus useless).
Because of the transformation (e.g. 15 degree rotation) the relevant components would have been at a different position, but the best explanation according
to this metric would have been at the same position. Thus this metric seems to reward explanations for not capturing relevant components.
Parts I Didn't Understand:
* In section 3.1, I don't understand the second reason that masking failed. In what sense is masking made meaningless? How is that sense different from the out of distribution concern from the first point?
Missing Details / Presentation Weaknesses:
* Missing reference to [1] which provides more metrics.
* The meaning of confidence is different than it normally is and this may be confusing.
Neural networks should be well calibrated, not necessarily confident (in the commonly used sense of [3]).
Minor flaws:
* Masking by replacing the background with grey (i.e., the bias of the first conv layer) rather than black is more common (e.g., [2] and Grad-CAM). A grey background negates the bias. It's not clear that the background should cancel the bias, but it would be nice to compare to both grey and black masking in Table 7.
[1]: Adebayo, Julius et al. “Sanity Checks for Saliency Maps.” NeurIPS (2018).
[2]: Zeiler, Matthew D. and Rob Fergus. “Visualizing and Understanding Convolutional Networks.” ECCV (2013).
[3]: Guo, Chuan et al. “On Calibration of Modern Neural Networks.” ICML (2017).
Final Evaluation
---
This paper relies solely on theoretical arguments to show its metrics capture meaningful information. Empirically, it only shows that the proposed metrics can differentiate between some popular explanations. It does not empirically show that the differentiation is meaningful (e.g., by measuring agreement with human judgement). This by itself isn't a problem. However, above I detailed significant flaws in the theoretical justification for the metrics, so I can't recommend these metrics (this paper) on either a theoretical or an empirical basis.
Quality: Per above, I do not think the arguments/evidence in the paper support its conclusions.
Clarity: The paper could be clearer, but can be understood without too much effort.
Originality: These metrics are new enough, being novel variations on prior approaches.
Significance: If I was convinced the metrics made sense then I would guess this paper would be very impactful. As is, I don't think it will have much impact.
The quality of the paper is my reason for the low rating. I'm interested to see whether what others think to make sure I've understood the paper correctly and analyzed it accurately. If my understanding is incorrect I could definitely raise my rating.
Post-Rebuttal Evaluation
---
After reading the other reviews and the author responses and taking a brief look at the updated paper I still think this paper should be rejected.
The authors' response to my comments clarified my understanding of the consistency metric. Now I understand it and think it is a useful metric.
However, I did not find clarification about the confidence or correctness metrics, though I agree they are not redundant. They still don't really quite make sense to me. This puts me in about the same position as R3, who also doubts those metrics. In the end, this leaves my initial evaluation essentially unchanged. I still recommend rejection because the paper relies on a theoretical understanding of what makes confidence and correctness metrics useful and that understanding is not provided.


Many automated machine learning methods, such as those for hyperparameter and neural architecture optimization, are computationally expensive because they involve training many different model configurations. In this work, we present a new method that saves computational budget by terminating poor configurations early on in the training. In contrast to existing methods, we consider this task as a ranking and transfer learning problem. We qualitatively show that by optimizing a pairwise ranking loss and leveraging learning curves from other data sets, our model is able to effectively rank learning curves without having to observe many or very long learning curves. We further demonstrate that our method can be used to accelerate a neural architecture search by a factor of up to 100 without a significant performance degradation of the discovered architecture. In further experiments we analyze the quality of ranking, the influence of different model components as well as the predictive behavior of the model.

The paper proposes a new method to rank learning curves of neural networks which can be used to speed up neural architecture search.
Compared to previous work, the learning curve model not only takes hyperparameter configurations into account, but by training it on offline generated data, it is able to model learning curves across different datasets.
Applied to a simple neural architecture search strategy, the proposed method achieves higher speed-ups on image classification tasks than other methods from the literature.
While the method seems interesting, I don't think the paper is ready for acceptance yet, since it a) misses some important details and b) the empirical evaluation is not sufficient.
More precisely the following points need to be addressed:
 - In section 3, the paper says that all automated methods follow broadly the same principal that the first model is trained to completion. This is not correct, commonly used methods, such as Hyperband (Li et al.) or BOHB (Falkner et al.), use successive halving (Jamieson et al.) which trains a batch of configurations for just a minimum budget and then already discards poorly performing configurations. I also miss a discussion of these methods in the related work section.
Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization
Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A., and Talwalkar, A.
Journal of Machine Learning Research 2018
BOHB: Robust and efficient hyperparameter optimization at scale
S Falkner, A Klein, F Hutter
Proceedings of the 35th International Conference on Machine Learning
Non-stochastic best arm identification and hyperparameter optimization
K Jamieson, A Talwalkar
Artificial Intelligence and Statistics, 240-248
 - Is the learning curve model updated during the optimization process with the new observed data? If not, is there any way the model can adapt to new data? Also, what happens if learning curves are fundamentally different to the training data, e.g if different learning rate schedules are used than for generating the offline data?
 - A simple baseline that is missing, is to use the last observed value as approximation for the final performance which often works competitively to more powerful learning curve extrapolation methods (e.g see Klein et al.).
 - The method is only applied for a simple random search strategy, however, in practice, one would use more sophisticated methods, such as for example Bayesian optimization. The question is, how effective is the proposed method in this setting, since the distribution of learning curves might be dramatically different and biased towards more well-performing configurations with almost identical learning curves.
- I think the experiments would be more convincing, if the method shows strong performance when deploy in commonly used NAS/HPO methods, such as Hyperband or successive halving. This should be straight forward, since decisions which configuration is promoted to a higher budget can be made based on the model instead of just the last observed value.
- What is delta in the experiments? How sensitive is this hyperparameter in practice?
Minor comments:
- Related to this paper is the work by Gargiani et al. which also models learning curves across datasets on offline generated data
 Probabilistic Rollouts for Learning Curve Extrapolation Across Hyperparameter Settings
 M Gargiani, A Klein, S Falkner, F Hutter
 arXiv preprint arXiv:1910.04522
- Figure 5: visually it seems that all learning curves are almost identical, maybe it would be better if the plot could zoom in at least for the final stage of training.
after the rebuttal
-----------------------
I thank the authors for answering my  questions. In the rebuttal the authors addressed my concerned about the insufficient empirical evaluation and included other relevant baselines. I will increase my score.


Unpaired image-to-image translation has attracted significant interest due to the invention of CycleGAN, a method which utilizes a combination of adversarial and cycle consistency losses to avoid the need for paired data. It is known that the CycleGAN problem might admit multiple solutions, and our goal in this paper is to analyze the space of exact solutions and to give perturbation bounds for approximate solutions. We show theoretically that the exact solution space is invariant with respect to automorphisms of the underlying probability spaces, and, furthermore, that the group of automorphisms acts freely and transitively on the space of exact solutions. We examine the case of zero 'pure' CycleGAN loss first in its generality, and, subsequently, expand our analysis to approximate solutions for 'extended' CycleGAN loss where identity loss term is included. In order to demonstrate that these results are applicable, we show that under mild conditions nontrivial smooth automorphisms exist. Furthermore, we provide empirical evidence that neural networks can learn these automorphisms with unexpected and unwanted results. We conclude that finding optimal solutions to the CycleGAN loss does not necessarily lead to the envisioned result in image-to-image translation tasks and that underlying hidden symmetries can render the result utterly useless.

This paper focuses on CycleGAN method to show theoretically when the exact solution space is invariant with respect to authomorphisms of the underlying probability spaces for unpaired image-to-image translation.
- The paper provides interesting theoretical results on identifying conditions under which CycleGAN admits nontrivial symmetries and has a natural structure of a principal homogeneous space.  Proposition 2.1 provides interesting insights into the invariance of the kernel space.
- Propositions 2.5 and 2.6 are interesting in that they show that the existence of authomorphisms can worsen the performance of CycleGAN, however, it is unclear that in practice, how could one verify the conditions efficiently before applying CycleGAN.
- The experimental results are interesting, however, they are very limited. Having a toy experiment is a good sanity check, but it would be more interesting to see the performance on a real-world applications, such as medical images or other use-cases brought in the introduction. Also, more discussion on the results provided in Fig 3, confusion matrices, would be very helpful. Are there any intuitions behind the large and low values in the table? It could be interesting to see what are the effects of other parameters such as alpha in producing the results.
- Overall this paper presents interesting results regarding the theory of CycleGANs, however, the numerical results are very limited, and do not justify the motivations discussed in the introduction and the abstract. Moreover, although the paper introduces novel attempts and theoretically analyzing the CycleGAN, the scope of the work seems to be limited, and thus, it does not have a sufficient significance to be published in ICLR. I strongly suggest the authors to expand and provide more experimental evaluations.
** update:
Thanks for your comments! I found the additional experiment useful and better aligned with the purpose of the model. The discussion added clarified the confusion about the automorphism. That is why I decided to change my score.


Measuring Mutual Information (MI) between high-dimensional, continuous, random variables from observed samples has wide theoretical and practical applications. Recent works have developed accurate MI estimators through provably lowbias approximations and tight variational lower bounds assuming abundant supply of samples, but require an unrealistic number of samples to guarantee statistical significance of the estimation. In this work, we focus on improving data efficiency and propose a Data-Efficient MINE Estimator (DEMINE) that can provide a tight lower confident interval of MI under limited data, through adding cross-validation to the MINE lower bound (Belghazi et al., 2018). Hyperparameter search is employed and a novel meta-learning approach with task augmentation is developed to increase robustness to hyperparameters, reduce overfitting and improve accuracy. With improved data-efficiency, our DEMINE estimator enables statistical testing of dependency at practical dataset sizes. We demonstrate the effectiveness of DEMINE on synthetic benchmarks and real world fMRI data, with application of inter-subject correlation analysis.

Second update:
We thank the authors for updating their paper.
The work is now improving, and is on the right track for publication at a future conference.  There are a few comments on the new results, and suggestions for further improvement:
* The issue of possible false positives due to sample dependence in fMRI data has again been ignored in the rebuttal. Without careful treatment of this effect, these results are vacuous.
* There is still no comparison with competing nonparametric tests on the fMRI data.
* the results linking the COCO and MINE estimators are interesting. Some statements don't make sense, however, eg. "HSIC-based estimators tend to have loose confidence intervals due to the need to bound generalization error of kernels f and g on unseen data points." First, f and g are functions, not kernels. Second, testing with HSIC or COCO does not require generalisation to unseen data points: this is why testing is an easier problem than regression.
For HSIC testing, I am surprised to read in the footnote that the Gamma approximation report significantly more than 5% errors, especially given the Table 4 results that show the correct level. In any case, I very strongly suggest using a permutation approach to obtain the test threshold for HSIC, which is by far the most robust and reliable method. The Gamma approximation has no statistical guarantees, as stated explicitly in the HSIC testing paper of NeurIPS 2007. Permutation gives a guarantee of the correct level. See the NeurIPS paper for details. Once you have verified the correct false positive rate for the permutation threshold, then you can compute the p-value on the alternative.
While the comparison with HSIC is a helpful one, it is also required to compare with the competing method closest to yours, i.e. the Berrett  and Samworth test. In addition, HSIC is a non-adaptive test, but your test is adaptive, so a fairer comparison would be to a modern adaptive test such as "An Adaptive Test of Independence with Analytic Kernel Embeddings."
* The false positive rate in the sanity check is far below the design level of 0.05. This is as I expected, given the use of the Hoeffding bound. This should be stated clearly in the main text, and not disclosed in the final sentence of the final page of the appendix.
====================
Update: thank you for your rebuttal.
"the necessary and sufficient condition of dependency, is more general and is complementary to other techniques that make stronger assumptions about the data. "
The alternative tests listed are nonparametric. That is to say, unlike Pearson correlation, they do not make specific parametric assumptions on the nature of the dependence. Rather they make generic smoothness assumptions (your test also makes such assumptions by the choice of neural network architecture). Thus, comparison with the prior work in Statistics and Machine Learning is relevant, since these tests have the same aims and scope as your tests.
The cited Berrett and Samworth MI test uses a permutation approach to obtaining the test threshold, not an asymptotic approach  (see the results of Section 4 of that paper).   Several of the other cited tests also use a permutation approach for the test threhsold. These tests are therefore relevant prior work.
" If things change very little from one second to the next, the signals could be very similar and may not really be, intuitively, independent samples and may bias result of the study. However, which independence assumptions to use is not in scope for our paper,  because our fMRI study is trying to show that dependency testing works "
In the work cited by Chwialkowski et al, failure to account for the dependence between samples results in excessive false positives. This is because, for dependent data, the effective sample size is reduced, and the tests must be made more conservative to correct for this effect. It is therefore the case that the fMRI results may be false positives.
Re level:"This proof could be experimentally verified ..."  This should be verified. In particular, Hoeffding can be very loose in practice, which is likely to be observed in experiments.
======
The authors propose a procedure for improving neural mutual information estimates, via a combination of data augmentation and cross validation. They then use these estimates in hypothesis testing, on low dimensional toy datasets and on high dimensional real-world fMRI data.
The improved training procedures for MI estimation are of interest, however the hypothesis testing parts of the paper could still be improved.
In hypothesis testing, it is important to verify that the test has the correct level (false positive rate). This is all the more essential when the estimate has required optimisation over parameters. It is not clear from the presentation that this has been confirmed.
There are a number of prior approaches to testing for multivariate statistical dependence in the machine learning and statistics literature (including a 2017 paper which uses mutual information). A small selection is given below, although a literature search will reveal many more papers. In the absence of citation or comparison with any of the prior work on multivariate statistical dependence testing, the current submission is not suitable for publication.
In statistics:
---------------
https://arxiv.org/abs/1711.06642
Nonparametric independence testing via mutual information
Thomas B. Berrett, Richard J. Samworth
2017
Measuring and testing dependence by correlation of distances
Gábor J. Székely, Maria L. Rizzo, and Nail K. Bakirov
Ann. Statist.
Volume 35, Number 6 (2007), 2769-2794.
Large-scale kernel methods for independence testing
Qinyi ZhangEmail Sarah Filippi, Arthur Gretton, Dino Sejdinovic
Statistics and Computing
January 2018, Volume 28, Issue 1, pp 113–130| Cite as
In machine learning:
---------------------
Multivariate tests of association based on univariate tests
Heller, Ruth and Heller, Yair
Advances in Neural Information Processing Systems 29
2016
A Kernel Statistical Test of Independence
Gretton, Arthur and Fukumizu, Kenji and Choon H. Teo and Song, Le and Sch\"{o}lkopf, Bernhard and Alex J. Smola
Advances in Neural Information Processing Systems 20
2008
http://papers.nips.cc/paper/3201-a-kernel-statistical-test-of-independence.pdf
http://proceedings.mlr.press/v70/jitkrittum17a/jitkrittum17a.pdf
An Adaptive Test of Independence with Analytic Kernel Embeddings
Wittawat Jitkrittum, Zoltán Szabó, Arthur Gretton ; ICML 2017, PMLR 70:1742-1751
Time dependence
----------------
It is also the case that if the variables have time dependence, then appropriate corrections must be made for the test threshold, to avoid excessive false positives. Does the fMRI data exhibit time dependence?  For the case of multivariate statistical dependence testing, such corrections are described e.g. in:
https://papers.nips.cc/paper/5452-a-wild-bootstrap-for-degenerate-kernel-tests.pdf
A Wild Bootstrap for Degenerate Kernel Tests
Kacper Chwialkowski, Dino Sejdinovic, Arthur Gretton
NeurIPS 2014


Previous work shows that adversarially robust generalization requires larger sample complexity, and the same dataset, e.g., CIFAR-10, which enables good standard accuracy may not suffice to train robust models. Since collecting new training data could be costly, we focus on better utilizing the given data by inducing the regions with high sample density in the feature space, which could lead to locally sufficient samples for robust learning. We first formally show that the softmax cross-entropy (SCE) loss and its variants convey inappropriate supervisory signals, which encourage the learned feature points to spread over the space sparsely in training. This inspires us to propose the Max-Mahalanobis center (MMC) loss to explicitly induce dense feature regions in order to benefit robustness. Namely, the MMC loss encourages the model to concentrate on learning ordered and compact representations, which gather around the preset optimal centers for different classes. We empirically demonstrate that applying the MMC loss can significantly improve robustness even under strong adaptive attacks, while keeping high accuracy on clean inputs comparable to the SCE loss with little extra computation.

[Post rebuttal: The rebuttal addresses most of my concerns. I have updated my score.]
This paper points out some limitation of the traditional softmax cross-entropy (SCE) loss, e.g., causing indirect and unexpected supervisory signals on the learned features so that the points with low loss values tend to spread over the space sparsely. To remedy this, the authors then proposes the max-mahalanobis center (MMC) loss. By analyzing the sample density, it is proved that the MMC loss has a sample density proportional to the number of data samples in each class. This means MMC loss induces higher feature densities than the SCE loss, thus is expected to be more robust under adverssarial attacks.
I find the paper fairly well written in general, and the arguments are well supported by the development of the theoretical results. There are, however, several questions raised by the reviewer.
One question is that by looking at the definition (8), it seems the MMC loss essentially tries to move data from different classes apart. This seems to be quite similar to the max margin loss. I would expect the authors differentiate the differences between these two. Consequently, I would also like to see empirical comparisons between these two losses.
Another question is even if you have Theorem 2, the transfer from Theorem 2 to robustness does not seem direct, e.g., how do you guarantee when the density form in eq.9 is better than that in eq.7 for robustness?
Then when looking back at the sample density definition in eq. (2). I wonder how the Vol is defined to guarantee eq.2 is a valid density function? e.g., to guarantee the integration equals to 1.
In terms of experiments, since it is claimed that the proposed loss adds little computation cost compared to the SCE loss, I think it is better to include running time comparison in the results.
The last two lines in page 8, what are unseen attacks? From the experiments, it seems that it means the PGD with different steps. I think the authors  should be careful to use the term, because some literature use the term *different attacks* to refer to attacks with different *norms*.
Second line below Table 5: it is said the MMC loss keeps state-of-the-art performance on clean data. However, by looking at Table 4, the state-of-the-art is obvious SCE. Why did you say that?
Others:
1. S_k,\tilde{k}^2 is undefined.
2. How do you solve the minmax problem below eq.8? I guess you use the algorithm in Appendix B.1? Please clarify.


Imitation learning can reproduce policies by observing experts, which poses a problem regarding policy privacy. Policies, such as human, or policies on deployed robots, can all be cloned without consent from the owners. How can we protect our proprietary policies from cloning by an external observer? To answer this question we introduce a new reinforcement learning framework, where we train an ensemble of near-optimal policies, whose demonstrations are guaranteed to be useless for an external observer. We formulate this idea by a constrained optimization problem, where the objective is to improve proprietary policies, and at the same time deteriorate the virtual policy of an eventual external observer. We design a tractable algorithm to solve this new optimization problem by modifying the standard policy gradient algorithm. Our formulation can be interpreted in lenses of confidentiality and adversarial behaviour, which enables a broader perspective of this work. We demonstrate the existence of "non-clonable" ensembles, providing a solution to the above optimization problem, which is calculated by our modified policy gradient algorithm. To our knowledge, this is the first work regarding the protection of policies in Reinforcement Learning.

Summary: The paper introduces a method for generating trajectories which prevent behavioral cloning in a policy gradient setting by learning varying experts which try to minimize the ability of a cloned policy. It runs experiments on a grid world to validate empirically that cloning is unsuccessful.
Recommendation: While this is a novel concept and interesting, I cannot recommend acceptance in its current state. The paper was a bit hard to follow and I found the experiments not robust enough to fully characterize the method at this point. It is unclear whether this method really would prevent cloning given an apples-to-apples comparison. My understanding from the paper -- which was a bit hard to follow -- is that cloned policies were tabular while the APE policies were NNs. I would be more confident in results if more environment variations were tested, the cloned policies used more current and apples-to-apples comparisons, and overall if there were more clear details about the methodology.
Comments:
+ It might be worth perusing the differential privacy and adversarial attack literature to think about whether demonstrations can simply be noised to retain information while crashing performance. This work seems relevant for example (it was put online in June which is sufficiently before the September deadline to mention it I believe): Behzadan, Vahid, and William Hsu. "Adversarial Exploitation of Policy Imitation." arXiv preprint arXiv:1906.01121 (2019).
+ In the discussion:
"We found in our preliminary results that using an RNN classifier which outputs p(c|τ1:t) simply ended up in with either optimal policies or crippled policies. In both cases, there was a relatively minor difference in performance between the policy ensemble and the cloned policy." --> There are no quantitative results for this so either results should be included and discussed or this should be future work.
+ The algorithm box doesn't really add a whole lot of information other than saying that trajectories are collected and then gradients are updated. It would be really nice to have a very clear picture of what's happening at each point in the algorithm. In its current state the paper is hard to follow and decipher this sequence.  See for example Algorithm one in: https://papers.nips.cc/paper/6391-generative-adversarial-imitation-learning.pdf .
+ Notation-wise, R(t) is a bit unusual notation for the RL literature, the advantage is usually r + \gamma V(s') - V(s), where r+\gamma V(s') is the action value Q(s,a). Given that the advantage is denoted as A(s,a), it would be clearer I think to use the Q(s,a) notation. Also the notation changes from section 2.2 to section 3.2 from A(s,a) to A(t). Keeping consistent notation would make this paper a lot easier to read.
+ The related work section is in the middle of the paper. it'd be nice to have it earlier to set the context of the work.
+ In the multiple policies section, a recent work has shown how to learn multiple policies from multiple experts using a mixture of experts framework -- though they frame it as options: Henderson, Peter, Wei-Di Chang, Pierre-Luc Bacon, David Meger, Joelle Pineau, and Doina Precup. "Optiongan: Learning joint reward-policy options using generative adversarial inverse reinforcement learning." In Thirty-Second AAAI Conference on Artificial Intelligence. 2018.
+ Part of the way this defeats behaviour cloning is through the assumption that there are multiple trajectories to be learned from. It would be interesting to see if methods like the one above or any of the others mentioned can recover optimal performance from noisy trajectories by similarly learning multiple policies. In fact,
+ "Policy Gradient (PG) (Sutton et al., 2000) and its variants (Schulman et al., 2015) aim to directly learn
the optimal policy π, parameterized by θ." --> I think some other citations of variants should be added for the final version instead of only referencing Schulman 2015. There are a lot now, so maybe adding PPO, DDPG, and a few others might be nice. Otherwise you could also just cut out the variants bit since it's not necessary.
+ All first quotation marks are backwards in the document
+ I think the experiments ran were a bit lacking in robustness and details. Since this is an adversarial method, I would expect more variance across seeds and 3 seeds may not be enough to characterize this. Table 1 has +/- but does not state what this represents. Standard Deviation or Standard error? Does Table 1 represent returns for rolled out policies after learning or across all episode returns during learning? For the behavioural cloning method, it says a "tabular policy" was trained. Does this mean that the experts were trained using policy gradients and neural networks while the behavioural cloning method used a tabular policy? If so, I think this would be at a detriment to the method being tricked. I think it is a necessary condition to validate this method across several gridworld environment variations, seeds, and with more robust cloning methods (if in fact the behavioural policy was underpowered (tabular vs. nn). Overall, it would be great to have more details. While the visualizations of the gridworld itself were nice, I think they took up a lot of space which could be replaced with more detailed explanations and robust quantitative results.


In recent years there has been a rapid increase in classification methods on graph structured data. Both in graph kernels and graph neural networks, one of the implicit assumptions of successful state-of-the-art models was that incorporating graph isomorphism features into the architecture leads to better empirical performance. However, as we discover in this work, commonly used data sets for graph classification have repeating instances which cause the problem of isomorphism bias, i.e. artificially increasing the accuracy of the models by memorizing target information from the training set. The problem does not vanish even in consideration of node labels. This prevents fair competition of the algorithms and raises a question of the validity of the obtained results. First, we characterize this effect theoretically by reducing the graph classification to a weighted classification problem and estimating the corresponding generalization gap. Then we analyze 54 data sets, previously extensively used for graph-related tasks, on the existence of isomorphism bias, give a set of recommendations to machine learning practitioners to properly set up their models, and open source new data sets for the future experiments.

The paper is concerned with the presence of isomorphism bias in commonly used graph learning benchmarks. In particular, the paper analyzes the amount of isomorphic graphs in 54 graph datasets and evaluates the performance of three graph classification methods under two isomorphism settings.
Careful analyses of commonly used benchmarks can be important contributions that provide new insights into the performance of state-of-the-art models. The present paper's results on graph isomorphism properties can indeed be valuable for the ablation of models and testing their performance with regard to this property. I also found the relatively high label disagreements on some datasets (even under stronger isomorphism constraints) to be a surprising and useful result.
However, the main assumption of the paper -- which equates the quality of a graph learning benchmark with the amount of isomorphic graphs that it contains, i.e., the lower the better -- seems questionable.
The paper argues that isomorphic graphs are akin to duplicate images in computer vision and should be removed from a dataset. While completely identical graphs are certainly problematic, the case seems different for isomorphic graphs. In the latter, a learning method is required to identify the correct bijection form V_1 to V_2 which is a non-trivial task. Testing on isomorphic graphs evaluates the ability of a model to infer these equivalence classes from data which is an important property. Moreover, being able to capture the equivalence relation can be important for various graph learning tasks, e.g., to facilitate that two topologically equivalent graphs are be classified similarly.  Going back to the computer vision analogy: it seems a more adequate comparison for graph isomorphism would be translation and scale invariance which are certainly desirable properties for CV models.
In addition, the dataset analysis could also be improved. For instance, the SYNTHETIC dataset includes continuous node attributes that are essential for classification and make the graphs non-isomorphic (when considering, for instance, each attribute vector as a unique node label). However, the attributes are not considered in the analysis what leads to a large number of isomorphic graphs. On a side note: the paper also incorrectly attributes the SYNTHETIC dataset to (Morris et al, 2016), but it is in fact from [1]. The synthetic dataset of Morris et al (SYNTHIE) does not consist of isomorphic graphs, while the SYNTHETIC dataset of [1] does so intentionally.
The results of Section 5 seem also not very surprising: After removing node labels, it is expected that the number of isomorphic graphs increases since a discriminating feature has been removed. Moreover, when accounting for node labels, many standard benchmarks seem to consist of significantly less isomorphic and mismatched graphs (as can be seen in the appendix).
Since graph isomorphism != graph identity, the assumption Y_iso \sub Y_train in Property 6 seems also not appropriate. The results of Theorem 6.1 on the other hand seems straightforward and would hold for any classification task for which the true label for an equivalence class of instances is known.