Representation learning in reinforcement learning (RL) algorithms focuses on extracting useful features for choosing good actions. Expressive representations are essential for learning well-performed policies. In this paper, we study the relationship between the state representation assigned by the state extractor and the performance of the RL agent. We observe that representations assigned by the better state extractor are more scattered than which assigned by the worse one. Moreover, RL agents achieving high performances always have high rank matrices which are composed by their representations. Based on our observations, we formally define expressiveness of the state extractor as the rank of the matrix composed by representations. Therefore, we propose to promote expressiveness so as to improve algorithm performances, and we call it Expressiveness Promoted DRL. We apply our method on both policy gradient and value-based algorithms, and experimental results on 55 Atari games show the superiority of our proposed method.

The paper aims at characterizing and discussing the impact of the expressivity of a state representation on the performance. It then discusses three possible ways of improving performance with a specific regularization on the state representation. Overall the idea
of studying the impact of expressiveness and comparing different ways of ensuring it is interesting. However, some parts in the paper are not well-supported and many important elements to understand the experiments are lacking.
Some elements are not well supported and probably not true as stated.
"For fixed h, the expressiveness is related to the MDP M: 1) if the transition p(xt+1|xt, at)π(at|xt) of the MDP is not ergodic, i.e., it can only visit a subset of observations, then the matrix {h(X1 ), · · · , h(Xt ), · · · } will more possibly be low rank; 2) if the reward is very sparse, the representation matrix will be low rank."
If you compare two MDPs, where the first one has two states and is ergodic, while the second one has many more states but is not ergodic, do you think your consideration still applies? For a given h, why would the sparsity of the reward function plays any role?
The paper states that the "method can be widely applied to current DRL algorithms, with little extra computational cost." However, it is not explained clearly how the regularization is enforced. If a supplementary loss is used and minimized at each step, the computational cost is not negligeable.
In the experiment section, the protocol is not clear. For instance the coefficient \alpha is apparently set differently for some of the games? How are they chosen? And why are the hyper-parameters chosen not given?
"However, as the observation type and the intrinsic reward mechanism vary considerably for each game, we also use some other regularization term and coefficient for some of games."
Why are there no details related to the NN architecture?
In Table 1, what does it mean "Times Better" and "Times Worse"?
Other comments:
- In Definition 1, what does it mean X_t \sim \mathcal M? \mathcal M is an MDP, not a distribution. Are the X_t taken following a given policy? are they taken on a given trajectory sequentially or i.i.d.?
- (minor) "From the above description, it is easy to see that the performance of an RL agent depends on two parts. First, it depends on whether the state extractor is good. With a good state extractor, the representation which is a depiction of the observation will retain necessary information for taking actions. Second, it depends on the accuracy of the policy: whether the feed-forward model can correctly take the optimal action given the state." Do the two elements in that paragraph mean the same: "A good state extractor provides an abstract representation from which a performant policy can be obtained?"
- Figure 2: The name of the ATARI game is not mentioned.
- In section 2.2, the MDP framework is introduced (with a state space \mathcal S) but there is no mention of the concept of observation x that is used throughout afterwards.
- In the conclusion, it is stated that "Experiments of A3C and DQN on 55 Atari games demonstrate that ExP DRL can promote their performances significantly." However, the algorithm is not tested on 55 games with DQN.
- The related work discusses papers about state representation but even more directly related to this paper, other papers have also discussed the importance of disentangled representation or entropy maximization for deep RL: https://arxiv.org/abs/1707.08475, https://arxiv.org/abs/1809.04506, ... And papers that discuss expressiveness in deep learning such as https://arxiv.org/pdf/1606.05336.pdf should also be discussed.
- There are many typos/english errors.


LEARNING FACTORIZED MULTIMODAL REPRESENTATIONS
Learning multimodal representations is a fundamentally complex research problem due to the presence of multiple heterogeneous sources of information. Although the presence of multiple modalities provides additional valuable information, there are two key challenges to address when learning from multimodal data: 1) models must learn the complex intra-modal and cross-modal interactions for prediction and 2) models must be robust to unexpected missing or noisy modalities during testing. In this paper, we propose to optimize for a joint generative-discriminative objective across multimodal data and labels. We introduce a model that factorizes representations into two sets of independent factors: multimodal discriminative and modality-specific generative factors. Multimodal discriminative factors are shared across all modalities and contain joint multimodal features required for discriminative tasks such as sentiment prediction. Modality-specific generative factors are unique for each modality and contain the information required for generating data. Experimental results show that our model is able to learn meaningful multimodal representations that achieve state-of-the-art or competitive performance on six multimodal datasets. Our model demonstrates flexible generative capabilities by conditioning on independent factors and can reconstruct missing modalities without significantly impacting performance. Lastly, we interpret our factorized representations to understand the interactions that influence multimodal learning.

This paper presents 'Multimodal Factorization model' that factorizes representations into shared multimodal discriminative factors and modality specific generative factors. This work applies 'Wassertein Auto-Encoders' by Tolstikhin et al (with proofs that this setup works in the multimodal case) for handling factorized joint distributions over the multimodal space. Can this method be considered as a generalization of the wasserstein autoencoder based method with a broader application? - the authors should discuss this more broadly in the paper.
Pros:
- There has been many recent work in the area of disentangling joint representations for improving generative auto-encoding architectures using VAEs, GANs, WAE and some variants of these. This work falls in this category with many interesting experiments showing SOTA generation and discrimination results on several tasks.
- This work is practical due to its robustness to noisy and/or missing data for one or more of the modalities in a multimodal machine learning classification (or generation) problem. Application of this technique for continuous multimodal time series data modeling and prediction for high accuracy requirement applications is very promising.
- The methods seems to be easily portable to other tasks. The authors say that they will make the code available to other researchers.
Cons:
- Some more comparison to other disentangling approaches such as beta-VAE, InfoGAN and partitioned VAE methods would have been useful for understanding the advantages and disadvantages of this techniques. (The authors do add a note about comparison with partitioned VAE method in the Appendix)
- For generation and classification tasks, the authors have chosen the tasks for digit recognition and sentiment analysis - I wonder if the results would hold for other types of multimodal tasks.
Overall the paper is very well-written with many experiments to support the claims.


This work presents a method for active anomaly detection which can be built upon existing deep learning solutions for unsupervised anomaly detection. We show that a prior needs to be assumed on what the anomalies are, in order to have performance guarantees in unsupervised anomaly detection. We argue that active anomaly detection has, in practice, the same cost of unsupervised anomaly detection but with the possibility of much better results. To solve this problem, we present a new layer that can be attached to any deep learning model designed for unsupervised anomaly detection to transform it into an active method, presenting results on both synthetic and real anomaly detection datasets.

(Since the reviewer was unclear about the OpenReview process, this review was earlier posted as public comment)
Most claims of novelty can be clearly refuted such as the first sentence of the abstract "...This work presents a new approach to active anomaly detection..." and the paper does not give due credit to existing work. Current research such as Das et al. which is the most relevant has been deliberately not introduced upfront with other works (because it shows lack of the present paper's novelty) and instead deferred to later sections. The onus of a thorough literature review and laying down a proper context is on the authors, not the reviewers. Detailed comments are below.

      1. Related Works: "...active anomaly detection remains an under-explored approach to this problem..."
          - Active learning in anomaly detection is well-researched (AI2, etc.). See related works section in Das et al. 2016 and:
            - K. Veeramachaneni, I. Arnaldo, A. Cuesta-Infante, V. Korrapati, C. Bassias, and K. Li, "Ai2: Training a big data machine to defend," International Conference on Big Data Security, 2016.

      2. "To deal with the cold start problem, for the first 10 calls of select_top...":
          - No principled approach to deal with cold start and one-sided labels (i.e., the ability to use labels when instances from only one class are labeled.)

      3. Many arbitrary hyper parameters as compared to simpler techniques:
          - The number of layers, nodes in hidden layers.
            - The number of instances (k) per iteration
            - The number of pretraining iterations
            - The number of times the network is retrained (100) after each labeling call
            - Dealing with cold start (10 labeling iterations of 10 labels each, i.e. 100 labels).

      4. The paper mentions that s(x) might not be differentiable. However, the sigmoid form of s(x) is differentiable.

      5. Does not acknowledge the well-known result that mixture models are unidentifiable. The math in the paper is mostly redundant. Some references:
          - Identifiability  Of  Nonparametric  Mixture  Models And  Bayes  Optimal  Clustering (pradeepr/arxiv npmix v.pdf)" target="_blank" rel="nofollow">https://www.cs.cmu.edu/ pradeepr/arxiv npmix v.pdf)
          - Semiparametric estimation of a two-component mixture model by Bordes, L., Kojadinovic, I., and Vandekerkhove, P., Annals of Statistics, 2006 (https://arxiv.org/pdf/math/0607812.pdf)
          - Inference for mixtures of symmetric distributions by David R. Hunter, Shaoli Wang, Thomas P. Hettmansperger, Annals of Statistics, 2007 (https://arxiv.org/pdf/0708.0499.pdf)
          - Inference on Mixtures Under Tail Restrictions by K. Jochmans, M. Henry, and B. Salanie, Econometric Theory, 2017 (http://econ.sciences-po.fr/sites/default/files/file/Inference.pdf)

      6. Does not acknowledge existing work that adds classifier over unsupervised detectors (such as AI2). This is very common.
        - This is another linear model (logistic) on top of transformed features. The difference is that the transformed features are from a neural network and optimization can be performed in a joint fashion. The novelty is marginal.

      7. While the paper argues that a prior needs to be assumed, it does not use any in the algorithm. There seems to be a disconnect. It also does not acknowledge that AAD (LODA/Tree) does use a prior. Priors for anomaly proportions in unsupervised algorithms are well-known (most AD algos support that such as OC-SVM, Isolation Forest, LOF, etc.).

      8. Does not compare against current state-of-the-art Tree-based AAD
          - Incorporating Expert Feedback into Tree-based Anomaly Detection by Das et al., KDD, 2017.

      9. The 'Generalized' in the title is incorrect and misleading. This is specific to deep-networks. Stacking supervised classifiers on unsupervised detectors is very common. See comments on related works.

      10. Does not propose any other query strategies than greedily selecting top.

      11. Question: Does this support streaming?


Optimistic Mirror Descent in Saddle-Point Problems: Going the Extra (Gradient) Mile
Owing to their connection with generative adversarial networks (GANs), saddlepoint problems have recently attracted considerable interest in machine learning and beyond. By necessity, most theoretical guarantees revolve around convexconcave (or even linear) problems; however, making theoretical inroads towards efficient GAN training depends crucially on moving beyond this classic framework. To make piecemeal progress along these lines, we analyze the behavior of mirror descent (MD) in a class of non-monotone problems whose solutions coincide with those of a naturally associated variational inequality -a property which we call coherence. We first show that ordinary, "vanilla" MD converges under a strict version of this condition, but not otherwise; in particular, it may fail to converge even in bilinear models with a unique solution. We then show that this deficiency is mitigated by optimism: by taking an "extra-gradient" step, optimistic mirror descent (OMD) converges in all coherent problems. Our analysis generalizes and extends the results of Daskalakis et al. [2018] for optimistic gradient descent (OGD) in bilinear problems, and makes concrete headway for provable convergence beyond convex-concave games. We also provide stochastic analogues of these results, and we validate our analysis by numerical experiments in a wide array of GAN models (including Gaussian mixture models, and the CelebA and CIFAR-10 datasets).

This work provides the converge proof of the last iterates of two stochastic methods (almost surely) that the author called  mirror descent and optimistic mirror descent under an assumption weaker than monotonicity called coherence.
Roughly, the definition of coherence is the equivalence between being a  saddle point and the solution of the Minty variational inequality.
Overall, I think that this paper try to tackle an interesting problem which is to prove convergence of saddle point algorithms under weaker assumption than monotonicity of the operator.
However, I have some concerns:
- I think that the properties of coherent saddle point could be more investigated. For instance is the set of coherent saddle point connected ? It would be very relevant for GANs. You claim that "neither strict, nor null coherence imply a unique solution to (SP)," but I do not see any proof of that statement (both provided examples have a unique SP). I agree that you can set $g$ to $0$ in some directions to get an affine space a of saddle points but is there examples where the set of solution is not an affine space (intersected with the constraints) ?
- First of all the results are only asymptotic. (I agree that it can be mitigated saying that there is (almost) no results on non-monotone VI and it is a first step to try to handle non-convexity of the objective functions.)
- One big pro of this work might have been new proof techniques to handle non-monotonicity in variational inequalities but the coherence assumption looks like to be the weakest condition to use the standard proof technique of convergence of the (MD) and (OMD). Nevertheless, this work is still interesting since it handles in a subtle way stochasticity (I did not have time to check Theorem 2.18 [Hall & Heyde 1980], I would be good to repeat it in the appendix for self-completeness)
- This work could be easily extended to non zero-sum games which is crucial in practice since most of the state of the art GANs (such as WGAN with gradient penalty or non saturating GAN) are non zero-sum games.
- Are you sure of the use of the denomination Optimistic mirror descent ? What you are presenting is the extragradient method. These two methods are slightly different, If you look at (5) in (Daskalaki et al., 2018) you'll notice that the updates are slightly different from you (OMD), particularly (OMD) require two gradient computations per iteration whereas (5) in (Daskalaki et al., 2018) requires only one. (it just requires to memorize the previous gradient)
Minor comment:
- For saddle point (and more generally variational inequalities) Mirror descent is no longer a descent algorithm. The name used by the literature is mirror-prox method (see Juditsky's paper)
- in (C.1) U_n is not defined anywhere but I guess it is $\hat g_n - g(X_n)$.
- Some cited paper are published paper but cited as arXiv paper.
- Lemma D.1 could be extended to the case (\sigma \neq 0) but the additional noise term might be hard to handle to get a result similar as Thm 4.1
for $\sigma \neq 0$.


Current Convolutional Neural Network (CNN)-based object detection models adopt strictly feedforward inference to predict the final detection results. However, the widely used one-way inference is agnostic to the global image context and the interplay between input image and task semantics. In this work, we present a general technique to improve off-the-shelf CNN-based object detection models in the inference stage without re-training, architecture modification or ground-truth requirements. We propose an iterative, bottom-up and top-down inference mechanism, which is named conscious inference, as it is inspired by prevalent models for human consciousness with top-down guidance and temporal persistence. While the downstream pass accumulates category-specific evidence over time, it subsequently affects the proposal calculation and the final detection. Feature activations are updated in line with no additional memory cost. Our approach advances the state of the art using popular detection models (Faster-RCNN, YOLOv2, YOLOv3) on 2D object detection and 6D object pose estimation.

The paper proposes a method called cautious inference to improve inference accuracy for object detection models. The main idea is inspired by the previous work of Guided Perturbations, which is applied to fully convolutional networks to improve the segmentation/accuracy accuracy purely during inference time.  The original idea is to use the predicted labels of the network as pseudo ground truths (after making the predictions to be a one-hot vector), and then back propagate the error signals to the network input to get the gradients. And finally the gradients are added back to the original inputs to perform another round of prediction. Here the inputs can be either the original image, or some intermediate feature maps. Experiments are shown for both 2D and 6D object detections.
Comments:
- I think overall it is an interesting idea to directly alter the input of the network in order to fit to the testing distribution. However, the motivation and story told in the introduction is a bit of an oversell compared to the experiment validation section. Most of the results shown are just doing training and testing of images drawn from the *same* distribution. Like coco train and test, or VOC train and test. It would be great to see if the cautious inference would work when the distribution is different. For example "elephant in the room" case, or new object categories are added during testing.
- I am actually curious to see this method can be used to improve the AP on the *training* set as well, just to understand it better -- is it trying to recover the generalization error of the network, or it is doing some implicit context reasoning inference that can help training as well.
- It might be better to compare/combine the method to other inference-only improvements for object detection. For example there is soft-NMS,
Bodla, Navaneeth, et al. "Soft-nms—improving object detection with one line of code." Computer Vision (ICCV), 2017 IEEE International Conference on. IEEE, 2017.
 - I am not sure I fully understand B-box part: I think it is easy to have B-obj and B-cls as one can just take the max of the class prediction and then use the inferred class label for one-hot vector construction, but I am confused about the box part as no ground-truth is given during testing. In Table 2 I also cannot find BP improving performance by itself in anyway.
- For COCO, please report results on test-dev set, the minival set images are used only for validation.


KNOWLEDGE FLOW: IMPROVE UPON YOUR TEACH- ERS
A zoo of deep nets is available these days for almost any given task, and it is increasingly unclear which net to start with when addressing a new task, or which net to use as an initialization for fine-tuning a new model. To address this issue, in this paper, we develop knowledge flow which moves 'knowledge' from multiple deep nets, referred to as teachers, to a new deep net model, called the student. The structure of the teachers and the student can differ arbitrarily and they can be trained on entirely different tasks with different output spaces too. Upon training with knowledge flow the student is independent of the teachers. We demonstrate our approach on a variety of supervised and reinforcement learning tasks, outperforming fine-tuning and other 'knowledge exchange' methods.

This paper proposes to feed the representations of various external "teacher" neural networks of a particular example as inputs to various layers of a student network.
The idea is quite intriguing and performs very well empirically, and the paper is also well written.  While I view the performance experiments as extremely thorough, I believe the paper could possibly use some additional ablation-style experiments just to verify the method actually operates as one intuitively thinks it should.
Other Comments:
- Did you verify that in Table 3, the p_w values for the teachers trained on the more-relevant C10/C100 dataset are higher than the p_w value for the teacher trained on the SVHN data?  It would be interesting to see the plots of these p_w over the course of training (similar to Fig 1c) to verify this method actually operates as one intuitively believes it should.
- Integrating the teacher-network representations into various hidden layers of the student network might also be considered some form of neural architecture search (NAS)  (by including parts of the teacher network into the student architecture).
See for example the DARTS paper: https://arxiv.org/abs/1806.09055
which similarly employs mixtures of potential connections.
Under this NAS perspective, the dependence loss subsequently distills the optimal architecture network back into the student network architecture.
Have you verified that this method is not just doing NAS, by for example, providing a small student network with a few teacher networks that haven't been trained at all? (i.e. should not permit any knowledge flow)
- Have the authors considered training the teacher networks jointly with the student? This could be viewed as teachers learning how to improve their knowledge flow (although might require large amounts of memory depending on the size of the teacher networks).
- Suppose we have an L-layer student network and T M-layer teacher networks.
Does this imply we have to consider O(L*M*T) additional weight matrices Q?
Can you comment on the memory requirements?
- The teacher-student setup should be made more clear in Tables 1 and 2 captions (took me some time to comprehend).
- The second and third paragraphs are redundant given the Related Work section that appears later on. I would like to see these redundancies minimized and the freed up  space used to include more results from the Appendix in the main text.


Robustness of neural networks has recently been highlighted by the adversarial examples, i.e., inputs added with well-designed perturbations which are imperceptible to humans but can cause the network to give incorrect outputs. In this paper, we design a new CNN architecture that by itself has good robustness. We introduce a simple but powerful technique, Random Mask, to modify existing CNN structures. We show that CNNs with Random Mask achieve state-of-the-art performance against black-box adversarial attacks without applying any adversarial training. We next investigate the adversarial examples which "fool" a CNN with Random Mask. Surprisingly, we find that these adversarial examples often "fool" humans as well. This raises fundamental questions on how to define adversarial examples and robustness properly.

The authors propose a simple method for increasing the robustness of convolutional neural networks against adversarial examples. This method is simple but seems to achieve surprisingly good results. It consist in randomly remove neurons from the network architecture. The deleted neurons are selected before training and remain deleted during the training and test phase.  The authors also study the adversarial examples that still fool the network after applying their method and find than those examples also fool human. This finding raises the question of what is an adversarial example if both humans and networks are fooled by the same example.
Using Random Masks in neural network is not a new idea since it was already proposed for DropOut or DropConnect (Regularization of Neural Networks using DropConnect, ICML2013) and in the context of adversarial attacks (Dhillon et al. 2018)  as reported by the authors. The discussion (Section 2) about the impact of random masks on what convolution layers capture in the spatial organisation of the input is interesting: whereas standard CNNs focus on detecting the presence of a feature in the output, random mask could force the CNN layers to learn how a specific feature distributes on the whole input maps. This limitation of the CNN has already been pointed up and solutions have been proposed for example Capsule Networks (Dynamic Routing Between Capsules, NIPS 2017). This intuition is experimentally supported by a simple random shuffle by block of the input image  (Appendix A).
In Section 3, the authors present a large number of experiments to demonstrate the robustness of their method. Most of the details are given in the 13 (!) pages of appendix. Experiments against black-box attack, random noise, white-box attack, grey-box are presented. Most of the experiments are on CIFAR10 but one experiment is also presented on MNIST. One could regret that only one architecture of CNN is tested (ResNet18) except for gray-box attack, for which DenseNet121 and VG19 are tested. One could ask why the type of models tested is not consistent across the different experiments.  For black-box attack, random masks compare favourably to Madry’s defence. For white box defence, Random Mask is not compared to another defence method, which seems a weakness to me but I am not familiar enough with papers in this area to estimate if this is a common practice. In most of the experiments, the drop ratio is between 0.5 and 0.9, which seems to indicate that the size the initial network could be reduced by more than 50% to increase the robustness to attack. This ratio is larger than what is usually used for dropout (0.5 at most).
In section 3.3, different strategies for random masks are explored : where to apply random masks, random mask versus random channels, random masks versus same masks. Results are given in table 2. The caption of Table 2 could be more explicit : what are the presented percent ?
Experiments on masking shallow versus deep layers are interesting. Best results for robustness are obtained with masking shallow layers at quite a high ratio (0.9). One could ask if this result could be due to the type or the parameters of adversarial attacks which are not adapted to such a high sparseness on shallow layers or to the specific kind of sparseness induced by the masks. A comparison to a regular network with the same number of free parameters as the masked network could give insight on this aspect.
pros : simple to implement, good robustness shown agains a variety of attack types
cons : mainly tested on a single architecture (ResNet) and on a single datatbase CIFAR. Maybe not robust against the latest techniques of adversarial attack.


BABYAI: A PLATFORM TO STUDY THE SAMPLE EFFI- CIENCY OF GROUNDED LANGUAGE LEARNING
Allowing humans to interactively train artificial agents to understand language instructions is desirable for both practical and scientific reasons. Though, given the lack of sample efficiency in current learning methods, reaching this goal may require substantial research efforts. We introduce the BabyAI research platform, with the goal of supporting investigations towards including humans in the loop for grounded language learning. The BabyAI platform comprises an extensible suite of 19 levels of increasing difficulty. Each level gradually leads the agent towards acquiring a combinatorially rich synthetic language, which is a proper subset of English. The platform also provides a hand-crafted bot agent, which simulates a human teacher. We report estimated amount of supervision required for training neural reinforcement and behavioral-cloning agents on some BabyAI levels. We put forward strong evidence that current deep learning methods are not yet sufficiently sample-efficient in the context of learning a language with compositional properties.

This paper focuses on grounded language learning with a human in the loop, in the sense where the language is synthetic, the environment is a 2D grid world, and the human is a simulated human teacher implemented using heuristics. This setup is dubbed the BabyAI platform, and includes curriculum learning over 19 levels of increasing difficulty.
Overall, the BabyAI platform is conceptually similar to numerous previous works that seek to learn grounded language in simulation environments. These efforts differ along various axes, for example visual realism, 2D vs 3D, partially vs. fully observed, different tasks, world-state manipulation or not, etc. The main original aspect of the BabyAI platform is the simulated human-teacher.
Strengths
- Learning with a human in the loop is an extremely important problem to study, although currently efforts are hampered by cost, lack or reproducibility, and the sample inefficiency of existing learning methods. This paper addresses all three of these issues, albeit by removing the human and natural language. This is simultaneously the greatest weakness of this approach. The contribution of this paper therefore rests on the quality/interestingness/utility of the provided synthetic language and the synthetic teacher.
- Fortunately, the synthetic language does exhibit interesting compositional properties, it is readily extensible, it has the appealing property that it can be readily interpreted as a subset of english, and it is accompanied by a verifier to check if the specified actions were completed.
Weaknesses
- If the ultimate goal is learning with a human in the loop, the usefulness of the synthetic teacher is not clear, particularly as it is apparently easier to imitate from an RL trained agent than the teacher. The explanation 'This can explained by the fact that the RL expert has the same neural network architecture as the learner' does no seem obvious to me.
- Regarding the human in the loop, since this is aspirational and not an aspect of the paper, the title of the paper does not seem reflective of its content (even with the 'First steps' qualifier).
- If the main unique aspect is the simulated human-teacher, it is not clear why it is necessary to create a new environment, rather than re-using an existing environment. The effect of this is to limit comparisons with recent work and an increasing fragmentation of research across tasks that are related but can’t be compared.
Summary:
This paper represents an important direction, in that it provides a testbed for studying the sample efficiency of grounded language learning in a simplified (yet still challenging and compositional) environment. I believe the environment and the provided synthetic language and verifier will prove useful to the community, and despite some reservations about the title and the simulated human-teacher, I recommend acceptance.


A GENERATIVE MODEL FOR ELECTRON PATHS
Chemical reactions can be described as the stepwise redistribution of electrons in molecules. As such, reactions are often depicted using "arrow-pushing" diagrams which show this movement as a sequence of arrows. We propose an electron path prediction model (ELECTRO) to learn these sequences directly from raw reaction data. Instead of predicting product molecules directly from reactant molecules in one shot, learning a model of electron movement has the benefits of (a) being easy for chemists to interpret, (b) incorporating constraints of chemistry, such as balanced atom counts before and after the reaction, and (c) naturally encoding the sparsity of chemical reactions, which usually involve changes in only a small number of atoms in the reactants. We design a method to extract approximate reaction paths from any dataset of atom-mapped reaction SMILES strings. Our model achieves excellent performance on an important subset of the USPTO reaction dataset, comparing favorably to the strongest baselines. Furthermore, we show that our model recovers a basic knowledge of chemistry without being explicitly trained to do so.

The paper presents a novel end-to-end mechanistic generative model of electron flow in a particular type of chemical reaction (“Linear Electron Flow” reactions) . Interestingly, modeling the flow of electrons aids in the prediction of the final product of a chemical reaction over and above problems which attack this “product prediction problem” directly. The method is also shown to generalize well to held-out reactions (e.g. from a chemistry textbook).
General Impressions
+ For me the biggest selling point is that it improves performance in predicting the ultimate reaction outcome. It should do because it provides strictly more supervision, but it’s great that it actually does.
+ Because it models the reaction mechanism the model is interpretable, and it’s possible to enforce constraints, e.g. that dynamics are physically possible.
+ Generalises outside of the dataset to textbook problems :-)
+ Well-founded modeling choices and neural network architectures.
- Only applies to a very particular type of reaction (heterolytic LEF).
- Requires supervision on the level of electron paths. This seems to inhibit applying the model to more datasets or extending it to other types of reactions.
- Furthermore the supervision extraction does not seem take advantage of symmetries noted in the section(s) about difficulty evaluating inference.
- It would be nice to margin out the electron flow model and just maximize the marginal likelihood for the product prediction problem.
Novelty
I’m not an expert on the literature of applying machine learning to the problems of reaction {product, mechanism} prediction but the paper appears to conduct a thorough review of the relevant methods and occupy new territory in terms of the modeling strategy while improving over SOTA performance.
Clarity
The writing/exposition is in general extremely clear. Nicely done. There are some suggestions/questions which I think if addressed would improve clarity.
Ways to improve the paper
1. Better motivate the use of machine learning on this problem. What are the limitations of the arrow-pushing models?
2. Explain more about the Linear Electron Flow reactions, especially:
- Why does the work only consider “heterolytic” LEF reactions, what other types of LEF reactions are omitted?
- Is the main blocker to extending the model on the modeling front or the difficulties of extracting ground-truth targets? It appears to be the latter but this could be made more clear. Also that seems to be a pretty severe limitation to making the algorithm more general. Could you comment on this?
Questions
1. Is splitting up the electron movement model into bond “removal” and “addition” steps just a matter of parameterization or is that physically how the movements work?
2. It appears that Jin et al reports Top 6/8/10 whereas this work reports Top 1/3/5 accuracy on the USPTO dataset. It would be nice if there was overlap :-). Do your Top 6/8/10 results with the WLDN model agree with the Jin et al paper?
Nits
Section 2.3, first paragraph “...(LEF) topology is by far the most important”: Could you briefly say why? It’s already noted that they’re the most common in the database. Why?
Section 3.ElectionMovement, first paragraph. “Observer that since LEF reactions are a single path of electrons…”. Actually, it’s not super clear what this means from the brief description of LEF. Can you explain these reactions in slightly more detail?
Section 3.ElectionMovement, second paragraph. “Differently, the above distribution can be split…”. Awkward phrasing. How about “In contrast, the above distribution can be split…”.
Section 3.Training, last sentence “...minibatches of size one reaction”. Slightly awkward phrasing. Maybe “...minibatches consisting of a single reaction”?
Section 5.2, second sentence. “However, underestimates the model’s actual predictive accuracy…”. It looks like a word accidentally got deleted here or something.
Section 5.2, paragraph 4. “To evaluate if our model predicts the same major project”... Did you mean “the same major product”?


SLALOM: FAST, VERIFIABLE AND PRIVATE EXECUTION OF NEURAL NETWORKS IN TRUSTED HARDWARE
As Machine Learning (ML) gets applied to security-critical or sensitive domains, there is a growing need for integrity and privacy for outsourced ML computations. A pragmatic solution comes from Trusted Execution Environments (TEEs), which use hardware and software protections to isolate sensitive computations from the untrusted software stack. However, these isolation guarantees come at a price in performance, compared to untrusted alternatives. This paper initiates the study of high performance execution of Deep Neural Networks (DNNs) in TEEs by efficiently partitioning DNN computations between trusted and untrusted devices. Building upon an efficient outsourcing scheme for matrix multiplication, we propose Slalom, a framework that securely delegates execution of all linear layers in a DNN from a TEE (e.g., Intel SGX or Sanctum) to a faster, yet untrusted, co-located processor. We evaluate Slalom by running DNNs in an Intel SGX enclave, which selectively delegates work to an untrusted GPU. For canonical DNNs (VGG16, MobileNet and ResNet variants) we obtain 6× to 20× increases in throughput for verifiable inference, and 4× to 11× for verifiable and private inference.

The authors propose a new method of securely evaluating neural networks. The approach builds upon existing Trusted Execution Environments (TEE), a combination of hardware and software that isolates sensitive computations from the untrusted software stack. The downside of TEE is that it is expensive and slow to run. This paper proposes outsourcing the linear evaluation portions of the DNN to an untrusted stack that's co-located with the TEE. To achieve privacy (i.e., the input isn't revealed to the untrusted evaluator), the approach adds a random number r to the input vector x, evaluates f(x+r) on the untrusted stack, then subtracts off f(r) from the output. This limits the approach to be applicable to only linear functions. To achieve integrity (verify the correctness of the output), the paper proposes testing with random input vectors (an application of Freivalds theorem, which bounds the error probability). The techniques for integrity and privacy works only on integer evaluations, hence the network weights and inputs need to be quantized. The paper tries to minimize degradation in accuracy by quantizing as finely as numerically allowable, achieving <0.5% drop in accuracy on two example DNNs. Overall, compared to full evaluation in a TEE, this approach is 10x faster on one DNN, and 40x to 64x faster on another network (depending on how the network is formulated).
Disclaimer: I am a complete outsider to the field of HW security and privacy. The paper is very readable, so I think I understand its overall gist. I found the approach to be novel and the results convincing, though I may be missing important context since I'm not familiar with the subject.
To me, the biggest missing piece is a discussion of the limitations of the approach. How big of a network can be evaluated this way? Is it sufficient for most common applications? What are the bottlenecks to scaling this approach?
It's also not clear why integrity checks are required. Is there a chance that the outsourcing could result in incorrect values? (It's not obvious why it would.)
Lastly, a question about quantization. You try to quantize as finely as possible (to minimize quantization errors) by multiplying by the largest power of 2 possible without causing overflow. Since quantization need to be applied to both input and network weights, does this mean that you must also bound the scale of the input? Or do you assume that the inputs are pre-processed to be within a known scale? Is this possible for intermediate outputs (i.e., after the input has been multiplied through a few layers of the DNN)?
Pros:
- Simple yet effective approach to achieve the goals laid out in the problem statement
- Clearly written
- Thorough experiments and benchmarks
- Strong results
Cons:
- No discussion of limitations
- Minor questions regarding quantization and size limits
Disclaimer: reviewer is generally knowledgeable but not familiar with the subject area.


MARGINAL POLICY GRADIENTS: A UNIFIED FAMILY OF ESTIMATORS FOR BOUNDED ACTION SPACES WITH APPLICATIONS
Many complex domains, such as robotics control and real-time strategy (RTS) games, require an agent to learn a continuous control. In the former, an agent learns a policy over R d and in the latter, over a discrete set of actions each of which is parametrized by a continuous parameter. Such problems are naturally solved using policy based reinforcement learning (RL) methods, but unfortunately these often suffer from high variance leading to instability and slow convergence. Unnecessary variance is introduced whenever policies over bounded action spaces are modeled using distributions with unbounded support by applying a transformation T to the sampled action before execution in the environment. Recently, the variance reduced clipped action policy gradient (CAPG) was introduced for actions in bounded intervals, but to date no variance reduced methods exist when the action is a direction, something often seen in RTS games. To this end we introduce the angular policy gradient (APG), a stochastic policy gradient method for directional control. With the marginal policy gradients family of estimators we present a unified analysis of the variance reduction properties of APG and CAPG; our results provide a stronger guarantee than existing analyses for CAPG. Experimental results on a popular RTS game and a navigation task show that the APG estimator offers a substantial improvement over the standard policy gradient.

Summary
This paper derives a new policy gradient method for when continuous actions are transformed by a
normalization step, a process called angular policy gradients (APG). A generalization based on
a certain class of transformations is presented. The method is an instance of a
Rao-Blackwellization process and hence reduces variance.
Detailed comments
I enjoyed the concept and, while relatively niche, appreciated the work done here and do believe it has clear applications. I am not convinced that the measure theoretic perspective is always
necessary to convey the insights, although I appreciate the desire for technical correctness. Still,
appealing to measure theory does reduces readership, and I encourage the authors to keep this in
mind as they revise the text.
Generally speaking it seems like a lot of technicalities for a relatively simple result:
marginalizing a distribution onto a lower-dimensional surface.
The paper positions itself generally as dealing with arbitrary transformations T, but really is
about angular transformations (e.g. Definition 3.1). The generalization is relatively
straightforward and was not too surprising given the APG theory. The paper would gain in clarity
if its scope was narrowed.
It's hard for me to judge of the experimental results of section 5.3, given that there are no other
benchmarks or provided reference paper. As a whole, I see APG as providing a minor benefit over PG.
Def 4.4: "a notion of Fisher information" -- maybe "variant" is better than "notion", which implies there are different kinds of Fisher information
Def 3.1 mu is overloaded: parameter or measure?
4.4, law of total variation -- define
Overall
This was a fun, albeit incremental paper. The method is unlikely to set new SOTA, but I appreciated
the appeal to measure theory to formalize some of the concepts.
Questions
What does E_{pi|s} refer to in Eqn 4.1?
Can you clarify what it means for the map T to be a sufficient statistic for theta? (Theorem 4.6)
Experiment 5.1: Why would we expect APG with a 2d Gaussian to perform better than a 1d Gaussian
on the angle?
Suggestions
Paragraph 2 of section 3 seems like the key to the whole paper -- I would make it more prominent.
I would include a short 'measure theory' appendix or equivalent reference for the lay reader.
I wonder if the paper's main aim is not actually to bring measure theory to the study of policy
gradients, which would be a laudable goal in and of itself. ICLR may not in this case be the right
venue (nor are the current results substantial enough to justify this) but I do encourage authors to
consider this avenue, e.g. in a journal paper.
= Revised after rebuttal =
I thank the authors for their response. I think this work deserves to be published, in particular because it presents a reasonably straightforward result that others will benefit from. However, I do encourage further work to
1) Provide stronger empirical results (these are not too convincing).
2) Beware of overstating: the argument that the framework is broadly applicable is not that useful, given that it's a lot of work to derive closed-form marginalized estimators.


Policy optimization struggles when the reward feedback signal is very sparse and essentially becomes a random search algorithm until the agent accidentally stumbles upon a rewarding or the goal state. Recent works utilize intrinsic motivation to guide the exploration via generative models, predictive forward models, or more ad-hoc measures of surprise. We propose EMI, which is an exploration method that constructs embedding representation of states and actions that does not rely on generative decoding of the full observation but extracts predictive signals that can be used to guide exploration based on forward prediction in the representation space. Our experiments show the state of the art performance on challenging locomotion task with continuous control and on image-based exploration tasks with discrete actions on Atari.

The paper proposes an approach for exploration via reward bonuses based on a form of surprise. The surprise factor is based on the next state of a particular transition, and the error in the embedding space to satisfy a linear dynamics formulation. The embedding space of the states and actions are optimized to increase the mutual-information in predicting next state, and current action - encouraging meaningful embeddings with more training, and hence gradual fading away of the extrinsic rewards.
The paper is mostly well-written, and the idea is interesting. The experimental results do show that the proposed reward augmentation leads to better performing policies, but the claims in the experimental section need to be less strong ("outperforms the baseline by a large margin" - Figure 4 - overlapping error bars; "state of the art" - Figure 5 - again, error bars, and no improvement in some domains.) But overall, I think the paper can be accepted as it is an interesting approach.
Below are some comments that I hope the authors address in their rebuttal, followed by some possible typos in the current draft.
- Theorem 1 content placement: The organization here is rather unclear. Currently, you present Theorem 1, and then talk about using "JSD instead of MI". Maybe this is a last minute mistake. In either case, it is strongly suggested that the section be reworked to be clearer.
- JSD is upper bounded by ln(2); your bounds (7) and (8) would change consequently too.
- Training regime employed:
  3 epochs-512 minibatch -> assuming distinct minibatches are sampled, (512x3) samples used
  Collected (5k*500; sparsehalfcheetah)/(50*500; swimmercatcher)/(100k*4500; atari)
  Is this the sample usage for training? Axis labels for all plots are missing -- specifically scale of x-axis.
  Commenting on the sample complexity -- especially as the embedding network seems easy-to-train (or insufficiently trained), would be good; optimizing a lower-bound insufficiently leads one to doubt if the bound is meaningful at all. Is the huge batch of samples mostly used in TRPO/RL part of the infrastructure?
- Discussing extrinsic rewards: the pros. vs. cons of the two reward formulations, why both are used etc. would be useful.
- Embedding dimension: d=8 in Gravitar and Solaris, but performance is less significant (no significance) in these domains. Is this due to insufficient training?
- RL method: Including details about form of TRPO used in appendix would be good (vine/single-path). Further if entropy regularization is used, how does the exploration interplay work.
- A.2 is an interesting section. A linear dynamics model being effective in MuJoCo tasks seems plausible. But an Atari example is definitely more interesting. Therefore this section can be clearer - specifically distinction between residual error and sample error.
Typos:
- Appendix \lambda parameters unclear
- Appendix step-size information contradictory.
Post-response comment: while I do think the approach is interesting, the utility of it is mostly demonstrated currently through empirical experiments. These experiments are preliminary, but used to make strong arguments for the effectiveness of the proposed approach. Further, upon highlighting this in my review, the authors disagree and think it’s empirical validity is rather superior. This leaves me concerned, and after thinking about it further, I do not think this is sufficient for acceptance. Therefore, I’m reducing my score to a 5.
PS: the characterization of irreducible error as a product of the limitation of a linear model may be inaccurate.


Neural network quantization has an inherent problem called accumulated quantization error, which is the key obstacle towards ultra-low precision, e.g., 2-or 3-bit precision. To resolve this problem, we propose precision highway, which forms an end-to-end high-precision information flow while performing the ultralow-precision computation. First, we describe how the precision highway reduce the accumulated quantization error in both convolutional and recurrent neural networks. We also provide the quantitative analysis of the benefit of precision highway and evaluate the overhead on the state-of-the-art hardware accelerator. In the experiments, our proposed method outperforms the best existing quantization methods while offering 3-bit weight/activation quantization with no accuracy loss and 2-bit quantization with a 2.45 % top-1 accuracy loss in ResNet-50. We also report that the proposed method significantly outperforms the existing method in the 2-bit quantization of an LSTM for language modeling.

This paper studies methods to improve the performance of quantized neural networks.  The paper is largely centered around the idea of "precision highways" (full-precision residual connections) that run in parallel to fully-quantized convolutions.  However, the paper also throws in a toolbox of other methods like distillation from a teacher network, a quantization method based on the Laplace distribution, and a fine tuning scheme.
The paper reports performance for the resulting networks that is impressive but still believable.   They also do very extensive experiments, including an ablation study in Table 1 that I really liked, and a study of how the precision of the skip connections impacts overall performance.   I also like the visualizations of how quantization impacts the loss surface.
My main concern about this paper is that is has conceptual overlap with other approaches.  The authors are not the first to quantize resnets, and other papers have looked at teacher training and distillation as a method of refinement.  The authors are fairly upfront about this though, and I think this paper is the first to do a really thorough investigation of the impacts of skip connections in their own right.    Realistically, fully binarizing neural nets without modification is unlikely to lead to good performance.  The idea of leaving the skip connections with higher precision is a good compromise that achieves hardware friendliness along with strong performance, so I think it's worth having a paper like this that takes a closer look at this approach.
A few questions I had:
1)  I can't tell exactly what methods are being used in Table 1.  When the "highway" box is unchecked, does this mean the skip connection is absent?  Or that it exists but with full precision?  Or maybe that the skip connection branched after the quantization instead of before?   Also, what fine-tuning methods is used when the "teacher" box is un-checked?
2) You implemented your own version of Zhuang's method.  However, I'd like to know how your numbers compare to the original reported numbers in Zhuang's paper.
One other minor criticism - When you fine-tune a modified network, the activations and weights will change.  It could be that the networks is modifying its parameters to account for (i.e., cancel out) the quantization errors.  For this reason I don't interpret Figure 4 as evidence for accumulation of error.  Perhaps this type of behavior would exists if you fine-tuned two full-precision networks using different random seeds, or different teacher networks.


We identify a phenomenon, which we refer to as multi-model forgetting, that occurs when sequentially training multiple deep networks with partially-shared parameters; the performance of previously-trained models degrades as one optimizes a subsequent one, due to the overwriting of shared parameters. To overcome this, we introduce a statistically-justified weight plasticity loss that regularizes the learning of a model's shared parameters according to their importance for the previous models, and demonstrate its effectiveness when training two models sequentially and for neural architecture search. Adding weight plasticity in neural architecture search preserves the best models to the end of the search and yields improved results in both natural language processing and computer vision tasks.

This paper discusses the phenomena of “neural brainwashing”, which refers to that the performance of one model is affected via another model sharing model parameters. To solve the issue, the authors derived a new loss out from maximizing the posterior of the parameters. With the new loss, the neural brainwashing is largely diminished.
The derived new loss looks meaningful to me and I think this is a valuable work for handling the weights coadaptation between two neural models, which with no doubt will bring great interests within the community of neural architecture search.
Here are some comments on the aspects that this paper can be improved:
1)	A very important related work [1] is missed in this paper. [1] discussed the properties of “one-shot model”, which means that several different architectures are unified into the same model by sharing model weights. Furthermore, [1] discussed “neural brainwashing” (although not with the same name) and how to handle it in a very simple way (by randomly dropping path). This definitely should be a baseline to compare with. In addition, a very recent work [2] also leverages model sharing to conduct neural architecture search.
2)	Although I understand that to improve accuracy of NAS is not the main goal of this paper, the baseline number to be improved over is too weak. For example, 4.87 of CIFAR10 in ENAS. Per my own hands on experience, it does not need too many hyperparameter tuning of ENAS to obtain < 4% error rate. Please provide more convincing baseline numbers and supporting evidences of the better performance of WPL in NAS.
[1] Bender, Gabriel, et al. "Understanding and simplifying one-shot architecture search." International Conference on Machine Learning. 2018.
[2] Luo, Renqian, et al. "Neural architecture optimization." NIPS (2018).


Effective performance of neural networks depends critically on effective tuning of optimization hyperparameters, especially learning rates (and schedules thereof). We present Amortized Proximal Optimization (APO), which takes the perspective that each optimization step should approximately minimize a proximal objective (similar to the ones used to motivate natural gradient and trust region policy optimization). Optimization hyperparameters are adapted to best minimize the proximal objective after one weight update. We show that an idealized version of APO (where an oracle minimizes the proximal objective exactly) achieves global convergence to stationary point and locally second-order convergence to global optimum for neural networks. APO incurs minimal computational overhead. We experiment with using APO to adapt a variety of optimization hyperparameters online during training, including (possibly layer-specific) learning rates, damping coefficients, and gradient variance exponents. For a variety of network architectures and optimization algorithms (including SGD, RMSprop, and K-FAC), we show that with minimal tuning, APO performs competitively with carefully tuned optimizers.

I raised my rating. After the rebuttal.
- the authors address most of my concerns.
- it's better to show time v.s. testing accuracy as well. the per-epoch time for each method is different.
- anyway, the theory part acts still more like a decoration. as the author mentioned, the assumption is not realistic.
-------------------------------------------------------------
This paper presents a method to update hyper-parameters (e.g. learning rate) before updating of model parameters. The idea is simple but intuitive. I am conservative about my rating now, I will consider raising it after the rebuttal.
1. The focus of this paper is the hyper-parameter, please focus and explain more on the usage with hyper-parameters.
- no need to write so much in section 2.1, the surrogate is simple and common in optimization for parameters. After all, newton method and natural gradients method are not used in experiments.
- in section 2.2, please explain more how gradients w.r.t hyper-parameters are computed.
2. No need to write so much decorated bounds in section 3. The convergence analysis is on Z, not on parameters x and hyper-parameters theta. So, bounds here can not be used to explain empirical observations in Section 5.
3. Could authors explain the time complexity of inner loop in Algorithm 1? Does it take more time than that of updating model parameters?
4. Authors have done a good comparison in the context of deep nets.  However,
- could the authors compare with changing step-size? In most of experiments, the baseline methods, i.e. RMSProp are used with fixed rates. Is it better to decay learning rates for toy data sets? It is known that SGD with fixed step-size can not find the optimal for convex (perhaps, also simple) problems.
- how to tune lambda? it is an important hyper-parameter, but it is set without a good principle, e.g., "For SGD-APO, we used lambda = 0.001, while for SGDm-APO, we used lambda = 0.01", "while for RMSprop-APO, the best lambda was 0.0001". What are reasons for these?
- In Section 5.2, it is said lambda is tuned by grid-search. Tuning a good lambda v.s. tuning a good step-size, which one costs more?


EXCESSIVE INVARIANCE CAUSES ADVERSARIAL VULNERABILITY
Despite their impressive performance, deep neural networks exhibit striking failures on out-of-distribution inputs. One core idea of adversarial example research is to reveal neural network errors under such distribution shifts. We decompose these errors into two complementary sources: sensitivity and invariance. We show deep networks are not only too sensitive to task-irrelevant changes of their input, as is well-known from -adversarial examples, but are also too invariant to a wide range of task-relevant changes, thus making vast regions in input space vulnerable to adversarial attacks. We show such excessive invariance occurs across various tasks and architecture types. On MNIST and ImageNet one can manipulate the class-specific content of almost any image without changing the hidden activations. We identify an insufficiency of the standard cross-entropy loss as a reason for these failures. Further, we extend this objective based on an informationtheoretic analysis so it encourages the model to consider all task-dependent features in its decision. This provides the first approach tailored explicitly to overcome excessive invariance and resulting vulnerabilities.

This paper studies a new perspective on why adversarial examples exist in machine learning -- instead of seeing adversarial examples as the result of a classifier being sensitive to changes in irrelevant information (aka nuisance), the authors see them as the result of a classifier being invariant to changes in relevant (aka semantic) information. They show how to efficiently find such adversarial examples in bijective networks. Moreover, they propose to modify the training objective so that the bijective networks could be more robust to such attacks.
Pros:
 -- clarity is good (except for a few places, e.g. no definition of F(x)_i in Definition 1; Page 6 "three ways forward" item 3: I(y;z_n|z_s) = I(y;z_s) should be I(y;z_n|z_s) = I(y;z_n).)
 -- the idea is original to the best of my knowledge
 -- the mathematical motivation is sound
 -- Figure 6 seems to show that the proposed defense works on MNIST (However, would you provide more details on how you interpolated z_n? Moreover, what do the images generated with z_s from one input and z_n from another input look like (in your method)?)
Cons:
 -- scope: as all the presented problems and solutions assume bijective mapping, I wonder how is it relevant to the traditional perspective of adversarial attack and defense? It seems to me that the contribution of this paper is identifying a problem of bijective networks and then proposing a solution, thus its significance is restricted.
 -- method: while the mathematical motivation is sound, I'm not sure if the proposed training objective can achieve that goal. To elaborate, I see problems with both terms added in the proposed loss function:
 (a.) for the objective of maximizing the cross entropy of the nuisance classifier, it is possible that I(y;z_n) is not reduced, but rather the information about y is encoded in a way that the nuisance classifier is not able to decode, similar to what happens in a one-way function (for example, see https://en.wikipedia.org/wiki/Cryptographic_hash_function ). In the MNIST experiments, the nuisance classifier is a three-layer MLP, which may be too weak and susceptible to information concealing.
 (b.) for the objective of maximizing the likelihood of a factorized model of p(z_s, z_n), I don't see how optimizing it would reduce I(z_s; z_n). In general, even if z_s and z_n are strongly correlated, one can still fit such a factorized model. This only ensures that I(Z_s; Z_n) = 0 for Z_s, Z_n *sampled from the model*, but does not necessarily reduce I(z_s; z_n) for z_s, z_n *used to train the model*. The discrepancy between p(Z_s, Z_n) and p(z_s, z_n) could be huge, in which case one has the model misspecification problem which is another topic.
 (c.) a side question: why is the MLE objective using likelihood rather than log likelihood? Since the two cross entropy losses are similar to log likelihood, I feel there is a mismatch here.
----------------------------------------
AFTER REBUTTAL:
Thanks for your reply to my comments. The new revision has improved clarity and provided new supporting evidences. I would like to raise my rating to 6.
That being said, (as you agreed) the link from the conceptual goal to the proposed objective has mostly empirical support. Therefore I hope it may encourage future investigation on when and why the proposed objective is successful in achieving the conceptual goal.


Despite deep neural networks have demonstrated extraordinary power in various applications, their superior performances are at expense of high storage and computational costs. Consequently, the acceleration and compression of neural networks have attracted much attention recently. Knowledge Transfer (KT), which aims at training a smaller student network by transferring knowledge from a larger teacher model, is one of the popular solutions. In this paper, we propose a novel knowledge transfer method by treating it as a distribution matching problem. Particularly, we match the distributions of neuron selectivity patterns between teacher and student networks. To achieve this goal, we devise a new KT loss function by minimizing the Maximum Mean Discrepancy (MMD) metric between these distributions. Combined with the original loss function, our method can significantly improve the performance of student networks. We validate the effectiveness of our method across several datasets, and further combine it with other KT methods to explore the best possible results. Last but not least, we fine-tune the model to other tasks such as object detection. The results are also encouraging, which confirm the transferability of the learned features.

This paper targets knowledge distillation of a large network to a smaller network. The approach is summarized by equations (3) and (4), which in short proposes that one should use the maximum-mean-discrepancy (MMD) of the network activations as a loss term for distillation.
When considering CIFAR image classification tasks, it is shown that only when using a specific quadratic polynomial kernel (which as described in https://arxiv.org/pdf/1701.01036.pdf is tantamount of applying neural style transfer) the proposed approach is able to match the performance of the seminal paper of Hinton et al.  When embarking to imagenet, the proposed approach is only able to match the performance of standard knowledge distillation by adding the quadratic term (texture in neural style synthesis jargon). This is actually a sensible proposal. Yet, the claims about MMD as a way of explaining neural style transfer has appeared in the paper cited above, which the authors mention.
The idea of transferring from one domain to another using MMD as a regularizer appeared in https://arxiv.org/pdf/1605.06636.pdf by Long et al --- indeed equation (3) of this paper matches exactly equation (10) of Long et al. Note too that Long et al also discuss what kernels work well and which work poorly due to vanishing gradients and propose parametrised solutions. This is something this paper failed to do.
The two works cited above make me wonder about the novelty of the current paper.  In fact, this paper ends us being an application of the neural style transfer loss function to network distillation. As such this could be useful, if not already done by someone else previously.
I find that the paper is poorly written, with many typos, and lacks focus on a single concrete story. The CIFAR experiments fail to use KD+NST (ie the thing that works for imagenet - neural style transfer) and section 5.3 appears trivial in light of the cited works. For all these reasons, I am inclined to reject this paper.


LEARNING SELF-IMITATING DIVERSE POLICIES
The success of popular algorithms for deep reinforcement learning, such as policygradients and Q-learning, relies heavily on the availability of an informative reward signal at each timestep of the sequential decision-making process. When rewards are only sparsely available during an episode, or a rewarding feedback is provided only after episode termination, these algorithms perform sub-optimally due to the difficultly in credit assignment. Alternatively, trajectory-based policy optimization methods, such as cross-entropy method and evolution strategies, do not require per-timestep rewards, but have been found to suffer from high sample complexity by completing forgoing the temporal nature of the problem. Improving the efficiency of RL algorithms in real-world problems with sparse or episodic rewards is therefore a pressing need. In this work, we introduce a self-imitation learning algorithm that exploits and explores well in the sparse and episodic reward settings. We view each policy as a state-action visitation distribution and formulate policy optimization as a divergence minimization problem. We show that with Jensen-Shannon divergence, this divergence minimization problem can be reduced into a policy-gradient algorithm with shaped rewards learned from experience replays. Experimental results indicate that our algorithm works comparable to existing algorithms in environments with dense rewards, and significantly better in environments with sparse and episodic rewards. We then discuss limitations of self-imitation learning, and propose to solve them by using Stein variational policy gradient descent with the Jensen-Shannon kernel to learn multiple diverse policies. We demonstrate its effectiveness on a challenging variant of continuous-control MuJoCo locomotion tasks.

The paper describes a method to improve reinforcement learning for task with sparse rewards signals.
The basic idea is to select the best episodes from the system's experience, and learn to imitate them step by step as the system evolves, aiming at providing a less sparse learning signal.
The math works out to a gradient that is of similar form as a policy gradient, which makes it easy to interpolate both of them. The resulting training procedure is a policy gradient that gets additional reinforcement of the system's best runs.
The experiments show the validity especially for the most extreme case (episodic rewards), while, as expected, for the other extreme of dense rewards, the method's effect is not consistently positive.
The paper then critiques its own method and identifies a critical weakness: the reliance on good exploration. I like that a lot. The paper goes on to suggest an extension to address this by training an ensemble, and shows the effectiveness of this for a number of tasks. However, I feel that the description of this extension is less clear than that of the core idea, and introduces too many new ideas and concepts in a too condensed text.
The paper seems a significant in that it provides a notable improvement for sparse-rewards tasks, which are a common sub-class of real-world problems.
My background is not RL. While I am quite confident in my understanding of the paper's math, I am not 100% familiar with the typical benchmark sets. Hence, I cannot judge whether the results include good baselines, or whether the task selection is biased. I can also not judge the completeness of the related work, and how novel the work is. For these questions, I hope that the other reviewers can provide more information.
Pros:
 - intuitive idea for a common problem
 - solution elegantly has the form of a modified policy gradient
 - convincing experimental results
 - self-critique of core idea, and extension to address its main weakness
 - nicely written text, does not leave a lot of questions
Cons:
 - while the core idea is nicely motivated and described and good to follow, Section 2.3 feels very dense and too short.
Overall, I find the core idea quite intuitive and elegant. The paper's background, motivation, and core method are well-written and, with some effort, quite readable for someone who is not an RL expert. I found that several questions I had during reading were preempted promptly and addressed. However, the description of the secondary method (Section 2.3) is too dense.
To me, the paper solidly meets the threshold of publication. Since I have no good comparison to other papers, I rate it a "clear accept" (8).
Minor points:
I noticed a few superfluous "the", please double-check.
In Table 1, please use the same exponent for directly comparable numbers, e.g. instead of "1.8e5 4.4e4", say "18e4 4.4e4". Or best just print the full numbers without exponent, I think you have the space.
When reading Table 1, I could bnot immediately line up "PPO" and "Self-imitation" in the caption with the table columns. It took a while to infer that PPO refers to \nu=0, and SI to \nu=0.8. Can you add PPO and SI to the table headings?
You define p as "the masking probability", but it is not clear whether that is the probability for keeping a "1" in the mask,
or for masking out the value. I can only guess from the results. I suggest to rephrase as "the probability of retaining a reward". Also, how about using plain words in Table 1's heading, such as "Noisy rewards\nSuppressing 10% of rewards", so that one can understand the table without having to search for its description in the text?


State-of-the-art deep neural networks (DNNs) typically have tens of millions of parameters, which might not fit into the upper levels of the memory hierarchy, thus increasing the inference time and energy consumption significantly, and prohibiting their use on edge devices such as mobile phones. The compression of DNN models has therefore become an active area of research recently, with connection pruning emerging as one of the most successful strategies. A very natural approach is to prune connections of DNNs via 1 regularization, but recent empirical investigations have suggested that this does not work as well in the context of DNN compression. In this work, we revisit this simple strategy and analyze it rigorously, to show that: (a) any stationary point of an 1 -regularized layerwisepruning objective has its number of non-zero elements bounded by the number of penalized prediction logits, regardless of the strength of the regularization; (b) successful pruning highly relies on an accurate optimization solver, and there is a trade-off between compression speed and distortion of prediction accuracy, controlled by the strength of regularization. Our theoretical results thus suggest that 1 pruning could be successful provided we use an accurate optimization solver. We corroborate this in our experiments, where we show that simple 1 regularization with an Adamax-L1(cumulative) solver gives pruning ratio competitive to the state-of-the-art.

This paper discusses the effect of L1 penalization for deep neural network. In particular it shows the stationary point of an l1 regularized layer has bounded non-zero elements.
The perspective of the proof is interesting: By chain rule, the stationary point satisfies nnz(W^j) linear equations, but the subgradients of the loss function w.r.t. the logits have at most N\times ks variables. If the coefficients of the linear equation are distributed in general positions, then the number of variables should not be larger than the number of equations.
While I mostly like the paper, I would like to point out some possible issues:
main concerns:
1. the columns of V may not be independent during the optimization(training) process. In this situation, I am not quite sure if the assumption of “general position” still holds. I understand that in literatures of Lasso and sparse coding it is common to assume “general position”. But in those problems the coefficient matrix is not Jacobian from a learning procedure.
2. the claim is a little bit counter intuitive: Theorem 1 claims the sparse inequality holds for any \lambda. It is against the empirical observation that when lambda is extremely small, effect of the regularizer tends to be almost zero. Can authors also show this effects empirically, i.e., when the regularization coefficients decrease, the nnz does not vary much? (Maybe there is some optimization details or approximations I missed?)
Some minor notation issues:
1. in theorem 1: dim(W^{(j)})=d should be dim(vec(W^{(j)}))=d
2. in theorem 1: Even though I understand what you are trying to say, I would suggest we describe the jacobian matrix V in details. Especially it is confusing to stack vec(X^J) (vec(W^j)) in the description.
3. the notations of subgradient and gradient are used without claim


GLOBAL-TO-LOCAL MEMORY POINTER NETWORKS FOR TASK-ORIENTED DIALOGUE
End-to-end task-oriented dialogue is challenging since knowledge bases are usually large, dynamic and hard to incorporate into a learning framework. We propose the global-to-local memory pointer (GLMP) networks to address this issue. In our model, a global memory encoder and a local memory decoder are proposed to share external knowledge. The encoder encodes dialogue history, modifies global contextual representation, and generates a global memory pointer. The decoder first generates a sketch response with unfilled slots. Next, it passes the global memory pointer to filter the external knowledge for relevant information, then instantiates the slots via the local memory pointers. We empirically show that our model can improve copy accuracy and mitigate the common out-of-vocabulary problem. As a result, GLMP is able to improve over the previous state-of-theart models in both simulated bAbI Dialogue dataset and human-human Stanford Multi-domain Dialogue dataset on automatic and human evaluation.

The paper presents a new model for reading and writing memory in the context of task-oriented dialogue. The model contains three main components: an encoder, a decoder, and an external KB. The external KB is in the format of an SVO triple store. The encoder encodes the dialogue history and, in doing so, writes its hidden states to memory and generates a "global memory pointer" as its last hidden state. The decoder takes as input the global memory pointer, the encoded dialogue state history, and the external KB and then generates a response using a two-step process in which it 1) generates a template response using tags to designate slots that need filling and 2) looks up the correct filler for each slot using the template+global memory pointer as a query. The authors evaluate the model on a simulated dialogue dataset (bAbI) and on a human-human dataset (Stanford Multi-domain Dialogue or SMD) as well as in a human eval. They show substantial improvements over existing models on SMD (the more interesting of the datasets) in terms of entity F1--i.e. the number of correctly-generated entities in the response. They also show improvement on bAbI specifically on cases involving OOVs. On the human evaluation, they show improvements in terms of both "appropriateness" and "human-likeness".
Overall, I think this is a nice and well-motivated model. I very much appreciate the thoroughness of the evaluation (two different datasets, plus a human evaluation). The level of analysis of the model was also good, although there (inevitably) could have been more. Since it is such a complex model, I would have liked to see more thorough ablations or at least better descriptions of the baselines, in order to better understand which specific pieces of the model yield which types of gains. A few particular questions below:
- You describe the auxiliary loss on the global pointer, and mention an ablation study that show that this improves performance. Maybe I am overlooking something, but I cannot find this ablation in the paper or appendix. It would be nice to see how large the effect is.
- Following on the above, why no similar auxiliary losses on additional components, e.g. the template generation? Were these tried and deemed unnecessary or vice-versa (i.e. the default was no auxiliary loss and they were only added when needed)? Either way, it would be nice to better communicate the experiments/intuitions that motivated the particular architecture you arrived at.
- I really appreciate that you run a human eval. But why not have humans evaluate objective "correctness" as well? It seems trivial to ask people to say whether or not the answer is correct/communicates the same information as the gold.


We study the implicit bias of gradient descent methods in solving a binary classification problem over a linearly separable dataset. The classifier is described by a nonlinear ReLU model and the objective function adopts the exponential loss function. We first characterize the landscape of the loss function and show that there can exist spurious asymptotic local minima besides asymptotic global minima. We then show that gradient descent (GD) can converge to either a global or a local max-margin direction, or may diverge from the desired max-margin direction in a general context. For stochastic gradient descent (SGD), we show that it converges in expectation to either the global or the local max-margin direction if SGD converges. We further explore the implicit bias of these algorithms in learning a multi-neuron network under certain stationary conditions, and show that the learned classifier maximizes the margins of each sample pattern partition under the ReLU activation.

This paper considers the binary classification problem with exponential loss and ReLu activation function (single neuron). The authors characterize the asymptotic loss landscape by three different types of critical points. They prove that gradient descent (GD) will result in four different regions and provide convergence rates for GD to converge to an asymptotic global minimum, asymptotic local minimum and local minimum under certain assumptions. The authors also provide convergence results for stochastic gradient descent (SGD) and provide extensions to leaky ReLu activation and multi-neuron networks. The paper is well written and the results are mostly clearly presented. This paper mostly follows the line of research by Soudry et al. (2017, 2018), while it has its own merit due to the ReLu activation function considered. However, there are many strong assumptions that are not carefully verified and I really have concerns about the contribution of this paper since they simplify their analysis and results merely by imposing stringent conditions. In particular, I have the following major comments about the paper:
1.	In the definition of max-margin direction, why you use \argmin_{w} max_{i} (w^{\top}x_i)? It seems to me that the definition should be \argmax_{w} min_{i} (w^{\top}x_i). This definition keeps appearing in multiple places in the main paper.
2.	In the proof of Theorem 3.2, I am confused by the argument of the case that \hat w^{+} is not in the linearly separable region. More clarification is needed to make the proof rigorous.
3.	In the analysis of Theorem 3.3 and 3.4, the authors make a very stringent assumption that the iterate w_t staying in linear separable region for all t>\mathcal{T}. This assumption seems too strong, which should be verified rather than imposed in analysis of SGD. Note that even the example shown in Proposition 2 is still very restrictive (you require all the positive examples or negative examples are very close to one another).
4.	Furthermore, in the analysis of SGD, the authors did not specify the assumption that \hat w^{+} lies in the linear separable region, which is also required in this theorem and also very strong. Given such strong assumptions, the analytic results seem to be trivial and it is hard to evaluate the authors’ contribution.
5.	For the convergence results of SGD, the current rate is derived on the distance between \|E[w_t] - \hat{w}\|^2. Can you provide similar results for mean square error (E\| w_t - \hat{w} \|^2)?
6.	In multi-neuron case, the authors again make very strong assumptions that all the neurons have unchanging activation status. This is not easily achievable without careful characterization or other rigorous assumptions. Under such strong assumptions, the extension to multi-neuron again seems not very meaningful.
Other minor comments:
1.	The references are not correctly cited. For instance, please correct the use of parenthesis in “… which is different from that in (Soudry et al., 2017, Corollary 8)” and “… hold for various other types of gradient-based algorithms Gunasekar et al. (2018)”.
2.	The sentence “…, which the nature of convergence is different from …” does not read well. Should it be “where” or “of which”?


Transfer in Reinforcement Learning (RL) refers to the idea of applying knowledge gained from previous tasks to solving related tasks. Learning a universal value function (Schaul et al., 2015), which generalizes over goals and states, has previously been shown to be useful for transfer. However, successor features are believed to be more suitable than values for transfer (Dayan, 1993;Barreto et al., 2017), even though they cannot directly generalize to new goals. In this paper, we propose (1) Universal Successor Features (USFs) to capture the underlying dynamics of the environment while allowing generalization to unseen goals and (2) a flexible end-to-end model of USFs that can be trained by interacting with the environment. We show that learning USFs is compatible with any RL algorithm that learns state values using a temporal difference method. Our experiments in a simple gridworld and with two MuJoCo environments show that USFs can greatly accelerate training when learning multiple tasks and can effectively transfer knowledge to new tasks.

Summary: This paper proposes a generalisation of the SFs framework to a goal conditioning representation that could, in principle, generalise over a collection of goals at test time. This is akin to universal value functions [1] (and more generally GVFs). Although I like the idea  and it seems a very interesting direction for generalisation to new goals, I do think the execution, the particular instantiation and (lack of) in-depth evaluation with (at least some of the) existing methods in literature -- including UVFAs [1] and the different ways SFs have been used for generalisation [2,3,4] -- is unfortunately letting it down.
Clarity: Reasonably well-written, easy to follow. A couple of things in the experimental section can be improved:
- It’s not totally clear to me what the their baseline Multi-goal DQN is. Does it have the same architecture as Figure 1, but just using (2).
- In the plots, the only difference between DQN and DQN+USF is that the second as the additional loss L_{\psi} ? Or is there any other difference?
Originality and Significance:
I’m a bit split here: I like in principle the idea, but I think this instantiation is (fairly) incremental with respect to the current literature. Even the claimed contributions are a bit thin. Suitability of SFs to any TD-based learning, comes from SRs/SFs satisfying a Bellman eq. which was point out, explored and paired with control algorithms before [2,4]. Also, the particular way of learning the features \phi, without going through the rewards, was already proposed and explored in [3]. That might be a missing reference.
The experiments seems to show slight improvements with respect to a baseline (Multi-DQN). It is not clear to me exactly what this is or if it would dominated even something vanilla UVFAs. I think this is a missing and somewhat mandatory comparison. I know the authors noted that is was because ‘UVFAs are prone to instabilities and may require further prior knowledge’, but I think that refers only to the two-stage (factorisation) procedure proposed in the original paper, not the common adoption in the literature. At the end of the day, the proposed architecture in Fig. 1 is a kind of UVFA, just with a bit more structure, so it would be surprising to me if UVFAs would actually fail in these environments. But if that’s the case, that’s a very interesting data point that the additional structure actually helps considerably beyond the incremental advantage exemplified here.
Other comments/questions:
1) Clarification on the training procedure. The value function $Q(s,a,g)$  are training via eq. (3) with the i) actual reward (coming from the environment) or ii) the ‘fictitious’ reward coming from r(s,a,s’|g) = \phi(s,a,s’)^T w(g)? Note that these are very different and only one ensures compatibility between the rewards and the value functions in learning.
The SFs will give you the value function for the reward r(s,a,s’|g) = \phi(s,a,s’)^T w(g) and if this is not align with the real reward, the corresponding value function obtained via SFs will not be the value function optimising the real reward. As far as I can see there’s not criteria that forces this to be the case.
2) Comparison with SF transfer literature. Although discussed in the related work section, there is no quantitative comparison to the way SFs were shown to transfer knowledge[2,4], via evaluation and (generalised) policy improvement. Because these ways of generalisation are very different, it’s not clear go they would stack against each other, or in which scenarios one would be more appropriate than the other.
To give a more concrete example: The training procedure in 3.1 makes sure that there’s fairly good coverage of the whole state-space by sampling goals conditioned on the room. Now if one would train SFs on these train tasks only (even independently), we would have policies that would know how to go to any of the rooms. And for the test tasks we would have the evaluation of these policies to the collection of goals. Which means that applying the methodology of transfer in [2,4] we would zero-shot get policies that reach any of the states encountered in the path of the 12 goals used in the train phase. And even if the test goals are not part of this collection, it stands to reason that a policy that can already go to the goal’s room and be easily adaptable to reaching the test goal -- aka the evaluation the policy that already reached that room is a good starting point for the improvement step [4].
Note: I am willing to reconsider when/if the above have been reconciled/resolved.
References:
[1] Schaul, T., Horgan, D., Gregor, K. and Silver, D., 2015, June. Universal value function approximators. In International Conference on Machine Learning (pp. 1312-1320).
[2] Andre Barreto, Will Dabney, Remi Munos, Jonathan J Hunt, Tom Schaul, Hado P van Hasselt, and ´ David Silver. Successor features for transfer in reinforcement learning. In Advances in Neural Information Processing Systems, pp. 4055–4065, 2017.
[3] Machado, M.C., Rosenbaum, C., Guo, X., Liu, M., Tesauro, G. and Campbell, M., 2018. Eigenoption Discovery through the Deep Successor Representation, International Conference on Learning Representations, 2018.
[4] Barreto, A., Borsa, D., Quan, J., Schaul, T., Silver, D., Hessel, M., Mankowitz, D., Zidek, A. and Munos, R., 2018, July. Transfer in deep reinforcement learning using successor features and generalised policy improvement. In International Conference on Machine Learning (pp. 510-519).


Processing sequential data with long term dependencies and learn complex transitions are two major challenges in many deep learning applications. In this paper, we introduce a novel architecture, the Auto-addressing and Recurrent Memory Integrating Network (ARMIN) to address these issues. The ARMIN explicitly stores previous hidden states and recurrently integrate useful past states into current time-step by an efficient memory addressing mechanism. Compared to existing memory networks, the ARMIN is more light-weight and inference-time efficient. Our network can be trained on small slices of long sequential data, and thus, can boost its training speed. Experiments on various tasks demonstrate the efficiency of the ARMIN architecture. Codes and models will be available.

This paper introduces a memory-augmented RNN (MARNN) which aims at being lightweight and   differentiable. In a nutshell, authors propose to augment a LSTM-type architecture with several memory cells. At each time-step, MARNN retrieves one memory cell, updates his state, and updates the memory cell content. To learn the retrieval operation that requires discrete addressing,  authors rely on the Gumbel-Softmax. Authors evaluate their approach on PennTreeBank character level modelling where they demonstrate competitive performances. They also report state-of-art performance on the Thumos dataset. The paper is overall clear and pleasant to read.
Authors highlight that MARNN is more lightweight compared to existing memory networks. MARNN can indeed retrieves only memory cells at inference. However,  since MARNN uses a Gumbel-Softmax to train the discrete addressing scheme, it is it not clear if there is any advantage in term of memory and computation of MARNN relatively to other network during training? It would be nice to compare the computation time/memory usage of MARNN with other memory augmented network such as TARDIS, NTM or Memory Network during training and inference.
Another claim is that MARNN can possibly boost training speed by reducing the lengths of TBTT.  But MARNN also haves a training time overhead as showed in Figure 2.  How does the overall training time/performances of MARNN with TBPTT of 50 compared to a LSTM with TBPTT of 100/150?
The writing can be sometime a bit imprecise. For instance authors say that MARNN “learns better representations that many hierarchical RNN structure”. I agree that MARNN outperforms in term of accuracy, however, it is not clear what the author are referring to by “better representation” of the MARNN hidden state? Performance gain of MARNN could also be due to the external memory which allows  to retain more information of the input? In addition, it would be nice to precise which type of hierarchical RNN structure MARNN does (or doesn’t) outperform. Another claim is that MARNN can “easily learn long-term dependencies”. While this is reasonable, I am unsure that the empirical evaluation support this.  It would be nice to show how the gradients backpropagated through time behave in practice to support this claim?
Memory-augmented network are a very important research directions and the MARNN architecture is interesting. However, it is not entirely clear to me what is the main advantage of MARNN relatively to other memory networks network such as TARDIS, NTM or Memory Network for training and/or inference. Although authors do compare with TARDIS, further comparison with the other networks and in term of computation time and memory could help clarify those points.


We consider an environment with multiple reward functions. One of them represents goal achievement and the others represent instantaneous safety conditions. We consider a scenario where the safety rewards should always be above some thresholds. The thresholds are parameters with values that differ between users. We efficiently compute a family of policies that cover all threshold-based constraints and maximize the goal achievement reward. We introduce a new parameterized threshold-based scalarization method of the reward vector that encodes our objective. We present novel data structures to store the value functions of the Bellman equation that allow their efficient computation using the value iteration algorithm. We present results for both discrete and continuous state spaces.

I generally like the paper. The paper discussed a constrained value iteration setting where the safety contraints must be greater some threshold, and thresholds \delta are parameters. The paper attempts to develop an value iteration algorithm to compute a class of optimal polices with such a parameter. The algorithm is mainly based on a special design of representation/data structure of PWC function, which can be used to store value functions and allows to efficiently compute several relevant operations in bellman equation. A graph-based data structure is developed for continuous state domains and hence value iteration can be extended to such cases.
In general, the paper presents an interesting direction which can potentially help solve RL problems with the proposed constraint setting. However, the paper spends lots of effort explaining representations, but only a few sentences explaining about how the proposed representations/data structures can help find a somehow generic value iteration solution, which allows to efficiently compute/retrieve a particular solution once a \delta vector is specified. The paper should show in detail (or at least give some intuitive explanations) that using the proposed method can be more efficient than solving a value iteration for each individual constraint given that the constraints are independent. Specifically, the author uses the patient case to motivate the paper, saying that different patients may have different preferred thresholds and it is good to find class of policies so that any one of those policies can be retrieved once a threshold is specified. However, in this case, when dealing with only one patient, the dimension of reward is reduced to 1 (d = 1), while the computation of the algorithm is exponential in d, plus that the retrieval time is not intuitive to be better, so it is unsure whether computing such a class of policies worth.
In terms of novelty, the scalarization method of the vector-valued reward seems intuitive, since breaking a constraint means a infeasible solution. Furthermore, it is also unclear why the representation of PWC in discrete case is novel. A partial order on a high-dimensional space is naturally to be based on dominance relation, as a result, it seems natural to store value function by using right-top coordinates of a (hyper)rectangle.
As for the clarity, though the author made the effort to explain clearly by using examples after almost every definition/notation, some important explanations are missing. I would think the really interesting things are the operations based on those representations. For example, the part of computing summation of two PWC function representation is not justified. Why the summation can be calculated in that way? Though the maximum operation is intuitive, however, the summation should have some justification. I think a better way to explain those things is to redefine a new bellman operator, which can operate on those defined representations of PWC function.
I think it could be a nice work if the author can improve the motivation and presentation. Experiments on some simple domains can be also helpful.