In this paper we propose a novel approach to model compression termed Architecture Compression. Instead of operating on the weight or filter space of the network like classical model compression methods, our approach operates on the architecture space. A 1-D CNN encoder/decoder is trained to learn a mapping from discrete architecture space to a continuous embedding and back. Additionally, this embedding is jointly trained to regress accuracy and parameter count in order to incorporate information about the architecture's effectiveness on the dataset. During the compression phase, we first encode the network and then perform gradient descent in continuous space to optimize a compression objective function that maximizes accuracy and minimizes parameter count. The final continuous feature is then mapped to a discrete architecture using the decoder. We demonstrate the merits of this approach on visual recognition tasks such as CIFAR-10/100, FMNIST and SVHN and achieve a greater than 20x compression on CIFAR-10.

This paper deals with Architecture Compression, where the authors seem to learn a mapping from a discrete architecture space which includes various 1D convnets. The aim is to learn a continuous latent space, and an encoder and decoder to map both directions between the two architecture spaces. Two further regressors are trained to map from the continuous latent space to accuracy, and parameter count. By jointly training all these networks, the authors are now able to compress a given network by mapping it's discrete architecture into the latent space, then performing gradient descent towards higher accuracy and lower parameter count (according to the learned regressors).
The authors perform experiments on 4 standard datasets, and show that they can in some cases get a 20x reduction in parameters with negligible performance decrease. They show better Cifar10 results than a few baselines - I am not aware whether this is SOTA for that parameter budget, and the authors do not specify.
Overall I really like the idea in this paper, the latent space is well justified, but I cannot recommend acceptance of the current manuscript. There are many notational issues which I go into below, but the key issue is experiments and reproducability.
The search space is not clearly defined. Current literature shows that the performance of these methods depends a lot on the search space. The manuscript does make clear that a T-layer CNN is represented as a 5XT tensor, with each column representing layer type, kernel size etc. However the connectivity is not defined at all, which implies that layers are simply sequentially stacked. This seems to preclude even basic architectural advancement like skip connections / ResNet - the authors even mention this in section 3.1, and point to experiments on resnets in section 4.4, but the words "skip" and "resnet" do not appear anywhere else in the paper. I presume from the emphasis on topological sort that this is possible, but I don't see how.
If this paper is simply dealing with linear chains of modules, then the mapping to a continuous representation, and accuracy regression etc would still be interesting in principle. However it does mean that essentially all the big architecture advancements post-VGG (ie inception, resnet, densenet...) are impossible to represent in this space. Most of the Architecture Search works cited do have a search space which allows the more recent advances.
I don't see a big reason why the method could not be extended - taking the 5D per-layer representation and adding a few more dimensions to denote connectivity would seem reasonable. If not, the authors should clearly mention the limitations of their search space.
In terms of experiments, Figure 3 is very hard to interpret. The axes labellings are nearly too small to read, but it's also unclear what loss this even is - I presume this is the 'train' loss of L_d + \lambda_1L_a + \lambda_2L_p, but it could also be the 'compress' loss. It also behaves very unusually - the lines all end up lower than where they started, but oscillate around a lot, making me wonder if the curves from a second set of runs would look anything alike. It's not obvious why there's not just a 'normal' monotonic decrease.
A key point that is not really addressed is how well the continuous latent space actually captures what it should. I am extremely interested to know whether the result of 'compress', ie a new concrete architecture found by gradient descent in the latent space, actually has the number of parameters and the accuracy that the regressors predict. This could be added as columns in Table 1 - eg the concrete architecture for Cifar10 gets 20.33x compression and no change in accuracy, but does the regressor for the latents space predict this compression ratio / accuracy as well? If this is the case, then I feel that the latent space is clearly very informative, but it's not obvious here.
It would also be really useful to see some concrete input / output values in discrete architecture space. Presumably along the way to 20x compression of parameter count, the optimisation passes through a number of progressively smaller discrete architectures - what do these looks like? Is it progressively fewer layers / smaller filters / ??? Given that the discrete architecture encoding appears to have a fixed length of T, it's not even clear how layers would be removed. Figure 1 implies you would fill columns with zeros to delete layers, but I don't see this mentioned elsewhere in the text.
More minor points:
Equation numbers would be extremely useful throughout the paper.
Notation in section 3 is unclear. If theta represents trained parameters, then surely the accuracy on a given dataset would be a deterministic value. Assuming that the distribution P_{\theta}(a | A, D) is used to represent the non-determinism of SGD training, is \theta supposed to represent the initialised values of the weights?
There are 3 functions denoted by 'g' defined on page 3 and they all refer to completely different things - this is unnecessarily confusing.
The formula for expected accuracy - surely this should be averaging over N different training / evaluation runs, something like:
E_{\theta}[a | A, D] \simto \frac{1}{N} \sigma_{i}^N g_{\theta}(A, D, \theta_i)
The decoder computes a 6xT output instead of a 5xT output - what is this extra row for?
In the definition of "ground truth parameter count" p^* - presumably the standard deviation here is the standard deviation of the l vector? This formulation is a bit surprising, as convolutional layers will generally have few parameters, and final dense layers could have many. Did you consider alternative formulations like simply taking the log of the number of parameters? Having a huber loss with scale 1 for this part of the loss function was also surprising, it would be good to have some justification for this (ie, what range are the p^* values in for typical networks?)
In algorithm 1 line 4 - here you are subtracting \bar{p} from num_params before dividing by standard deviation, which does not appear in the formulation above.
In the experiments:
How were the 1500 random architectures generated? I presume by sampling uniformly a lot of 5xT tensors, but this encoding is not clearly defined. x_i is defined as being in the set of integers, does this include negative numbers? What are the upper / lower limits, and is there anything to push towards standard kernel sizes like 3x3, 5x5, etc? These random architectures were then trained five times for 5 epochs - what optimizer / hyperparameters / regularization was used? Similarly, the optimization algorithm used in the outer loop to learn the {en,de}coders/regressors is not specified.
I would move the lemma and theorem into the appendix - they seem quite unrelated to the overall thrust of the paper. To me, saying that an embedding is not uniquely defined, but can be learnt is not that controversial, and I don't need proofs that some architecture search space has a finite number of entries. Surely the fact that the architecture is represented as a 5xT tensor, and practically there are upper limits to kernel size, stride etc beyond which an increase has no effect, already implies a finite space? Either way, this section of the paper did not add much value from my perspective.
I want to close by encouraging the authors to resubmit after addressing the above issues, I do believe the underlying idea here is potentially very interesting.


A MAX-AFFINE SPLINE PERSPECTIVE OF RECURRENT NEURAL NETWORKS
We develop a framework for understanding and improving recurrent neural networks (RNNs) using max-affine spline operators (MASOs). We prove that RNNs using piecewise affine and convex nonlinearities can be written as a simple piecewise affine spline operator. The resulting representation provides several new perspectives for analyzing RNNs, three of which we study in this paper. First, we show that an RNN internally partitions the input space during training and that it builds up the partition through time. Second, we show that the affine slope parameter of an RNN corresponds to an input-specific template, from which we can interpret an RNN as performing a simple template matching (matched filtering) given the input. Third, by carefully examining the MASO RNN affine mapping, we prove that using a random initial hidden state corresponds to an explicit 2 regularization of the affine parameters, which can mollify exploding gradients and improve generalization. Extensive experiments on several datasets of various modalities demonstrate and validate each of the above conclusions. In particular, using a random initial hidden states elevates simple RNNs to near state-of-the-art performers on these datasets.

This paper builds upon recent work by Balestriero and Baraniuk (ICML 2018) that concern max-affine spline opertaor (MASO) interpretation of a substantial class of deep networks. In the new paper a special focus is put on Recurrent Neural Networks (RNNs), and it is highlighted based on theoretical considerations leveraging the MASO and numerical experiments that in the case of a piecewise affine and convex activation function, using noise in initial hidden state acts as regularization.
Overall I was impressed by the volume of contributions presented throughout the paper and also I very muched like the light shed on important classes of models that turn out to be not as black box as they could seem. My enthouasiasm was somehow tempered when discovering that the MASO modelling here was in fact a special case of Balestriero and Baraniuk (ICML 2018), but it seems that despite this the specific contribution is well motivated and justified, especially regarding application results. Yet, the other thing that has annoyed me and is causing me to only moderately champion the paper so far is that I found the notation heavy, not always well introduced nor explained, and while I believe that the authors have a clear understanding of things, it appears to me that the the opening sections 1 and 2 lack notation and/or conceptual clarity, making the paper hard to accept without additional care. To take a few examples:
a) In equation (3), the exponent (\ell) in A and B is not discussed. On a different level, the term "S" is used here but doesn't seem to be employed much in next instances of MASOs...why?
b) In equation (4), sure you can write a max as a sum with an approxiate indicator (modulo unicity I guess) but then what is called Q^{(\ell)} here becomes a function of A^{(\ell)}, B^{(\ell)}, z^{(\ell-1)}...?
c) In proposition 1, the notation A_sigma is not introduced. Of course, there is a notation table later but this would help (to preserve the flow and sometimes clarify things) to introduce notations upon first usage...
d) Still in prop 1, braket notation not so easy to grasp. What is A[z]z?
e) Still in prop 1, recall that sigma is assumed piecewise-linear and convex?
f) In th1, abusive to say that the layer "is" a mapping, isn't it?
g) In Theorem 2, what is f? A generic term for a deterministic function?
Also, below the Theorem, "affine" or "piecewise affine"?
h) I found section 4 somehow disconnected and flow-breaking. Put in appendix and use space to better explain the rest?
i) Section 5 is a strong and original bit, it seems. Should be put more to the fore in abstract/intro/conclusion?


Multi-agent reinforcement learning systems aim to provide interacting agents with the ability to collaboratively learn and adapt to the behaviour of other agents. In many real-world applications, the agents can only acquire a partial view of the world. Here we consider a setting whereby most agents' observations are also extremely noisy, hence only weakly correlated to the true state of the environment. Under these circumstances, learning an optimal policy becomes particularly challenging, even in the unrealistic case that an agent's policy can be made conditional upon all other agents' observations. To overcome these difficulties, we propose a multi-agent deep deterministic policy gradient algorithm enhanced by a communication medium (MADDPG-M), which implements a two-level, concurrent learning mechanism. An agent's policy depends on its own private observations as well as those explicitly shared by others through a communication medium. At any given point in time, an agent must decide whether its private observations are sufficiently informative to be shared with others. However, our environments provide no explicit feedback informing an agent whether a communication action is beneficial, rather the communication policies must also be learned through experience concurrently to the main policies. Our experimental results demonstrate that the algorithm performs well in six highly non-stationary environments of progressively higher complexity, and offers substantial performance gains compared to the baselines.

This paper is clearly written and explains everything in a good detail. I have a few questions about the design of the algorithm and experiments that I will explain next. Most importantly, I am confused why the communication actions are modeled with continuous actions. Also, the communicating agent idea is incorporated in MADDPG paper, and the contribution of the proposed network is unclear to me.  Right now, I am leaning toward weak reject now but I might update my evaluations after seeing the authors' feedback.
1) First, your construction of communication medium simply seems to be learning a method for graph sparsification and this deserves some explanation.  Also, I think that using the graph terminology for describing the communication medium structure will significantly improve the clarity of the paper. For example, I assume that by saying that m^t = ...=m^{t+C-1} you mean that you simply fix the communication graph structure for C steps, not the communicating observations. Based on your notations, it is a little bit confusing -- in your notations $m^t$ is the set of observation that flows through the graph which should be different than $m^{t+1}$.
2) Even MADDPG is very challenging to train! Now, this paper utilizes two MADDPG, and that is something that concerns me a lot. I don't think that replicating the results of this paper is possible by other people. How much was the cost of the hyper-parameters search?
3) Why the decision of where to send the observations is modeled with a continuous control action? It can be simply modeled with discrete action in a more efficient way, right? What I mean is that $c_i$ can be a binary which tells whether send an observation or not. Am I missing anything?
4) In section 2, you argue that in the original MADDPG paper, there is no inter-agent communication. As far as I remember, they have some experiments for cooperative communication or covert communication in which the communication is allowed between the agents. I would like to know more about this statement; maybe I am missing something. Why you are not designing the communication network which is a differentiable medium such as Foerster 2015? Isn't that efficient?
5) In alternating case, I don't see (intuitively) why the communication should help to improve upon MADDPG. My intuition is that each agent will be the gifted one 1/3 on average. This means that the agents cannot perceive who gives the correct information and the policy should converge to a point where the communication does not give any new information.
6) I would like to see what will happen with C=1? I think this hyper-parameter deserves some analysis to see how it affects the performance of the proposed method.
7) In section 5, you say that in original DDPG, there is no real need for inter-agent communication". This is a little bit strict statement, I guess. For example in the case with full observability, the agents can send messages which conveys their intention and help each other.
Minor:
* I would suggest using partially observability terminology instead of saying noisy observation because I think it includes a more general class of the problems to solve.
* "that a coupled through a communication medium" -> "that are coupled through a communication medium"
* In section 4.2, it is unclear to me what is the exploration strategy. Please explain more.
* section 5.2: Using the term lower bound is not accurate. Try changing it to something else or use with quotations: "lower bound"
* What will happen you choose the top-k rule for sending the information? For example, does top-2 (two-to-one) rule improve the results? The experiments might be added in future.
-----------------------------------
post rebuttal: the authors have responded to my main questions, and I would like to increase my score, but I cannot agree with them on possile future extensions of this work, e.g. in learning from pixels.


Recent results from linear algebra stating that any matrix can be decomposed into products of diagonal and circulant matrices has lead to the design of compact deep neural network architectures that perform well in practice. In this paper, we bridge the gap between these good empirical results and the theoretical approximation capabilities of Deep diagonalcirculant ReLU networks. More precisely, we first demonstrate that a Deep diagonalcirculant ReLU networks of bounded width and small depth can approximate a deep ReLU network in which the dense matrices are of low rank. Based on this result, we provide new bounds on the expressive power and universal approximativeness of this type of networks. We support our experimental results with thorough experiments on a large, real world video classification problem.

The experiments in the paper are similar to those explored in previous work! The main contribution claimed in the paper is the theoretical formulation for compact design of neural networks using circulant matrices instead of fully connected matrices.
I do not think the claim is sufficiently justified by the theoretical results provided.
Earlier result already shows how any matrix fully connected matrices can be approximated by 2n-1 circulant matrices. As the authors themselves point out, this theoretical result does not necessarily imply reduction in number of parameters since the for a depth l network, the equivalent diagonal-circulant-ReLU network will now require (2n-1)l depth, or 2n(2n-1)l parameters.
The main results (Proposition 3, 4) show that if the fully connected networks of depth l network are parameterized by (approximately) rank k matrices, then the resultant depth of diagonal-circulant network required to approximate the original network is (4k+1)l, which results in a total of 8n(4k+1)l parameters. Similar to the case of full rank fully connected networks (proposition 2), this result does not necessarily indicate a compression of number of parameters either. In particular, if fully connected networks are indeed rank k, then we only need nkl parameters parameters to represent the matrix, which is lower than the number of parameters required by the diagonal-circulant network.
So I do not see how the result can be seen as a justification for using diagonal-circulant networks as compact representations.
Writing:
Theorem 1: The statement about approximability with B_1B_2…B_{2n-1} is independent of p and S.
Proposition 3: The expression for depth should be \sum_{i=1}^l (4k_i+1)  — sum should go from i=1 to l and there should be no multiplicative factor l
Other non-critical comments: Multiplication by circulant matrices amounts to circular convolution with full dimensional kernel. In this sense, replacing a fully connected layers by circulant matrices is similar to replacing it with convolutional layers.  May be this connection can be explicitly stated in the paper.


Discovering and exploiting the causal structure in the environment is a crucial challenge for intelligent agents. Here we explore whether modern deep reinforcement learning can be used to train agents to perform causal reasoning. We adopt a meta-learning approach, where the agent learns a policy for conducting experiments via causal interventions, in order to support a subsequent task which rewards making accurate causal inferences. We also found the agent could make sophisticated counterfactual predictions, as well as learn to draw causal inferences from purely observational data. Though powerful formalisms for causal reasoning have been developed, applying them in real-world domains can be difficult because fitting to large amounts of high dimensional data often requires making idealized assumptions. Our results suggest that causal reasoning in complex settings may benefit from powerful learning-based approaches. More generally, this work may offer new strategies for structured exploration in reinforcement learning, by providing agents with the ability to perform-and interpret-experiments.

Note: This review is coming in a bit late, already after one round of responses. So I write this with the benefit of having read the helpful previous exchange.
I am generally positive about the paper and the broader project. The idea of showing that causal reasoning naturally emerges from certain decision-making tasks and that modern (meta-learning) RL agents can become attuned to causal structure of the world without being explicitly trained to answer causal questions is an attractive one. I also find much about the specific paper elegant and creative. Considering three grades of causal sophistication (from conditional probability to cause-effect reasoning to counterfactual prediction) seems like the right thing to do in this setting.
Despite these positive qualities, I was confused by many of the same issues as other reviewers, and I think the paper does need some more serious revisions. Some of these are simple matters of clarification as the authors acknowledge; others, however, require further substantive work. It sounds like the authors are committed to doing some of this work, and I would like to add one more vote of encouragement. While the paper may be slightly too preliminary for acceptance at this time, I am optimistic that a future version of this paper will be a wonderful contribution.
(*) The authors say at several points that the approach “did not require explicit knowledge of formal principles of causal inference.” But there seem to be a whole of lot of causal assumptions that are critically implicit in the setup. It would be good to understand this better. In particular, the different agents are hardwired to have access to different kinds of information. The interventional agent is provided with data that the conditional agent simply doesn’t get to see. Likewise, the counterfactual agent is provided with information about noise. Any sufficiently powerful learning system will realize that (and even how) the given information is relevant to the decision-making task at hand. A lot of the work (all of the work?) seems to be done by supplying the information that we know would be relevant.
(*) Previous reviewers have already made this point - I think it’s crucial - and it’s also related to the previous concern: It is not clear how difficult the tasks facing these agents actually are, nor is it clear that solving them genuinely requires causal understanding. What seems to be shown is that, by supplying information that’s critical for the task at hand, a sufficiently powerful learning agent is able to harness that information successfully. But how difficult is this task, and why does it require causal understanding? I do think that some of the work the authors did is quite helpful, e.g., dividing the test set between the easy and hard cases (orphan / parented, unconfounded / confounded). But I do not feel I have an adequate understanding of the task as seen, so to say, from the perspective of the agent. Specifically:
(*) I completely second the worry one of the reviewers raised about equivalence classes and symmetries. The test set should be chosen more deliberately - not randomly - to rule out deflationary explanations of the agents’ purported success. I’m happy to hear that the authors will be looking more into this and I would be interested to know how the results look.
(*) The “baselines” in this paper are often not baselines at all, but rather various optimal approaches to alternative formulations of the task. I feel we need more actual baselines in order to see how well the agents of interest are doing. I don’t know how to interpret phrases like “close to perfect” without a better understanding of how things look below perfection.
As a concrete case of this, just like the other reviewers, I was initially quite confused about the passive agents and why they did better than the active agents. These are passive agents who actually get to make multiple observations, rather than baseline passive agents who choose interventions in a suboptimal way. I think it would be helpful to compare against an agent who makes the same number of observations but chooses them in a suboptimal (e.g., random) way.
(*) In relation to the existing literature on causal induction, it’s telling that implementing a perfect MAP agent in this setting is even possible. This makes me worry further about how easy these tasks are (again, provided one has all of the relevant information about the task). But it also shows that comparison with existing causal inference methods is simply inappropriate here, since those methods are designed for realistic settings where MAP inference is far from possible. I think that’s fine, but I also think it should be clarified in the paper. The point is not (at least yet) that these methods are competitors to causal inference methods that do “require explicit knowledge of formal principles of causal inference,” but rather that we have a proof-of-concept that some elementary causal understanding may emerge from typical RL tasks when agents are faced with the right kinds of tasks and given access to the right kinds of data. That’s an interesting claim on its own. The penultimate paragraph in the paper (among other passages) seems to me quite misleading on this point.
(*) One very minor question I have is why actions were softmax selected even in the quiz phase. What were the softmax parameters? And would some of the agents not perform a bit better if they maximized?


ABSTRACT Federated learning distributes model training among a multitude of agents, who, guided by privacy concerns, perform training using their local data but share only model parameter updates, for iterative aggregation at the server. In this work, we explore the threat of model poisoning attacks on federated learning initiated by a single, non-colluding malicious agent where the adversarial objective is to cause the model to mis-classify a set of chosen inputs with high confidence. We explore a number of strategies to carry out this attack, starting with simple boosting of the malicious agent's update to overcome the effects of other agents' updates. To increase attack stealth, we propose an alternating minimization strategy, which alternately optimizes for the training loss and the adversarial objective. We follow up by using parameter estimation for the benign agents' updates to improve on attack success. Finally, we use a suite of interpretability techniques to generate visual explanations of model decisions for both benign and malicious models, and show that the explanations are nearly visually indistinguishable. Our results indicate that even a highly constrained adversary can carry out model poisoning attacks while simultaneously maintaining stealth, thus highlighting the vulnerability of the federated learning setting and the need to develop effective defense strategies.

The paper considers the federated learning setting as introduced by McMahan et al. (2017) and aims at securing it against model poisoning attacks.
Cons:
While I appreciated the writing clarity of the paper, the paper misses the whole point of defensive ML research: in the model poisoning case, a minimal requirement for a defense mechanism is to be formally proven *whatever is the behavior of the attacker* (within the threat model). Experiments alone are not sufficient for this purpose given the size of the space of possible attacks. Especially that (unlike evasion attacks) proofs are relatively easy to be made in the poisoning case.
For instance the literature cited by the paper (Chen 2017, Chen 2018, Blanchard 2017) + the recent follow-ups ((1)Alistarh et al. NIPS 2018, (2) El Mhamdi  et al. ICML 2018,  (3) Yin et al. ICML 2018 etc) are full of approaches the authors can follow to formally support their claims.
Also, the literature review has been done very lightly: Chen et al. 2017b (And most cited above) do *not* assume a single Byzantine agent as said in the paper, but assume up to <50% malicious (potentially colluding) agents.
Besides absence of formal support, how does the approach compare to the optimal results in (1) and (3) at least in the convex case ?
In the abstract, it is said (ii) that in the i.i.d situation, it will be easy to make spurious update standout among benign ones), this was proven wrong in (2) when the dimension of the model is large and the loss function highly non-convex, the case of neural networks for example. As a general comment, the defense mechanisms of the paper are all relying on a distance computation and thus will all provide the sqrt(d) leeway for an attacker as described in (2) and will fail preventing high-dimensionality attacks.
Pros:
I was very excited by the ideas in section 5, this work is the first to my knowledge to attempt at interpreting poisoning attacks. I suggest to the authors to either fix the issues mentioned above (and formally analyze their work), or to focus more on the interoperability question, if they want to keep the paper in the empiricist nature.


In this paper, we propose a modular approach which separates the instruction-toaction mapping procedure into two separate stages. The two stages are bridged via an intermediate representation called a goal. The first stage maps an input instruction to a goal, while the second stage maps the goal to an appropriate policy selected from a set of robot policies. The policy is selected with an aim to guide the robot to reach the goal as close as possible. We implement the above two stages as a framework consisting of two distinct modules: an instruction-goal mapping module and a goal-policy mapping module. Given a human instruction in the evaluation phase, the instruction-goal mapping module first translates the instruction to a robot-interpretable goal. Once a goal is derived by the instructiongoal mapping module, the goal-policy mapping module then follows up to search through the goal-policy pairs to look for policy to be mapped by the instruction. Our experimental results show that the proposed method is able to learn an effective instruction-to-action mapping procedure in an environment with a given instruction set more efficiently than the baselines. In addition to the impressive data-efficiency, the results also show that our method can be adapted to a new instruction set and a new robot action space much faster than the baselines. The evidence suggests that our modular approach does lead to better adaptability and efficiency.

This paper presents an instruction-following model consisting of two modules: a
goal-prediction model that maps commands to goal representations, and an
execution model that maps goal representations to policies. The second module is
trained without command supervision via a goal exploration process, while the
first module is trained supervisedly in a metric learning framework.
This paper contains an important core insight---much of what's hard about
instruction following is generic planning behavior that doesn't depend on the
semantics of instructions, and pre-learning this behavior makes it possible to
use natural language supervision more effectively. However, the paper also
contains a number of serious evaluation and presentation issues. It is obviously
not ready to publish (uncaptioned figures, paragraphs interrupted mid-sentence,
etc.) and should not have been submitted to ICLR in its present form.
SUPERVISION AND COMPARISONS
I found comparisons between supervision conditions in this paper difficult to
understand. It is claimed that the natural language instruction following
approaches described in the first paragraph "require a large amount of human
supervision" in the form of action sequences. This is not exactly true, as some
approaches (e.g. Artzi 2013), can be trained with only task completion signals.
More problematically, all these approaches are contrasted with reinforcement and
imitation learning approaches, which are claimed to use "little human
supervision". In fact, most of the approaches listed in this section use exactly
the same supervision---either action sequences (imitation learning) or task
completion signals (reinforcement learning). Indeed, the primary distinction is
that the "NLP-style" approaches are typically evaluated on their ability to
generalize to new instructions, while the "RL-style" approaches are evaluated on
the (easier) problem of fitting the complete instruction distribution as quickly
as possible.
This confusion carries into the evaluation of the approach proposed in this
paper, which is compared to RL and IL baselines. It's hard to tell from the
text, but it appears that this is an "RL-style" evaluation setting, where we
only care about rapid convergence rather than generalization. But the baselines
are inadequately described, and it's not clear to me that they condition on the
commands at all. More significantly, it's not clear what an evaluation based on
"timesteps" means for a behavior-cloning approach---is this the number of
distinct trajectories observed? The number of gradient steps taken? Without
these explanations it is impossible to interpret the experimental results.
GENERALITY OF PROPOSED APPROACH
Despite the advantages of the high-level two-phase model proposed, the specific
implementation in this paper has two significant shortcomings:
- No evidence that it works with real language: despite numerous claims
  throughout the paper that the model is designed to interpret "human
  instructions", it is revealed on p7 that these instructions consist of one or two
  5-way indicator features. This is an extremely impoverished instruction space,
  especially compared to the numerous papers cited in the introduction that make
  use of large datasets of complex natural-language strings generated by human
  annotators. The present experiments do not support the use of the word "human"
  anywhere in the paper.
- No support for combinatorial action spaces. Even if we set aside the
  distinctions between human-generated instructions and synthetic command
  languages like used in Hermann Hill & al., the goal -> policy module is
  defined by a buffer of cached trajectories and goal representations. While
  this works for the simple environments considered in this paper, it cannot
  generalize to real-world instruction-following scenarios where the number of
  distinct goal configurations is too large to tractably enumerate. Again, this
  is a shortcoming that existing approaches do not suffer from (given
  appropriate assumptions about the structure of goal space), so the lack of
  comparisons is problematic.
CLARITY
The whole paper would benefit from copy-editing by an experienced English
speaker, but a few sections are particularly problematic:
- The first paragraph of 4.1.1 is extremely difficult to understand What does
  the fingertip do? What exactly is the action space?
- The end of the second paragraph is also difficult to understand; after reading
  it I still don't know what the extra "position" targets do.
- 4.1.4 is cut off mid-way through a sentence.
- last sentence of 4.2
The figures are also impossible to interpret: three of the four are captioned
"overview of the proposed framework", and none are titled.


Generative adversarial networks (GANs) are a class of deep generative models which aim to learn a target distribution in an unsupervised fashion. While they were successfully applied to many problems, training a GAN is a notoriously challenging task and requires a significant amount of hyperparameter tuning, neural architecture engineering, and a non-trivial amount of "tricks". The success in many practical applications coupled with the lack of a measure to quantify the failure modes of GANs resulted in a plethora of proposed losses, regularization and normalization schemes, and neural architectures. In this work we take a sober view of the current state of GANs from a practical perspective. We reproduce the current state of the art and go beyond fairly exploring the GAN landscape. We discuss common pitfalls and reproducibility issues, open-source our code on Github, and provide pre-trained models on TensorFlow Hub.

This paper seems to be an exposition on the primary performance affecting aspects of generative adversarial networks (GANs).  This can possibly affect our understanding of GANs, helping practitioners get the most in their applications, and perhaps leading to innovations that positively affect GAN performance.
Normally, expositions such as this I find difficult to recommend for publication. In these times, one can find "best practices" with a reasonable amount of rigor on data science blogs and such. An exposition that I would recommend for publication, would need to exhibit a high sense of depth and rigor for me to deem it publication worthy. This paper, for me, achieves this level of quality.
The authors start off by giving a precise, constrained list of hyperparameters and architectural components that they would explore. This is listed in the title and explained in detail in the beginning of the paper. The authors are right in explaining that they could not cover all hyperparameters and chose what I feel are quite salient ones. My one ask would have been a survey of how activations might affect performance. I sense that everyone has settled upon LeakyReLUs for internal layers, but a survey of that work and experimentation within the authors' framework would have been nice.
The authors then explain the metrics for evaluation and datasets. The datasets offered a healthy variety for typical image recognition tasks. It would be interesting to see what these metrics would reveal when applied to other types of data (e.g. scientific images).
The  authors explain, with graphs, the results of the loss, normalization, and architectures. I feel the discussion on loss was rushed, and I gained no insight on what the authors thought was a prominent difference between the three losses studied. Perhaps the authors had no salient observations for loss, but explicitly stating such would be useful to the reader. The only observation I gained as far as this is that non-saturating loss would possibly be stable across various datasets.
Regularization and normalization are discussed in much more detail, and I think the authors made helpful and interesting observations, such as the benefits of spectral normalization and the fact that batch normalization in the discriminator might be a harmful thing. These are good takeaways that could be useful to a vast number of GANs researchers.
For architectures to be a main pillar of the paper, I feel that this area could have been explored in greater detail. I feel that this discussion devolved into a discussion, again, about normalization rather than the architectural differences in performance. Unless I am misunderstanding something, it seems that the authors simply tested one more architecture, for the express purpose of testing whether their observations about normalization would hold.
As a bonus, the authors bring up some problems they had in making comparisons and reproducing results. I think this is an extremely important discussion to have, and I am glad that the authors detailed the obstacles in their journey. Hopefully this will inspire other researchers to avoid adding to the complications in this field.
The graphs were difficult to parse. I was able to make them out, but perhaps separating the top row (FID and diversity graphs) into separate figures, separate lines, or something would have reduced some confusion. In addition, different charts presenting only one loss function, with their spectral normalization and gradient penalty variants, would have made the effects of the normalization more obvious on the FID distribution graphs. If this can be changed before publication, I would strongly suggest it.
I appreciate that the authors provided source code via GitHub. However, in the future, the authors should be careful to provide an anonymous repository for review purposes. I had to be careful not to allow myself to focus on the author names which are prominent in the repository readme, and one of whom has his/her name in the GitHub URL itself. I didn't immediately recognize the names and thus it was easy for me not to retain them or focus on them. However, if it had been otherwise, it might have risked biasing the review.
In all, I think this is a good and useful paper from which I have learned and to which I will refer in the future as I continue my research into GANs and VAEs. I would suggest changing the title to be more appropriate and accurate (the researchers are primarily focused on showing the positive and negative effects of normalization across various loss functions and architectures). But altogether, I believe this is a paper worth publishing at ICLR.


iclr2019_conference
Learning explainable patient temporal embeddings from observational data has mostly ignored the use of RNN architectures that excel in capturing temporal data dependencies but at the expense of explainability. This paper addresses this problem by introducing and applying an information theoretic approach to estimate and quantify the degree of explainability of such architectures. Using a communication paradigm, we formalize metrics of explainability by estimating the amount of information that an AI model needs to convey to a human end user to explain and rationalize its outputs. A key aspect of this work is to model human preferences and prior knowledge at the receiving end and measure a lack of explainability as a deviation from these human preferences. We apply this paradigm to medical patient representation problems by regularizing loss functions of temporal autoencoders according to the derived explainability metrics to guide the learning process towards models producing explainable outputs. We illustrate the approach with convincing experimental results for the generation of explainable temporal embeddings for critical care patient data.

Summary:
The authors propose a framework for training an external observer that tries explain the behavior of a prediction function using the minimal description principle. They extend this idea by considering how a human with domain knowledge might have different expectations for the observer. The authors test this framework on a multi-variate time series medical data (MIMIC-III) to show that, under their formulation, the external observer can learn interpretable embeddings.
Pros:
- Interesting approach: Trying to develop an external observer based on information theoretic perspective.
- Considering the domain knowledge of the human subject can potentially be an important element when we want to use interpretable models in practice.
Issues:
(1) In 2.4: So between M and M^O, which one is a member of M_reg?
(2) On a related note to issue (1): In 2.4.1, "Clearly, for each i, (M(X))_i | M^O(X) follows also a Gaussian distribution: First of all, I'm not sure if that expression is supposed to be  p(M(X)_i | M^O(X)) or if that was intended. But either way, I don't understand why that would follow a normal distribution. Can you clarify this along with issue (1)?
(3) In 2.4.2: The rationale behind using attention & compactness to estimate the complexity of M^O is weak. Can you elaborate this in the future version?
(4) What do each figure in Figure 4 represent?
(5) More of a philosophical question: the authors train M and M^O together, but it seems more appropriate to train an external observer separately. If we are going to propose a framework to train an agent that tries to explain a black box function, then training the black-box function together with the observer can be seen as cheating. It can potentially make the job of the observer easier by training the black box function to be easily explainable. It would have been okay if this was discussed in the paper, but I can't find such discussion.
(6) The experiments can be made much stronger by applying this approach to a specific prediction task such as mortality prediction. The current auto-encoding task doesn't seem very interesting to apply interpretation.
(7) Most importantly: I like the idea very much, but the paper clearly needs more work. There are broken citations and typos everywhere. I strongly suggest polishing this paper as it could be an important work in the model interpretability field.


LEARNING TO DESCRIBE SCENES WITH PROGRAMS
Human scene perception goes beyond recognizing a collection of objects and their pairwise relations. We understand higher-level, abstract regularities within the scene such as symmetry and repetition. Current vision recognition modules and scene representations fall short in this dimension. In this paper, we present scene programs, representing a scene via a symbolic program for its objects, attributes, and their relations. We also propose a model that infers such scene programs by exploiting a hierarchical, object-based scene representation. Experiments demonstrate that our model works well on synthetic data and transfers to real images with such compositional structure. The use of scene programs has enabled a number of applications, such as complex visual analogy-making and scene extrapolation.

[Overview]
In this paper, the authors proposed a new format of representation called scene programs, to describe the visual scenes. To extract the scene programs from scenes, the authors exploited the off-the-shelf object detection and segmentations model, mask r-cnn to extract all objects and the corresponding attributes from the images, and then detect groups for those objects, which are then used to generate the programs which matches the input scenes. The experiments are performed on a synthetics datasets which consists of multiple shapes with different attributes. The experiments shows that the proposed model can infer more accurate programs from the scenes, and those generated programs can be used to recover the input scenes more accurately. Besides, the authors also showed that the generated scene programs can be used for image editing and making visual analogy.
[Strengthes]
1. The authors proposed a new representation, called scene programs, to describe the visual scenes with some textual program. This is a new scene representation, which could be potentially used in various scenarios, such as the image synthesis in graphics.
2. The authors proposed a hierarchical method to model the structures in scenes. Specifically, the objects in a scene are first extracted and then grouped into multiple clusters, which will be used to guide the scene program synthesis.
3. The experimental results demonstrate the effectiveness of the proposed method both qualitatively and quantitatively. The authors also showed the the programs generated  can be sued for image editing and cross-modality matching.
[Weaknesses]
1. It is a bit unfair to compare the proposed method with the two baseline methods listed in Table 2. The authors used a pre-trained mask-rcnn to detect all objects and predict the attributes for all objects. However, the counterpart methods have no access to this supervision. Even in this case, CNN-LSTM seems achieve comparable performance on the first three metrics.
2. The advantage of scene program compared with scene graph (Johnson et al) are not clear to me. Scene graph is also a symbolic representation for images. Also, for all the tasks mentioned in this paper, such as image editing and visual analogy, scene graph can probably also complete well. The authors should comment about the specific advantages of scene program in comparison with scene graph.
3. All the images shown in the paper seems arranged uniformly, which I think contains some bias to the proposed grouping strategy. I would like to see more diverse configurations of the foreground objects. It would be good to see if the proposed model can describe more complicated scenes.
[Summary]
This paper proposed a novel scene representations, called scene program. To extract the scene program, the authors proposed a hieratchical inference method. The resulting scene programs based on the proposed model outperforms several baseline models quantitatively. The authors also showed the proposed scene program is suitable for image editing and visual analogy making. However, as pointed above, there are some unclear points to me, especially the advantages of scene program compared with scene graph, and the representation power of scene program for complicated scenes.


SYSTEMATIC GENERALIZATION: WHAT IS REQUIRED AND CAN IT BE LEARNED?
Numerous models for grounded language understanding have been recently proposed, including (i) generic models that can be easily adapted to any given task and (ii) intuitively appealing modular models that require background knowledge to be instantiated. We compare both types of models in how much they lend themselves to a particular form of systematic generalization. Using a synthetic VQA test, we evaluate which models are capable of reasoning about all possible object pairs after training on only a small subset of them. Our findings show that the generalization of modular models is much more systematic and that it is highly sensitive to the module layout, i.e. to how exactly the modules are connected. We furthermore investigate if modular models that generalize well could be made more end-to-end by learning their layout and parametrization. We find that endto-end methods from prior work often learn inappropriate layouts or parametrizations that do not facilitate systematic generalization. Our results suggest that, in addition to modularity, systematic generalization in language understanding may require explicit regularizers or priors.

This paper presents a targeted empirical evaluation of generalization in models
for visual reasoning. The paper focuses on the specific problem of recognizing
(object, relation, object) triples in synthetic scenes featuring letters and
numbers, and evaluates models' ability to generalize to the full distribution of
such triples after observing a subset that is sparse in the third argument. It
is found that (1) NMNs with full layout supervision generalize better than other
state-of-the art visual reasoning models (FiLM, MAC, RelNet), but (2) without
supervised layouts, NMNs perform little better than chance, and without
supervised question attentions, NMNs perform better than the other models but
fail to achieve perfect generalization.
STRENGTHS
- thorough analysis with a good set of questions
WEAKNESSES
- some peculiar evaluation and presentation decisions
- introduces *yet another* synthetic visual reasoning dataset rather than
  reusing existing ones
I think this paper would have been stronger if it investigated a slightly
broader notion of generalization and had some additional modeling comparisons.
However, I found it interesting and think it successfully addresses the set of
questions it sets out to answer. If it is accepted, there are a few things that
can be done to improve the experiments.
MODELING AND EVALUATION
- Regarding the dataset: the proliferation of synthetic reasoning datasets is
  annoying because it makes it difficult to compare results without downloading
  and re-running a huge amount of code. (The authors have, to their credit, done
  so for this paper.) I think all the experiments here could have been performed
  successfully using either the CLEVR or ShapeWorld rendering engines: while the
  authors note that they require a "large number of different objects", this
  could have been handled by treating e.g. "red circle" and "red square" as
  distinct atomic primitives in questions---the fact that redness is a useful
  feature in both cases is no different from the fact that a horizontal stroke
  detector is useful for lots of letters.
- I don't understand the motivation behind holding out everything on the
  right-hand side. For models that can't tell that the two are symmetric, why
  not introduce sparsity everwhere---hold out some LHSs and relations?

- Table 1 test accuracies: arbitrarily reporting "best of 3" for some model /
  dataset pairs and "confidence interval of 5" for others is extremely
  unhelpful: it would be best to report (mean / max / stderr) for 5. Also, it's
  never stated which convidence interval is reported.
- Table 1 baselines: why not run Conv+LSTM and RelNet with easier #rhs/lhs data?
- How many MAC cells are used? This can have significant performance
  implications. I think if you used their code out of the box you'll wind up
  with way bigger structures than you need for this task.
- I'm not sure how faithful the `find` module used here is to the one in the
  literature, and one of the interesting claims in this work is that module
  implementation details matter! The various Hu papers use an attentional
  parameterization; the use of a ReLU and full convolution in Eq. 14 suggest
  that that one here can pass around more general feature maps. This is fine but
  the distinction should be made explicit, and it would be interesting to see
  additional comparisons to an NMN with purely attentional bottlenecks.
- Why do all the experiments after 4.3 use #rhs/lhs of 18? If it was 8 it would
  be possible to make more direct comparisons to the other baseline models.
- The comparison to MAC in 4.2 is unfair in the following sense: the NMN
  effectively gets supervised textual attentions if the right parameters are
  always plugged into the right models, while the MAC model has to figure out
  attentions from scratch. A different way of structuring things would be to
  give the MAC model supervised parameterizations in 4.2, and then move the
  current MAC experiment to 4.3 (since it's doing something analogous to
  "parameterization induction".

- The top-right number in Table 4---particularly the fact that it beats MAC and
  sequential NMNs under the same supervision condition---is one of the most
  interesting results in this paper. Most of the work on relaxing supervision
  for NMNs has focused on (1) inducing new question-specific discrete structures
  from scratch (N2NMN) or (2) finding fixed sequential structures that work well
  in general (SNMN and perhaps MAC). The result this paper suggests an
  alternative, which is finding good fixed tree-shaped structures but continuing
  to do soft parameterization like N2NMN.
- The "sharpness ratio" is not super easy to interpret---can't you just report
  something standard like entropy? Fig 4 is unnecessary---just report the means.
- One direction that isn't explored here is the use of Johnson- or Hu-style
  offline learning of a model to map from "sentences" to "logical forms". To the
  extent that NMNs with ground-truth logical forms get 100% accuracy, this turns
  the generalization problem studied here into a purely symbolic one of the kind
  studied in Lake & Baroni 18. Would be interesting to know whether this makes
  things harder (b/c no grounding signal) or easier (b/c seq2seq learning is
  easier.)
PRESENTATION
- Basically all of the tables in this paper are in the wrong place. Move them
  closer to the first metnion---otherwise they're confusing.
- It's conventional in this conference format to put all figure captions below
  the figures they describe. The mix of above and below here makes it hard to
  attach captions to figures.
- Some of the language about how novel the idea of studying generalization in
  these models is a bit strong. The CoGenT split of the CLEVR dataset is aimed
  at answering similar questions. The original Andreas et al CVPR paper (which btw
  appears to have 2 bib entries) also studied generalization to structurally
  novel inputs, and Hu et al. 17 notes that the latent-variable version of this
  model with no supervision is hard to train.
MISCELLANEOUS
- Last sentence before 4.4: "NMN-Chain" should be "NMN-Tree"?
- Recent paper with a better structure-induction technique:
  https://arxiv.org/abs/1808.09942. Worth citing (or comparing if you have
  time!)


We consider the problem of representation learning for graph data. Convolutional neural networks can naturally operate on images, but have significant challenges in dealing with graph data. Given images are special cases of graphs with nodes lie on 2D lattices, graph embedding tasks have a natural correspondence with image pixel-wise prediction tasks such as segmentation. While encoder-decoder architectures like U-Net have been successfully applied on many image pixel-wise prediction tasks, similar methods are lacking for graph data. This is due to the fact that pooling and up-sampling operations are not natural on graph data. To address these challenges, we propose novel graph pooling (gPool) and unpooling (gUnpool) operations in this work. The gPool layer adaptively selects some nodes to form a smaller graph based on their scalar projection values on a trainable projection vector. We further propose the gUnpool layer as the inverse operation of the gPool layer. The gUnpool layer restores the graph into its original structure using the position information of nodes selected in the corresponding gPool layer. Based on our proposed gPool and gUnpool layers, we develop an encoder-decoder model on graph, known as the graph U-Net. Our experimental results on node classification tasks demonstrate that our methods achieve consistently better performance than previous models.

This paper proposes pooling and upsampling operations for graph structured data, to be interleaved with graph convolutions, following the spirit of fully convolutional networks for image pixel-wise prediction. Experiments are performed on node classification benchmarks, showing an improvement w.r.t. architectures that do not perform any downsampling/upsampling operations.
Given that the main contribution of the paper is the introduction of a pooling operation for graph structured data, it might be a good idea to evaluate the operation in a task that does require some kind of downsampling, such as graph classification / regression. Moreover, authors should compare to other graph pooling methods.
Authors claim that one of the motivations to perform their pooling operation is to increase the receptive field. It would be worth comparing pooling/upsamping to dilated convolutions to see if they have the same effect on the performance when dealing with graphs.
Some choices in the method seem rather arbitrary, such as the tanh non-linearity in \tilde y. Could the authors elaborate on that? How important is the gating?
It would be interesting to analyze which nodes where selected by the pooling operators. Are those nodes close together or spread out in the previous graph?
The proposed unpooling operation seems to be the same as unpooling performed to upsample images, that is using skip connections to track indices, by recovering the position where the max value comes from and setting the rest to 0. Have the authors tried other upsampling strategies analogous to the ones typically used for images (e.g. upsampling with nearest neighbors)?
When skipping information from the downsampling path to the upsampling path, is there a concatenation or a summation? How do both operations compare? (note that concatenation introduces many more parameters) How about only skipping only the indices (no summation nor concatenation)? This kind of analysis, as it has been done in the computer vision literature, would be interesting.
What is the influence of the first embedding layer to reduce the dimensionality of the features?
How do the models in Table 2 compare in terms of number of parameters?

What's the influence of imposing larger weights on self loop in the graph?
What about experiments in inductive settings?
Please add references for the following claim "U-Net models with depth 3 or 4 are commonly used..."
Please double check your references, e.g. in the introduction, citations used for CNNs do not always correspond to CNN architectures.
The literature review could be significantly improved, missing relevant papers to discuss include:
- Gori et al. A new model for learning in graph domains, 2005.
- Scarselli et al. The graph neural network model, 2009.
- Bruna et al. Spectral networks and locally connected networks on graphs, 2014.
- Henaff et al. Deep convolutional networks on graph-structured data, 2015.
- Niepert et al. Learning convolutional neural networks for graphs, 2016.
- Atwood and Towsley. Diffusion-convolutional neural networks, 2016.
- Bronstein et al. Geometric deep learning: going beyond Euclidean data, 2016.
- Monti et al. Geometric deep learning on graphs and manifolds using mixture model cnns, 2017.
- Fey et al. SplineCNN: Fast Geometric Deep Learning with Continuous B-Spline Kernels, 2017.
- Gama et al. Convolutional Neural Networks Architectures for Signals Supported on Graphs, 2018.
As well as other pixel-wise architecture for image-based tasks such as:
- Long et al. Fully Convolutional Networks for Semantic Segmentation, 2015.
- Jegou et al. The one hundred layers tiramisu: fully convolutional densenets for semantic segmentation, 2016.
- Isola et al. Image-to-image translation with conditional adversarial networks, 2016.
- Zhao et al. Stacked What-Where auto-encoders, 2015.


Object-based approaches for learning action-conditioned dynamics has demonstrated promise for generalization and interpretability. However, existing approaches suffer from structural limitations and optimization difficulties for common environments with multiple dynamic objects. In this paper, we present a novel self-supervised learning framework, called Multi-level Abstraction Objectoriented Predictor (MAOP), for learning object-based dynamics models from raw visual observations. MAOP employs a three-level learning architecture that enables efficient dynamics learning for complex environments with a dynamic background. We also design a spatial-temporal relational reasoning mechanism to support instance-level dynamics learning and handle partial observability. Empirical results show that MAOP significantly outperforms previous methods in terms of sample efficiency and generalization over novel environments that have multiple controllable and uncontrollable dynamic objects and different static object layouts. In addition, MAOP learns semantically and visually interpretable disentangled representations.

In this paper, the novel MAOP model is described for self-supervised learning on past video game frames to predict future frames. The presented results indicate that the method is capable of discovering semantically important visual components, and their relation and dynamics, in frames from arcade-style video games. Key to the approach is a multi-level self-learning approach:  more abstract stages focus on simpler problems that are easier to learn, which in turn guide the learning process at the more complex stages.
A downside is that it the method is complex, consisting of many specific sub-components and algorithms, which in turn have again other sub-components. This makes the paper a long read with a lot of repetition, and various times the paper refers to the names of sub-components that are only explained later. Other methodological details that are relevant to understand how the method operates are described in the Appendices. I expect that if the paper would be better structured, it would be easier to understanding how all the parts fit together. Another downside of this complexity is that the method seems designed for particular types of video game frames, with static backgrounds, a fixed set of objects or agents. It is unclear how the method would perform on other types of games, or on real-world videos. While the method therefore avoids the need for manual annotation, it instead encodes a lot of domain knowledge in its design and components.
I also didn't fully understand how the self-supervised model is used for Reinforcement Learning in the experiments. Is the MAOP first trained, and the fixed to perform RL with the learned agent models, or is the MOAP learned end-to-end during RL?
Pros:
+ MAOP seems successful on the tested games in the experiments
+ Demonstrates that, with a sufficiently engineered method, self-supervised learning can be used to discover different types of objects, and their dynamics.
Cons:
- writing could be improved, as the methodology currently reads as a summation of facts, and some parts are written out of order, resulting in various forward references to components that only become clear later. Several times, the paper states that some novel algorithm is used, but then provides no further explanation in the text as all description of this novelty is deferred to an appendix.
- method does not seem generic, hence it is unclear how relevant this architecture it is to other use cases
- many hyperparameters for the individual components, algorithms. Unclear how these parameter setting affect the results
Below are more detailed comments and questions:
General comments:
* The proposed MOAP method consists of many subalgorithms, resulting in various (hyper)parameters which may impact the results (e.g. see Appendix A, B). Appendix D lists several used hyperparameter settings, though various parameters for the algorithms are still missing (e.g. thresholds alpha, beta in Algo.2). Were the used parameters optimized? How are these hyperparameters set in practice? How does changing them impact your results?
* Methods seems particularly designed for 'video games', where the object and background structures have well defined sizes, appearance, etc. How will the MOAP fair in more realistic situations with noisy observations, occluded objects, changing appearances and lighting conditions, etc.?
* How about changing appearance of an agent during an action, e.g. a 'walking animation' ? Can your method learn the sequence of sprites to accurately predict the next image? Is that even part of the objective?
* Appendix D has important implementation details, but is never mentioned in the text I believe! Didn't realize it existed on first read through.
* Introduction:
	* What prediction horizon are you targeting? 1 step, T steps into the future, 1 to T steps in the future simultaneously?
	What are you trying to predict? Object motion? Future observations?
	* "... which includes a CNN-based Relation Net to ... ", the names Relation Net, Inertia Net, etc.. are used as if the reader is expected to know what these are already. If these networks were introduced in related work already, please add citations. Otherwise please rephrase to clarify that these are networks themselves are part of your novel design.
* Section 3.1
	* "It takes multiple-frame video images ... and produce the predictions of raw visual observations.". As I understand from this, the self-supervised approach basically performs supervised learning to predict a future frame (target output) given past frames (input). I do not understand how this relates to Reinforcement Learning (RL) as mentioned in the introduction and Related Work. Is there still some reward function in play when learning the MAOP parameters? Or is the idea to first self-supervised learn the MAOP, and afterwards fix its parameters and use it in separate a RL framework? I believe RL is not mentioned anymore until Section 4.2. This connection between self-supervised and reinforcement learning should be clarified, or otherwise the related work should be adjusted to include other (self-supervised) work on predicting future image frames.
	* "An object mask describes the spatial distribution of an object ..." Does the distribution capture uncertainty on the object's location, or does it capture the spread of the object's extent ('mass distribution') ?
	* "Note that Object Detector uses the same CNN architecture with OODP". What does OODP stand for? Add citation here. (first mention of OODP is in Experiments section)
	* "(similar with Section 3.2)" → "similar to". Also, I find it a confusing to say something is similar to what will be done in a future section, which has not yet been introduced. Can you not explain the procedure here, and in Section 3.2 say that the procedure is "similar to Section 3.1" instead?
	* "to show the detailed structure of the Effect Net module." First time I see the name 'Effect Net', what is it? This whole paragraph different nets are named, with a rough indication of their relation, such as "Dynamic Net", "Relation Net" and "Inertia Net". Is "Effect Net" a different name for any of the three previous nets? The paper requires the reader to puzzle from Fig.2 that Relation Net and Inertia Net are parts of Effect Net, which in turn is part of Dynamics Net. This wasn't clear from the text at all.
* Section 3.2:
	* p7.: "Since DISN leans" → "Since DISN learns" ?
	* There are many losses throughout the paper, but I only see at the end of Section 3.1 some mentioning that multiple losses are combined. How is this done for the other components, .e.g is the total loss for DISN a weighted sum of L_foreground and L_instance ? Are the losses for all three three MAOP levels weighted for full end-to-end learning?
	* This section states various times "we propose a novel [method]", for which then no explanation is given, and all details are explained in the Appendix. While the Appendix can hold important implementation details, I would still expect that novelties of the paper are clearly explained in the paper itself. As it stands, the appendix is used as an extension of the methodological section of an already lengthy paper.
	* "Conversely, the inverse function is ... " M has a mask for each of the n_o "object classes", hence the "Instance Localization Module" earlier to split out instances from the class masks. So how can there be a single motion vector STN^-1(M,M') if there are multiple instances for an object mask? How will STN^-1 deal with different amount of instances in M and M' ?
* Section 3.3:
	* What is the output of this level? I expect some mathematical formulation as in the previous sections, resulting in some symbol, that is then used in Section 3.2. E.g. is the output "foreground masks F" (found in Appendix A) ?  This paper is a bit of a puzzle through the pages for the reader.
* Section 4:
	* "We compare MAOP with state-of-the-art action-conditioned dynamics learning baselines, ..." Please re-iterate how these methods differ in assumptions, what they model, with respect to your novel method? For instance, is the main difference your "novel region proposal method" and such? Is the overall architecture different? E.g. explain here already the AC Model uses "pixel-level inference", and that OODP has "lacks knowledge on object-to-object relations" to underline their difference to your approach, and provide context for your conclusions in Section 4.1.
* Appendix A:
	* Algorithm 1, line 7: "sample a pixel coordinate" → is this non-deterministically sampling?


LEARNING EMBEDDINGS INTO ENTROPIC WASSERSTEIN SPACES
Euclidean embeddings of data are fundamentally limited in their ability to capture latent semantic structures, which need not conform to Euclidean spatial assumptions. Here we consider an alternative, which embeds data as discrete probability distributions in a Wasserstein space, endowed with an optimal transport metric. Wasserstein spaces are much larger and more flexible than Euclidean spaces, in that they can successfully embed a wider variety of metric structures. We exploit this flexibility by learning an embedding that captures semantic information in the Wasserstein distance between embedded distributions. We examine empirically the representational capacity of our learned Wasserstein embeddings, showing that they can embed a wide variety of metric structures with smaller distortion than an equivalent Euclidean embedding. We also investigate an application to word embedding, demonstrating a unique advantage of Wasserstein embeddings: We can visualize the high-dimensional embedding directly, since it is a probability distribution on a low-dimensional space. This obviates the need for dimensionality reduction techniques like t-SNE for visualization.

The paper ‘Learning Discrete Wasserstein Embeddings' describes a new embedding method that,
contrary to usual embedding approaches, does not try to embed (complex, structured) data into an
Hilbertian space where Euclidean distance is used, but rather to the space of probability measures
endowed with the Wasserstein distance. As such, data are embed on an empirical
distribution supported by Diracs, which locations can be determined by a map that is learnt from data.
Interestingly, authors note a 'potential universality' for W_p(R^3) (from a result of Andoni et al., 2015),
suggesting that having Diracs in R^3 could embed potentially any kind of metric on symbolic data.
Experimental validations are presented on graph and word embedding, and a discussion on visualization of
the embedding is also proposed (since the Diracs are located in a low dimensional space).
All in all the paper is very clear and interesting. The idea of embedding in a Wasserstein space is
original (up to my knowledge) and well described. I definitely believe that this work should be presented
at ICLR. I have a couple of questions and remarks for the authors:
 - It is noted in section 3.2 that both Diracs location and associated weights could be optimized. Yet the authors
   chose to only optimize locations. Why not only optimizing the weights (as in an Eulerian view of probability
   distributions) ? The sentence involving works of Brancolini and Claici 2018 is not clear to me. Why weighting
   does not improve asymptotically the approximation quality ?
 - Introducing the entropic regularization is mainly done for being able to differentiate the Wasserstein
   distance. However, few is said on the embeddability of metrics in W^\lambda_p(R). Is using an entropic
   version of W moderating the capacity of embedding ? At least experimentally, a discussion could be made
   on the choice of the regularization parameter, at least in section 4.1. In eq. (9), it seems that it is not
   the regularized version of W. ?
 - I assume that the mapping is hard to invert, but did the authors tried to experiment reconstructing an object
   of interest by following a geodesic in the Wasserstein space ?
 - It seems to me that authors never give generalization results. What is the performance of the metric approximation
   when tested on unseen graphs or words ? This point should be clarified in the experiment.


In recent years, reinforcement learning methods have been applied to model gameplay with great success, achieving super-human performance in various environments, such as Atari, Go and Poker. However, those studies mostly focus on winning the game and have largely ignored the rich and complex human motivations, which are essential for understanding the agents' diverse behavior. In this paper, we present a multi-motivation behavior modeling which investigates the multifaceted human motivations and models the underlying value structure of the agents. Our approach extends inverse RL to the vectored-valued setting which imposes a much weaker assumption than previous studies. The vectorized rewards incorporate Pareto optimality, which is a powerful tool to explain a wide range of behavior by its optimality. For practical assessment, our algorithm is tested on the World of Warcraft Avatar History dataset spanning three years of the gameplay. Our experiments demonstrate the improvement over the scalarization-based methods on real-world problem settings.

======== Summary ============
The authors consider a setup where there is a set of trajectories (s_t, a_t, r_t) where r_t is a *vector* of rewards. They assume that each agent is trying to maximize \sum_t \gamma^t (\phi . r_t) where \phi is a preference vector that lives on the simplex. Their goal is to calculate \phi (and maybe also an optimal policy under \phi?). The
The authors first prove that this problem can be decomposed into finding Q functions for optimal policies for each component of r_t individually, and then solving for \phi that rationalizes the trajectory of actions in terms of these Q functions. Given the entire collection of trajectories, they perform off-policy Q-learning on each component of r_t in order to learn the Q function for that component, and then use linear programming to solve for \phi based on these Q function.
========== Comments =============
I think it's a worthwhile direction to combine IRL with modeling a diversity of preferences among agents. I can imagine several reasons you might want to do this, but the authors are not clear what their goal is besides "to propose methods that can help to understand the intricacy and complexity of human motivations and their behaviors". Is the goal to do better policy prediction? To do better policy prediction conditional on \phi? To infer \phi to understand people's preferences from a social science perspective? These all seems reasonable but not sufficiently teased out in the work. (For comparison, IRL is typically - although not always - interested in learning the reward function in order to construct robust policies that maximize it). The authors also don't seem to solve a particular task of importance on the WoW dataset.
The theoretical approach seems sound, and I liked the way their algorithm was motivated and the way the problem was decomposed into off-policy Q-learning and then solving for \phi.
However, I found myself quite confused in the experimental section (4.3). The authors evaluate their approach by action prediction. Given the trajectories, is \phi computed for each player and then compute actions based on that value of \phi? Is \phi computed on the same trajectory data used for evaluation or a different subset? Or is action prediction performed in aggregate across the entire population? The experimental setup was never clarified for this (main) experiment.
I was also confused about the motivation for Figure 2 and Appendix D. The authors are showing that their predictions about which reward is motivating the players is consistent with external factors. But wouldn't you see the same thing if you just plotted the observed *rewards* themselves? E.g. players in a guild will achieve more Relationship reward.
The proposed approach takes the vector of reward, learns which actions are consistent with achieving each reward, then infers from the actions which reward is trying to be achieved. What advantages does this have vs. just looking at the empirical trajectory of rewards for each player/group?
I can certainly imagine that the IRL approach has certain advantages over looking at the empirical reward stream, but the authors have not talked about this nor compared against it experimentally.
The writing could also use some improvement for a future iteration, I've listed a few points below:
pg.1, Neither Brown & Sandholm nor Moravcik et al use "RL algorithms"
pg.1, Finn et al unmatched )
pg.1, "a scalar reward despite observed or not" -> "a scalar reward whether observed or not"
pg.2, "Either the range of" -> "Both the range of" (and this sentence needs further cleanup)
pg.2, "which records the pathing of players" ??
Theorem 3: "each of the set e_i has an unique element..." This isn't clear. I think you mean "For each e_i there is a unique vector v^\pi(s) for all \pi \in \Pi_{e_i} . The equality holds if these vectors are distinct for each e_i".
pg. 5 "If otherwise all elements in \phi are generative" how can they be negative if they are on the simplex?
pg.5 "we do not perform any scalarization on the reward...the model assumption is easier to be satisfied" I think this is a strange comparison to IRL because in IRL you're trying to find a (possibly parametric) function (s,a) -> R, whereas here you're *given* the vector R and are trying to find \phi. So while you have more degrees of freedom by adding \phi, you lose the original degrees of freedom in the reward function.


Regression-via-Classification (RvC) is the process of converting a regression problem to a classification one. Current approaches for RvC use ad-hoc discretization strategies and are suboptimal. We propose a neural regression tree model for RvC. In this model, we employ a joint optimization framework where we learn optimal discretization thresholds while simultaneously optimizing the features for each node in the tree. We empirically show the validity of our model by testing it on two challenging regression tasks where we establish the state of the art.

Summary:
The paper presents a novel supervised-learning method for regression using decision trees and neural nets. The core idea is based on a 90s technique called "regression via classification" by first apply discretization of target response y by some clustering, and apply any "classification" to those discretized values as class labels. Because real-valued y is one-dimensional and ordered, discretization means setting up any thresholds to give N-partitions of training {y_i}s. The proposed method tries to jointly learn these thresholds as well as node splitters of decision trees using neural nets. Because each node splitters are given by neural nets here, probability outputs for binary classification are also available. Regarding these probabilities as probabilistic splitting at each node, response y weighted by the path probabilities to leaves is the final prediction. The learning is in a greedy manner as in standard tree learning because exact joint learning is computationally hard.  Experiments on speaker profiling illustrate the performance improvements against standard nonlinear regression such as SVR and regression trees.
Comment:
This is a technically very interesting contribution, but several points can be considered more carefully as below.
- To be honest, it would be unconvincing that the approach "regression via classification (RvC)" is still valid. The proposed approach is an elaborate extension of this approach, but if we want prediction performance for regression, we would use some ensembles of regression trees such as Random forest, GBDT, ExtraTrees, ... instead of a single CART. Or even we can directly use deep learning based regression. The experiments against CART and SVR would be too naive in the current context of supervised learning. On the other hand, single CARTs are well interpretable and can be a nice tool to get some interpretations of the given data. But the proposed method seems to lose this type of interpretability because of introducing node splitters by neural nets. So the merits of the proposed approach would be somewhat unclear.
- In the context of tree learning, we need to consider two things.
First of all, node splitting by general binary splitters are called "multivariate trees", but interestingly this does not always bring the good prediction performance on current quite high-dimensional data. So I guess that both optimizing "threshold for RvC" and "nonlinear node splitters" cannot always bring the prediction performance. Limitations and conditions would need to be clarified more carefully.
Second of all, probabilistic consideration of decision trees such as eq(4) is almost like so-called "probabilistic decision trees" also known as "hierarchical mixtures of experts (HME)" in machine learning. See famous widely-cited papers of Jordan & Jacobs 1994 and Bishop & Svensen 2003. This can bring joint learning of probabilistic node splitter (gating networks) and decision functions at leaves (expert networks), and is also known to bring the smoothing effect into discrete and unstable regression trees, and hence the improved prediction performance. So which of probabilistic consideration or RvC contributes to the observed improvement is unclear...
- The target joint optimization of eq (3) is actually optimized by a number of heuristic ways, and it is quite unclear how it is truly optimized. In contrast, HME learning is formulated as a joint optimization (and solved by EM in the case of Jordan & Jacobs, for example).
- The experiments on single datasets of a very specific speaker profiling problem would be somewhat misleading. Probably, for this specific problem, there would be other existing methods. On the other hand, if this is for benchmarking purpose, a regression by neural nets and tree ensemble (random forest or something) can be included as other baselines, and also other types of regression problems can be tested.


Model interpretability and systematic, targeted model adaptation present central challenges in deep learning. In the domain of intuitive physics, we study the task of visually predicting stability of block towers with the goal of understanding and influencing the model's reasoning. Our contributions are two-fold. Firstly, we introduce neural stethoscopes as a framework for quantifying the degree of importance of specific factors of influence in deep networks as well as for actively promoting and suppressing information as appropriate. In doing so, we unify concepts from multitask learning as well as training with auxiliary and adversarial losses. Secondly, we deploy the stethoscope framework to provide an in-depth analysis of a state-of-the-art deep neural network for stability prediction, specifically examining its physical reasoning. We show that the baseline model is susceptible to being misled by incorrect visual cues. This leads to a performance breakdown to the level of random guessing when training on scenarios where visual cues are inversely correlated with stability. Using stethoscopes to promote meaningful feature extraction increases performance from 51% to 90% prediction accuracy. Conversely, training on an easy dataset where visual cues are positively correlated with stability, the baseline model learns a bias leading to poor performance on a harder dataset. Using an adversarial stethoscope, the network is successfully de-biased, leading to a performance increase from 66% to 88%.

This paper is about using "neural stethoscopes", small complementary neural networks that are added to a main network which with their auxilary loss functions can measure suitability of features or guide the learning process. The idea is incremental to multi-task learning and enable, in a single framework, to validate intermediate features for additional related tasks. Moreover it can promote or suppress the correlation of such features to the tasks related to the main one. The framework is applied to the task of visual stability prediction of block towers. The paper builds upon Groth et al. 2018, adding the concept of local stability as correlated secondary task, used with the proposed neural stethoscopes. Experiments with an extension of ShapeStacks (Groth et al. 2018) dataset where the local stability is added to the global stability class, show that it is possibile increase the performance using the additional task. Moreover, it is shown that neural stethoscopes can suppress nuisance information when using a biased training dataset where the local and global stability are purposely inversely correlated.
Strengths:
+ A very nice paper, well written and easy to read. Figures are helpful and the structure is clear.
+ The concept of neural stethoscope is interesting and simplify the concepts behind multitask learning.
+ Experiments are convincing, interesting and there is some novelty in vision stability prediction.
Weaknesses:
- The novelty is limited related to multitask learning, thus it is an incremental paper.


Neural models achieve state-of-the-art performance due to their ability to extract salient features useful to downstream tasks. However, our understanding of how this task-relevant information is included in these networks is still incomplete. In this paper, we examine two questions (1) how densely is information included in extracted representations, and (2) how similar is the encoding of relevant information between related tasks. We propose metrics to measure information density and cross-task similarity, and perform an extensive analysis in the domain of natural language processing, using four varieties of sentence representation and 13 tasks. We also demonstrate how the proposed analysis tools can find immediate use in choosing tasks for transfer learning.

This paper tries to quantify how "dense" representations we need for a specific task -- more specifically, how many dimensions are needed from a given representation (for a given task) to achieve a percentage of the performance of the entire representation. The second thing the paper tries to quantify is how well representations learned for one task can be fine tuned for another. Experiments are conducted with 4 different representation technique on a dozen or so tasks.
Quick summary: While I liked aspects of this -- including the motivation of having a lightweight way of understanding how well representations transfer across tasks, overall my concerns surrounding the methodology and some missing analysis leads me to believe this needs more work before it is ready for publication.
Quality: Below average
I believe the proposed techniques have some flaws which hurt the eventual method. There are also concerns about the motivations behind parts of the technique.
Clarity: Fair
There were some experimental details that were poorly explained but in general the paper was readable.
Originality: Fair
There were some nice ideas in the work but I remain concerned about aspects of it.
Significance: Below average
My concern is that the flaws in the method do not make it conducive to use as is.
Strengths / Things I liked:
+ I really liked the motivating problem of being able to (hopefully cheaply / efficiently) estimate transfer potential to understand how well representations will perform on a different task.
+ Multiple representations and tasks experimented with
Weaknesses / Things that concerned me:
(In no specific order)
- (W1) Adversely affected by rotations: One of my big concerns with the work is the way the CFS is computed. While it seems ok to estimate these different metrics using only linear models, my concern with this is that the linear models are only given a subset of the **exact** dimensions of the original representations. This is very much unlike the learning objectives of most of these representation learning methods and hence is highly biased and dependent on the actual methods and the random seeds used and the rotations it performs. (In many cases the representations are used starting with a fully connected layer bottom layer on top of the representations and hence rotations of the representations do not affect performance)
Let's take an example: Say there is a single dimension of the representation that is a perfect predictor of a task. Suppose we rotated these representations. Now the signal from the original dimension is split across multiple dimensions and hence the CFS may be deceivingly high.
To me this is a big concern as different runs of the same representation technique can likely have very different CFS scores based on initializations and random seeds.
- (W2) Related to the last line: I did not see any experiments / analysis showing how stable these different numbers are across different runs of the representation technique. Nor did I see any error bars in the experiments. This again greatly concerned me as I am not certain how stable these metrics are.
- (W3) Baselines for transfer learning: I felt this was another notable oversight. I would have liked to see results for both trivial baselines like random ranking as well as more informed baselines where we can estimate transfer potential using say k representation techniques, and then use that to help us understand how well it would do on the other representations. This latter baseline is a zero-cost baseline as it is not even dependent on the method.
- (W4) Metrics for ranking of transfer don't make sense (and some are missing) : I also don't understand how "precision" and NDCG are used as metrics. Based on my understanding the authors rank (which itself is questionable) the different tasks in order of potential for transfer and then call this the "gold" set. How is precision and NDCG calculated from this?
More importantly I don't believe looking at rank alone is sufficient since that completely obscures the actual performance numbers obtained via transfer. In most cases I would care about how well my model would perform on transfer not just which tasks I should transfer from. I would have wanted to understand something like the correlation of these produced scores with the actual ground truth performance numbers.
- (W5) Multi-task learning: I did not see any mention or experiments of what can be expected when the representations are themselves trained on multiple tasks. (This seems like something that could easily be done in the empirical analysis as well and would provide richer empirical signals as well)
- (W6) Motivation for CFS: I still don't fully understand the need to understand the density of the representation (especially in the manner proposed in the paper). Why is this an important problem? Perhaps expanding on this would be helpful
- (W7) Alternatives to CFS / Computational concerns: A big concern I had was the computational expense of the proposed approach. Unfortunately I did not see any discussion about this in the paper or empirically.
I find this striking because I can easily come up with cheaper alternatives to get at this "density". For example using LASSO / LARS like methods you can perhaps figure out a good reduced dimension set more efficiently.
If I were to go through the computation of then why not just train a smaller version of that representation technique instead and **directly** see how well it can encode data in k dimensions via that technique / for that task?
Alternatively why not try using a factorization technique to reduce the rank and then see how well the method does for different ranks?
- (W7b) Likewise I wonder if we could just measure transfer more directly as well and why we need to go via these CFS sets
- (W8) The proposed  CLF weight difference method has some concerning aspects as well. For example say we had two task with exact opposite labels. They would have a very low weight difference score though they are ideal representations for each other. Likewise looking at a difference of weight vectors seems arbitrary in other ways as well.