Published as a conference paper at ICLR 2018 RELATIONAL NEURAL EXPECTATION MAXIMIZATION: UNSUPERVISED DISCOVERY OF OBJECTS AND THEIR INTERACTIONS
Common-sense physical reasoning is an essential ingredient for any intelligent agent operating in the real-world. For example, it can be used to simulate the environment, or to infer the state of parts of the world that are currently unobserved. In order to match real-world conditions this causal knowledge must be learned without access to supervised data. To address this problem we present a novel method that learns to discover objects and model their physical interactions from raw visual images in a purely unsupervised fashion. It incorporates prior knowledge about the compositional nature of human perception to factor interactions between object-pairs and learn efficiently. On videos of bouncing balls we show the superior modelling capabilities of our method compared to other unsupervised neural approaches that do not incorporate such prior knowledge. We demonstrate its ability to handle occlusion and show that it can extrapolate learned knowledge to scenes with different numbers of objects.

Summary
---
This work applies a representaion learning technique that segments entities to learn simple 2d intuitive physics without per-entity supervision. It adds a relational mechanism to Neural Expectation Maximization and shows that this mechanism provides a better simulation of bouncing balls in a synthetic environment.
Neural Expectation Maximization (NEM) decomposes an image into K latent variables (vectors of reals) theta_k. A decoder network reconstructs K images from each of these latent variables and these K images are combined into a single reconstruction using pixel-wise mixture components that place more weight on pixels that match the ground truth. An encoder network f_enc() then updates the latent variables to better explain the reconstructions they produced.
The neural nets are learned so that the latent variables reconstruct the image well when used by the mixture model and match a prior otherwise. Previously NEM has been shown to learn variables which represent individual objects (simple shapes) in a compositional manner, using one variable per object.
Other recent neural models can learn to simulate simple 2d physics environments (balls bouncing around in a 2d plane). That work supervises the representation for each entity (ball) explicitly using states (e.g. position and velocity of balls) which are known from the physics simulator used to generate the training data. The key feature of these models is the use of a pairwise embedding of an object and its neighbors (message passing) to predict the object's next state in the simulation.
This paper paper combines the two methods to create Relational Neural Expectation Maximization (R-NEM), allowing direct interaction at inference time between the latent variables that encode a scene. The encoder network from NEM can be seen as a recurrent network which takes one latent variable theta_k at time t and some input x to produce the next latent variable theta_k at time t+1. R-NEM adds a relational module which computes an embedding used as a third input to the recurrent encoder. Like previous relational models, this one uses a pairwise embedding of the object being updated (object k) and its neighbors. Unlike previous neural physics models, R-NEM uses a soft attention mechanism to determine which objects are neighbors and which are not. Also unlike previous neural models, this method does not require per-object supervision.
Experiments show that R-NEM learns compositional representations that support intuitive physics more effectively than ablative baselines. These experiments
show:
1) R-NEM reconstructs images more accurately than baselines (RNN/LSTM) and NEM (without object interaction).
2) R-NEM is trained with 4 objects per image. It does a bit worse at reconstructing images with 6-8 objects per image, but still performs better than baselines.
3) A version of R-NEM without neighborhood attention in the relation module matches the performance of R-NEM using 4 objects and performs worse than R-NEM at 6-8 objects.
4) R-NEM learns representations which factorize into one latent variable per object as measured by the Adjusted Rand Index, which compares NEM's pixel clustering to a ground truth clustering with one cluster per object.
5) Qualitative and quantitative results show that R-NEM can simulate 2d ball physics for many time steps more effectively than an RNN and while only suffering gradual divergence from the ground truth simulation.
Qualitative results show that the attentional mechanism attends to objects which are close to the context object together, acting like the heuristic neighborhood mechanism from previous work.
Follow up experiments extend the basic setup significantly. One experiment shows that R-NEM demonstrates object permanence by correctly tracking a collision when one of the objects is completely occluded. Another experiment applies the method to the Space Invaders Atari game, showing that it treats columns of aliens as entities. This representation aligns with the game's goal.
Strengths
---
The paper presents a clear, convincing, and well illustrated story.
Weaknesses
---
* RNN-EM BCE results are missing from the simulation plot (right of figure 4).
Minor comments/concerns:
* 2nd paragraph in section 4: Are parameters shared between these 3 MLPs (enc,emb,eff)? I guess not, but this is ambiguous.
* When R-NEM is tested against 6-8 balls is K set to the number of balls plus 1? How does performance vary with the number of objects?
* Previous methods report performance across simulations of a variety of physical phenomena (e.g., see "Visual Interaction Networks"). It seems that supervision isn't needed for bouncing ball physics, but I wonder if this is the case for other kinds of phenomena (e.g., springs in the VIN paper). Can this method eliminate the need for per-entity supervision in this domain?
* A follow up to the previous comment: Could a supervised baseline that uses per-entity state supervision and neural message passsing (like the NPE from Chang et. al.) be included?
* It's a bit hard to qualitatively judge the quality of the simulations without videos to look at. Could videos of simulations be uploaded (e.g., via anonymous google drive folder as in "Visual Interaction Networks")?
* This uses a neural message passing mechanism like those of Chang et. al. and Battaglia et. al. It would be nice to see a citation to neural message passing outside of the physics simulation domain (e.g. to "Neural Message Passing for Quantum Chemistry" by Gilmer et. al. in ICML17).
* Some work uses neighborhood attention coefficients for neural message passing. It would be nice to see a citation included.
    * See "Neighborhood Attention" in "One-Shot Imitation Learning" by Duan et. al. in NIPS17
    * Also see "Programmable Agents" by Denil et. al.
Final Evaluation
---
This paper clearly advances the body of work on neural intuitive physics by incorporating NEM entity representation to allow for less supervision. Alternatively, it adds a message passing mechanism to the NEM entity representation technique. These are moderately novel contributions and there are only minor weaknesses, so this is a clear accept.


Common-sense or background knowledge is required to understand natural language, but in most neural natural language understanding (NLU) systems, the requisite background knowledge is indirectly acquired from static corpora. We develop a new reading architecture for the dynamic integration of explicit background knowledge in NLU models. A new task-agnostic reading module provides refined word representations to a task-specific NLU architecture by processing background knowledge in the form of free-text statements, together with the taskspecific inputs. Strong performance on the tasks of document question answering (DQA) and recognizing textual entailment (RTE) demonstrate the effectiveness and flexibility of our approach. Analysis shows that our models learn to exploit knowledge selectively and in a semantically appropriate way.

The main emphasis of this paper is how to add background knowledge so as to improve the performance of NLU (specifically QA and NLI) systems. They adopt the sensible perspective that background knowledge might most easily be added by providing it in text format. However, in this paper, the way it is added is simply by updating word representations based on this extra text. This seems too simple to really be the right way to add background knowledge.
In practice, the biggest win of this paper turns out to be that you can get quite a lot of value by sharing contextualized word representations between all words with the same lemma (done by linguistic preprocessing; the paper never says exactly how, not even if you read the supplementary material). This seems a useful observation which it would be easy to apply everywhere and which shows fairly large utility from a bit of linguistically sensitive matching!  As the paper notes, this type of sharing is the main delta in this paper from simply using a standard deep LSTM (which the paper claims to not work on these data sets, though I'm not quite sure couldn't be made to work with more tuning).
pp. 6-7: The main thing of note seems to be that sharing of representations between words with the same lemma (which the tables refer to as "reading" is worth a lot (3.5-6.0%), in every case rather more than use of background knowledge (typically 0.3-1.5%). A note on the QA results: The QA results are certainly good enough to be in the range of "good systems", but none of the results really push the SOTA. The best SQuAD (devset) results are shown as several percent below the SOTA. In the table the TriviaQA results are shown as beating the SOTA, and that's fair wrt published work at the time of submission, but other submissions show that all of these results are below what you get by running the DrQA (Chen et al. 2017) system off-the-shelf on TriviaQA, so the real picture is perhaps similar to SQuAD, especially since DrQA is itself now considerably below the SOTA on SQUAD. Similar remarks perhaps apply to the NLI results.
p.7 In the additional NLI results, it is interesting and valuable to note that the lemmatization and knowledge help much more when amounts of data (and the covarying dimensionality of the word vectors) is much smaller, but the fact that the ideas of this paper have quite little (or even negative) effects when run on the full data with full word vectors on top of the ESIM model again draws into question whether enough value is being achieved from the world knowledge.
Biggest question:
 - Are word embeddings powerful enough as a form of memory to store the kind of relational facts that you are accessing as background knowledge?
Minor notes:
 - The paper was very well written/edited. The only real copyediting I noticed was in the conclusion: and be used ➔ and can be used; that rely on ➔ that relies on.
 - Should reference to (Manning et al. 1999) better be to (Manning et al. 2008) since the context here appears to be IR systems?
 - On p.3 above sec 3.1: What is u? Was that meant to be z?
 - On p.8, I'm a bit suspicious of the "Is additional knowledge used?" experiment which trains with knowledge and then tests without knowledge. It's not surprising that this mismatch might hurt performance, even if the knowledge provided no incremental value over what could be gained from standard word vectors alone.
 - In the supplementary material the paper notes that the numbers are from the best result from 3 runs. This seems to me a little less good experimental practice than reporting an average of k runs, preferably for k a bit bigger than 3.


We introduce contextual explanation networks (CENs)-a class of models that learn to predict by generating and leveraging intermediate explanations. CENs are deep networks that generate parameters for context-specific probabilistic graphical models which are further used for prediction and play the role of explanations. Contrary to the existing post-hoc model-explanation tools, CENs learn to predict and to explain jointly. Our approach offers two major advantages: (i) for each prediction, valid instance-specific explanations are generated with no computational overhead and (ii) prediction via explanation acts as a regularization and boosts performance in low-resource settings. We prove that local approximations to the decision boundary of our networks are consistent with the generated explanations. Our results on image and text classification and survival analysis tasks demonstrate that CENs are competitive with the state-of-the-art while offering additional insights behind each prediction, valuable for decision support.

the paper is clearly written; it works on a popular idea of combining graphical models and neural nets.
this work could benefit from differentiating more from previous literature.
one key component is interpretability, which comes from the use of graphical models.  the authors claim that the previous art directly integrate neural networks into the graphical models as components, which renders the models uninterpretable. however, it is unclear, following the same logic, why the proposed method has interpretability. after all, how to go from the context to the parameters of the graphical models is still uninterpretable. specifically, it is helpful to pinpoint what is special in this model that makes it interpretable, compared to works like Gao, Y., Archer, E. W., Paninski, L., & Cunningham, J. P. (2016). NIPS or Johnson, M., Duvenaud, D. K., Wiltschko, A., Adams, R. P., & Datta, S. R. (2016). NIPS. also, is there any methodological advancement essential to CENs?
the other idea is to go context specific. this idea has been present in language modeling, for example, amortized embedding models like M. Rudolph, F. Ruiz, S. Athey, and D. Blei (2017). NIPS and L. Liu, F. Ruiz, S. Athey, and D. Blei.  (2017). NIPS. application to medical data is interesting. but it could be helpful for the readers to understand if the idea in this work is fundamentally different from these previous ideas from amortized inference.
a final thing. a common challenge with composing graphical models and neural networks (in interpretable or uninterpretable ways) is that the neural networks will usually eat up all the representational power. the variance captured by graphical models becomes negligible. to this end, the power of graphical models for interpretability is limited. interpretability in this case is not much different from fitting only a neural network, taking the penultimate layer to the output as "context specific features" can claim that we are composing a linear model with a neural network, and the linear model is interpretable. it would be interesting to be clear about how the authors get around this issue.


Published as a conference paper at ICLR 2018 DORA THE EXPLORER: DIRECTED OUTREACHING REINFORCEMENT ACTION-SELECTION
Exploration is a fundamental aspect of Reinforcement Learning, typically implemented using stochastic action-selection. Exploration, however, can be more efficient if directed toward gaining new world knowledge. Visit-counters have been proven useful both in practice and in theory for directed exploration. However, a major limitation of counters is their locality. While there are a few model-based solutions to this shortcoming, a model-free approach is still missing. We propose E-values, a generalization of counters that can be used to evaluate the propagating exploratory value over state-action trajectories. We compare our approach to commonly used RL techniques, and show that using E-values improves learning and performance over traditional counters. We also show how our method can be implemented with function approximation to efficiently learn continuous MDPs. We demonstrate this by showing that our approach surpasses state of the art performance in the Freeway Atari 2600 game.

The paper proposes an approach to exploration based on initializing a value function to 1 everywhere, then letting the value decay back toward zero as the state space is explored. I like the idea a lot. I don't really like the paper, though. I'd really like to see a strong theoretical and/or empirical justification for it, and both are lacking. On the theoretical side, can a bound be proven for this approach, even in the tabular case? On the empirical side, there are more (and more recent!) testbeds that have come to define the field---the mountain car problem is just not sufficient to convincingly argue that the method scales and generalizes. My intuition is that such an approach ought to be effective, but I really want to see additional evidence. Given the availability of so many RL testbeds, I worry that it had been tried but failed.
Detailed comments:
"Where γ is" -> ", <newline> where γ is".
"The other alternative" -> "The alternative"?
"without learning a model (Mongillo et al., 2014).": Seems like an odd choice for a citation for model-free RL. Perhaps select
the paper that first used the term? Or an RL survey?
Right before Section 1.1, put a period after the Q-learning update equation.
"new states may" -> "new states, may".
"such approaches leads" -> "such approaches lead".
"they still fails" -> "they still fail".
"evaluated with respect only to its immediate outcome": Not so. Several of the cited papers use counters to determine which
states are "known" and then solve an MDP to direct exploration past immediate outcomes.
" exploration bonus(Little & Sommer, 2014)" -> " exploration bonus (Little & Sommer, 2014)".
"n a model-free settings." -> "n model-free settings.".
" Therefore, a satisfying approach for propagating directed exploration in model-free reinforcement learning is still missing. ": I think you should cite http://research.cs.rutgers.edu/~nouri/papers/nips08mre.pdf , which also combines a kind of counter
idea with function approximation to improve exploration.
"initializing E-values to 1": I like this idea. I wonder if one could prove bounds similar to the delayed Q-learning algorithm with
this approach. It is reminiscent of https://arxiv.org/pdf/1205.2606.pdf , which also drives exploration by beginning with an
overly optimistic estimate and letting the data (in a function approximation setting) decay the influence of this initialization.
"So after visited n times" -> "So after being visited n times".
"figure 1a" -> "Figure 1a". (And, in other places.)
"An important property of E-values is that it decreases over repetition" -> "An important property of E-values is that they decrease over repetition".
"t utilize counters, can" -> "t utilize counters can".
" hence we were interested a convergence measure": Multiple problems in this sentence, please fix.
Figure 2: How many states are in this environment? Some description is needed.
Figure 3: The labels in this figure (and all the figures) are absurdly small and, hence, unreadable.
"now turn to show that by using counters," -> "now turn to showing that, by using counters,".
Theorem 3.1: I'm not quite getting why we want to take a stochastic rule and make it deterministic. Note that standard PAC-MDP algorithms choose deterministically. It's not clear why we'd want to start from a stochastic rule.
" models(Bellemare" -> " models (Bellemare".
"Efficient memory-based learning for robot control": This reference is incomplete. (I'm skeptical that it represents the first use of this problem, but I can't check it.)
"Softmax exploration fail" -> "Softmax exploration fails".
"whom also analyzed" -> "who also analyzed".
"non-Markovity" -> "non-Markovianness"?


We study the problem of semantic code repair, which can be broadly defined as automatically fixing non-syntactic bugs in source code. The majority of past work in semantic code repair assumed access to unit tests against which candidate repairs could be validated. In contrast, the goal here is to develop a strong statistical model to accurately predict both bug locations and exact fixes without access to information about the intended correct behavior of the program. Achieving such a goal requires a robust contextual repair model, which we train on a large corpus of real-world source code that has been augmented with synthetically injected bugs. Our framework adopts a two-stage approach where first a large set of repair candidates are generated by rule-based processors, and then these candidates are scored by a statistical model using a novel neural network architecture which we refer to as Share, Specialize, and Compete. Specifically, the architecture (1) generates a shared encoding of the source code using an RNN over the abstract syntax tree, (2) scores each candidate repair using specialized network modules, and (3) then normalizes these scores together so they can compete against one another in comparable probability space. We evaluate our model on a real-world test set gathered from GitHub containing four common categories of bugs. Our model is able to predict the exact correct repair 41% of the time with a single guess, compared to 13% accuracy for an attentional sequence-to-sequence model.

This paper introduces a neural network architecture for fixing semantic bugs in code.  Focusing on four specific types of bugs, the proposed two-stage approach first generates a set of candidate repairs and then scores the repair candidates using a neural network trained on synthetically introduced bug/repair examples. Comparing to a prior sequence-to-sequence approach, the proposed approach achieved dominantly better accuracy on both synthetic and real bug datasets. On a real bug dataset constructed from GitHub commits, it was shown to outperform human.
I find the application of neural networks to the problem of code repair to be highly interesting. The proposed approach is highly specialized for the specific four types of bugs considered here and appears to be effective for fixing these specific bug types, especially in comparison to the sequence-to-sequence model based approach.  However, I was wondering whether limiting the output choices (based on the bug type)  is going a long way toward improving the performance compared to seq-2-seq, which does not utilize such output constraints.  What if we introduce the same type of constraints for the seq-2-seq model? For example, one can simply modifying the decoding process such that for locations that are not in the candidate set, the network simply  makes no change, and for candidate-repair locations, the output space is limited to the specific choices provided in the candidate set.  This will provide a more fair comparison between the different models.
Right now it is not clear how much of the observed performance gain is due to the use of these constraints on the output space.
Is there any control mechanism used to ensure that the real bug test set do not overlap with the training set? This is not clear to me.
I find the comparison result to human performance to be interesting and somewhat surprising. This seems quite impressive.  The presented example  where human makes a mistake but the algorithm is correct is informative and provides some potential explanation to this. But it also raises a question. The specific example snippet could be considered to be correct when placed in a different context.  Bugs are context sensitive artifacts. The setup of considering each function independently without any context seems like an inherent limitation in the types of bugs that this method could potentially address.  Some discussion on the limitation of the proposed method seems to be warranted.

Pro:
Interesting application
Impressive results on a difficult task
Nice discussion of results and informative examples
Clear presentation, easy to read.
Con:
The comparison to baseline seq-2-seq does not seem quite fair
The method appears to be highly specialized to the four bug types. It is not clear how generalizable it will be to more complex bugs, and to the real application scenarios where we are dealing with open world classification and there is not fixed set of possible bugs.


Published as a conference paper at ICLR 2018 NERVENET: LEARNING STRUCTURED POLICY WITH GRAPH NEURAL NETWORKS
We address the problem of learning structured policies for continuous control. In traditional reinforcement learning, policies of agents are learned by multi-layer perceptrons (MLPs) which take the concatenation of all observations from the environment as input for predicting actions. In this work, we propose NerveNet to explicitly model the structure of an agent, which naturally takes the form of a graph. Specifically, serving as the agent's policy network, NerveNet first propagates information over the structure of the agent and then predict actions for different parts of the agent. In the experiments, we first show that our NerveNet is comparable to state-of-the-art methods on standard MuJoCo environments. We further propose our customized reinforcement learning environments for benchmarking two types of structure transfer learning tasks, i.e., size and disability transfer, as well as multi-task learning. We demonstrate that policies learned by NerveNet are significantly more transferable and generalizable than policies learned by other models and are able to transfer even in a zero-shot setting.

The authors present an interesting application of Graph Neural Networks to learning policies for controlling "centipede" robots of different lengths. They leverage the non-parametric nature of graph neural networks to show that their approach is capable of transferring policies to different robots more quickly than other approaches. The significance of this work is in its application of GNNs to a potentially practical problem in the robotics domain. The paper suffers from some clarity/presentation issues that will need to be improved. Ultimately, the contribution of this paper is rather specific, yet the authors show the clear advantage of their technique for improved performance and transfer learning on some agent types within this domain.
Some comments:
- Significant: A brief statement of the paper's "contributions" is also needed; it is unclear at first glance what portions of the work are the authors' own contributions versus prior work, particularly in the section describing the GNN theory.
- Abstract: I take issue with the phrase "are significantly better than policies learned by other models", since this is not universally true. While there is a clear benefit to their technique for the centipede and snake models, the performance on the other agents is mostly comparable, rather than "significantly better"; this should be reflected in the abstract.
- Figure 1 is instructive, but another figure is needed to better illustrate the algorithm (including how the state of the world is mapped to the graph state h, how these "message" are passed between nodes, and how the final graph states are used to develop a policy). This would greatly help clarity, particularly for those who have not seen GNNs before, and would make the paper more self-contained and easier to follow. The figure could also include some annotated examples of the input spaces of the different joints, etc. Relatedly, Sec. 2.2.2 is rather difficult to follow because of the lack of a figure or concrete example (an example might help the reader understand the procedure without having to develop an intuition for GNNs).
- There is almost certainly a typo in Eq. (4), since it does not contain the aggregated message \bar{m}_u^t.
Smaller issues / typos:
- Abstract: please spell out spell out multi-layer perceptrons (MLP).
- Sec 2.2: "servers" should be "serves"
- "performance By" on page 4 is missing a "."
Pros:
- The paper presents an interesting application of GNNs to the space of reinforcement learning and clearly show the benefits of their approach for the specific task of transfer learning.
- To the best of my knowledge, the paper presents an original result and presents a good-faith effort to compare to existing, alternative systems (showing that they outperform on the tasks of interest).
Cons:
- The contributions of the paper should be more clearly stated (see comment above).
- The section describing their approach is not "self contained" and is difficult for an unlearned reader to follow.
- The problem the authors have chosen to tackle is perhaps a bit "specific", since the performance of their approach is only really shown to exceed the performance on agents, like centipedes or snakes, which have this "modular" quality.
I certainly hope the authors improve the quality of the theory section; the poor presentation here brings down the rest of the paper, which is otherwise an easy read.


ADAPTIVE DROPOUT WITH RADEMACHER COMPLEX- ITY REGULARIZATION
We propose a novel framework to adaptively adjust the dropout rates for the deep neural network based on a Rademacher complexity bound. The state-of-the-art deep learning algorithms impose dropout strategy to prevent feature co-adaptation. However, choosing the dropout rates remains an art of heuristics or relies on empirical grid-search over some hyperparameter space. In this work, we show the network Rademacher complexity is bounded by a function related to the dropout rate vectors and the weight coefficient matrices. Subsequently, we impose this bound as a regularizer and provide a theoretical justified way to trade-off between model complexity and representation power. Therefore, the dropout rates and the empirical loss are unified into the same objective function, which is then optimized using the block coordinate descent algorithm. We discover that the adaptively adjusted dropout rates converge to some interesting distributions that reveal meaningful patterns. Experiments on the task of image and document classification also show our method achieves better performance compared to the state-of-the-art dropout algorithms.

==Main comments
The authors connect dropout parameters to a bound of the Rademacher complexity (Rad) of the network. While it is great to see deep learning techniques inspired by learning theory, I think the paper makes too many leaps and the Rad story is ultimately unconvincing.  Perhaps it is better to start with the resulting regularizer, and the interesting direct optimization of dropout parameters. In its current form, the following leaps problematic and were not addressed in the paper:
1) Why is is adding Rad as a regularizer reasonable? Rad is usually hard to compute, and most useful for bounding the generalization error. It would be interesting if it also turns out to be a good regularizer, but the authors do not say why nor cite anything. Like the VC dimension, Rad itself depends on the model class, and cannot be directly optimized. Even if you can somehow optimize over the model class, these quantities give very loose bounds, and do not equal to generalization error. For example, I feel even just adding the actual generalization error bound is more natural. Would it make sense to just add Rad to the objective in this way for a linear model?
2) Why is it reasonable to go from a regularizer based on RC to a loose bound of Rad? The actual resulting regularizer turns out to be a weight penalty but this seems to be a rather loose bound that might not have too much to do with Rad anymore. There should be some analysis on how loose this bound is, and if this looseness matter at all.
The empirical results themselves seem reasonable, but the results are not actually better than simpler methods in the corresponding tasks, the interpretation is less confident. Afterall, it seems that the proposed method had several parameters that were turned, where the analogous parameters are not present in the competing methods. And the per unit dropout rates are themselves additional parameters, but are they actually good use of parameters?
==Minor comments
The optimization is perhaps also not quite right, since this requires taking the gradient of the dropout parameter in the original objective. While the authors point out that one can use the mean, but that is more problematic for the gradient than for normal forward predictions. The gradient used for regular learning is not based on the mean prediction, but rather the samples.
tiny columns surrounding figures are ugly and hard to read


DIFFUSION CONVOLUTIONAL RECURRENT NEURAL NETWORK: DATA-DRIVEN TRAFFIC FORECASTING
Spatiotemporal forecasting has various applications in neuroscience, climate and transportation domain. Traffic forecasting is one canonical example of such learning task. The task is challenging due to (1) complex spatial dependency on road networks, (2) non-linear temporal dynamics with changing road conditions and (3) inherent difficulty of long-term forecasting. To address these challenges, we propose to model the traffic flow as a diffusion process on a directed graph and introduce Diffusion Convolutional Recurrent Neural Network (DCRNN), a deep learning framework for traffic forecasting that incorporates both spatial and temporal dependency in the traffic flow. Specifically, DCRNN captures the spatial dependency using bidirectional random walks on the graph, and the temporal dependency using the encoder-decoder architecture with scheduled sampling. We evaluate the framework on two real-world large scale road network traffic datasets and observe consistent improvement of 12% -15% over state-of-the-art baselines.

The paper proposes to build a graph where the edge weight is defined using the road network distance which is shown to be more realistic than the Euclidean distance. The defined diffusion convolution operation is essentially conducting random walks over the road segment graph. To avoid the expensive matrix operation for the random walk, it empirically shows that K = 3 hops of the random walk can give a good performance. The outputs of the graph convolutionary operation are then fed into the sequence to sequence architecture with the GRU cell to model the temporal dependency. Experiments show that the proposed architecture can achieve good performance compared to classic time series baselines and several simplified variants of the proposed model.
Although the paper argues that several existing deep-learning based approaches may not be directly applied in the current setting either due to using Euclidean distance or undirected graph structure, the comparisons are not persuasive. For example, the approach in the paper "DeepTransport: Learning Spatial-Temporal Dependency for Traffic Condition Forecasting" also consider directed graph and a diffusion effect from 2 or 3 hops away in the neighboring subgraph of a target road segment.
Furthermore, the paper proposes to use two convolution components in Equation 2, each of which corresponds to out-degree and in-degree direction, respectively. This effectively increase the number of model parameters to learn. Compared to the existing spectral graph convolution approach, it is still not clear how its performance will be by using the same number of parameters. The experiments will be improved if it can compare with "Spatio-temporal graph convolutional neural network: A deep learning framework for traffic forecasting" using roughly the same number of parameters.


Data augmentation is a widely used technique in many machine learning tasks, such as image classification, to virtually enlarge the training dataset size and avoid overfitting. Traditional data augmentation techniques for image classification tasks create new samples from the original training data by, for example, flipping, distorting, adding a small amount of noise to, or cropping a patch from an original image. In this paper, we introduce a simple but surprisingly effective data augmentation technique for image classification tasks. With our technique, named SamplePairing, we synthesize a new sample from one image by overlaying another image randomly chosen from the training data (i.e., taking an average of two images for each pixel). By using two images randomly selected from the training set, we can generate N 2 new samples from N training samples. This simple data augmentation technique significantly improved classification accuracy for all the tested datasets; for example, the top-1 error rate was reduced from 33.5% to 29.0% for the ILSVRC 2012 dataset with GoogLeNet and from 8.22% to 6.93% in the CIFAR-10 dataset. We also show that our SamplePairing technique largely improved accuracy when the number of samples in the training set was very small. Therefore, our technique is more valuable for tasks with a limited amount of training data, such as medical imaging tasks.

The paper investigates a method of data augmentation for image classification, where two images from the training set are averaged together as input, but the label from only one image is used as a target.  Since this scheme is asymmetric and uses quite unrealistic input images, a training scheme is used where the technique is only enabled in the middle of training (not very beginning or end), and in an alternating on-off fashion.  This improves classification performance nicely on a variety of datasets.
This is a simple technique, and the paper is concise and to the point.  However, I would have liked to see a few additional comparisons.
First, this augmentation technique seems to have two components:  One is the mixing of inputs, but another is the effective dropping of labels from one of the two images in the pair.  Which of these are more important, and can they be separated?  What if some of the images' labels are changed at random, for half the images in a minibatch, for example?  This would have the effect of random label changes, but without the input mixing.  Likewise, what if both labels in the pair are used as targets (with 0.5 assigned to each in the softmax target)?  This would mix the images, but keep targets intact.
Second, the bottom of p.3 says that multiple training procedures were evaluated, but I'd be interested to see the results of some of these.  In particular, is it important to alternate enabling and disabling SamplePairing, or does it also work to mix samples with and without it in each minibatch (e.g. 3/4 of the minibatch with pairing augmentation, and 1/4 without it)?
I liked the experiment mixing images from within a restricted training set composed of a subset of the CIFAR images, compared to mixing these images with CIFAR training set images outside the restricted sample (p.5 and Fig 5).  This suggests to me, however, that it's possible the label manipulations may play an important role.  Or, is an explanation why this performs not as well that the network will train these mixing images to random targets (that of the training image in the pair), and never see this example again, whereas by using the training set alone, the mixing image is likely to be repeated with its correct label?  Some more discussion on this would be nice.
Overall, I think this is an interesting technique that appears to achieve nice results.  It could be investigated deeper at some key points.


In this paper, we study the problem of optimizing a two-layer artificial neural network that best fits a training dataset. We look at this problem in the setting where the number of parameters is greater than the number of sampled points. We show that for a wide class of differentiable activation functions (this class involves most nonlinear functions and excludes piecewise linear functions), we have that arbitrary first-order optimal solutions satisfy global optimality provided the hidden layer is non-singular. We essentially show that these non-singular hidden layer matrix satisfy a "good" property for these big class of activation functions. Techniques involved in proving this result inspire us to look at a new algorithmic, where in between two gradient step of hidden layer, we add a stochastic gradient descent (SGD) step of the output layer. In this new algorithmic framework, we extend our earlier result and show that for all finite iterations the hidden layer satisfies the"good" property mentioned earlier therefore partially explaining success of noisy gradient methods and addressing the issue of data independency of our earlier result. Both of these results are easily extended to hidden layers given by a flat matrix from that of a square matrix. Results are applicable even if network has more than one hidden layer provided all inner hidden layers are arbitrary, satisfy non-singularity, all activations are from the given class of differentiable functions and optimization is only with respect to the outermost hidden layer. Separately, we also study the smoothness properties of the objective function and show that it is actually Lipschitz smooth, i.e., its gradients do not change sharply. We use smoothness properties to guarantee asymptotic convergence of O(1/number of iterations) to a first-order optimal solution.

This paper aims to study some of the theoretical properties of the global optima of single-hidden layer neural networks and also the convergence to such a solution. I think there are some interesting arguments made in the paper e.g. Lemmas 4.1, 5.1, 5.2, and 5.3. However, as I started reading beyond intro I increasingly got the sense that this paper is somewhat incomplete e.g. while certain claims are made (abstract/intro) the theoretical justification are rather far from these claims. Of course there is a chance that I might be misunderstanding some things and happy to adjust my score based on the discussions here.
Detailed comments:
1) My main concern is that the abstract and intro claims things that are never proven (or even stated) in the rest of the paper
Example 1 from abstract:
“We show that for a wide class of differentiable activation functions (this class involved “almost” all functions which are not piecewise linear), we have that first-order optimal solutions satisfy global optimality provided the hidden layer is non-singular.”
This is certainly not proven and in fact not formally stated anywhere in the paper. Closest result to this is Lemma 4.1 however, because the optimal solution is data dependent this lemma can not be used to conclude this.
Example 2 from intro when comparing with other results on page 2:
The authors essentially state that they have less restrictive assumptions in the form of the network or assumptions on the data (e.g. do not require Gaussianity). However as explained above the final conclusions are also significantly weaker than this prior literature so it’s a bit of apples vs oranges comparison.
2) Page 2 minor typos
We study training problem -->we study the training problem
In the regime training objective--> in the regime the training objective
3) the basic idea argument and derivative calculations in section 3 is identical to section 4 of Soltan...et al
4) Lemma 4.1 is nice, well done! That being said it does not seem easy to make it (1) quantifiable (2) apply to all W. It would also be nice to compare with Soudry et. al.
5) Argument on top of page 6 is incorrect as the global optima is data dependent and hence lemma 4.1 (which is for a fixed matrix) does not apply
6) Section 5 on page 6. Again the stated conclusion here that the iterates do not lead to singular W is much weaker than the claims made early on.

7) I haven’t had time yet to verify correctness of Lemmas 5.1, 5.2, and Lemma 5.3 in detail but if this holds is a neat argument to side step invertibility w.r.t. W, Nicely done!
8) What is the difference between Lemma 5.4 and Lemma 6.12 of Soltan...et al
9) Theorem 5.9. Given that the arguments in this paper do not show asymptotic convergence to a point where gradient vanishes and W is invertible why is the proposed algorithm better than a simple approach in which gradient descent is applied but a small amount of independent Gaussian noise is injected in every iteration over W. By adjusting the noise variance across time one can ensure a result of the kind in Theorem 5.9 (Of course in the absence of a quantifiable version of Lemma 4.1 which can apply to all W that result will also suffer from the same issues).


Sepsis is a life-threatening complication from infection and a leading cause of mortality in hospitals. While early detection of sepsis improves patient outcomes, there is little consensus on exact treatment guidelines, and treating septic patients remains an open problem. In this work we present a new deep reinforcement learning method that we use to learn optimal personalized treatment policies for septic patients. We model patient continuous-valued physiological time series using multi-output Gaussian processes, a probabilistic model that easily handles missing values and irregularly spaced observation times while maintaining estimates of uncertainty. The Gaussian process is directly tied to a deep recurrent Q-network that learns clinically interpretable treatment policies, and both models are learned together end-to-end. We evaluate our approach on a heterogeneous dataset of septic spanning 15 months from our university health system, and find that our learned policy could reduce patient mortality by as much as 8.2% from an overall baseline mortality rate of 13.3%. Our algorithm could be used to make treatment recommendations to physicians as part of a decision support tool, and the framework readily applies to other reinforcement learning problems that rely on sparsely sampled and frequently missing multivariate time series data.

This paper presents an important application of modern deep reinforcement learning (RL) methods to learning optimal treatments for sepsis from past patient encounters. From a methods standpoint, it offers nothing new but does synthesize best practice deep RL methods with a differentiable multi-task Gaussian Process (GP) input layer. This means that the proposed architecture can directly handle irregular sampling and missing values without a separate resampling step and can be trained end-to-end to optimize reward -- patient survival -- without a separate ad hoc preprocessing step. The experiments are thorough and the results promising. Overall, strong application work, which I appreciate, but with several flaws that I'd like the authors to address, if possible, during the review period. I'm perfectly willing to raise my score at least one point if my major concerns are addressed.
QUALITY
Although the core idea is derivative, the work is executed pretty well. Pros (+) and cons (-) are listed below:
+ discussion of the sepsis application is very strong. I especially appreciated the qualitative analysis of the individual case shown in Figure 4. While only a single anecdote, it provides insight into how the model might yield clinical insights at the bedside.
+ thorough comparison of competing baselines and clear variants -- though it would be cool to apply offline policy evaluation (OPE) to some of the standard clinical approaches, e.g., EGDT, discussed in the introduction.
- "uncertainty" is one of the supposed benefits of the MTGP layer, but it was not at all clear how it was used in practice, other than -- perhaps -- as a regularizer during training, similar to data augmentation.
- uses offline policy evaluation "off-the-shelf" and does not address or speculate the potential pitfalls or dangers of doing so. See "Note on Offline Policy Evaluation" below.
- although I like the anecdote, it tells us very little about the overall policy. The authors might consider some coarse statistical analyses, similar to Figure 3 in Raghu, et al. (though I'm sure you can come up with more and better analyses!).
- there are some interesting patterns in Table 1 that the authors do not discuss, such as the fact that adding the MGP layer appears to reduce expected mortality more (on average) than adding recurrences. Why might this be (my guess is data augmentation)?
CLARITY
Paper is well-written, for the most part. I have some nitpicks about the writing, but in general, it's not a burden to read.
+ core ideas and challenges of the application are communicated clearly
- the authors did not detail how they chose their hyperparameters (number of layers, size of layers, whether to use dropout, etc.). This is critical for fully assessing the import of the empirical results.
- the text in the figures are virtually impossible to read (too small)
- the image quality in the figures is pretty bad (and some appear to be weirdly stretched or distorted)
- I prefer the X-axis labels that Raghu uses in their Figure 4 (with clinically interpretable increments) over the generic +1, +2, etc., labels used in Figure 3 here
Some nitpicks on the writing
* too much passive voice. Example: third paragraph in introduction ("Despite the promising results of EGDT, concerns arose."). Avoid passive voice whenever possible.
* page 3, sec. 2.2 doesn't flow well. You bounce back and forth between discussion of the Markov assumption and full vs. partial observability. Try to focus on one concept at a time (and the solution offered by a proposed approach). Note that RNNs do NOT relax the Markov assumption -- they simply do an end run around it by using distributed latent representations.
ORIGINALITY
This work scores relatively low in originality. It really just combines ideas from two MLHC 2017 papers [1][2]. One could read those two papers and immediately conclude this paper's findings (the GP helps; RL helps; GP + RL is the best). This paper adds few (if any) new insights.
One way to address this would be to discuss in greater detail some potential explanations for why their results are stronger than those in Raghu and why the MTGP models outperform their simpler counterparts. Perhaps they could run some experiments to measure performance as a function of the number of MC samples (if perhaps grows with the number of samples, then it suggests that maybe it's largely a data augmentation effect).
SIGNIFICANCE
This paper's primary significance is that it provides further evidence that RL could be applied successfully to clinical data and problems, in particular sepsis treatment. However, this gets undersold (unsurprising, given the ML community's disdain for replication studies). It is also noteworthy that the MTGP gives such a large boost in performance for a relatively modest data set -- this property is worth exploring further, since clinical data are often small. However, again, this gets undersold.
One recommendation I would make is that the authors directly compare the results in this paper with those in Raghu and to point out, in particular, the confirmatory results. Interestingly, the shapes of the action vs. mortality rate plots (Figure 4 in Raghu, Figure 3 here) are quite similar -- that's not precisely replication, but it's comforting.
NOTE ON OFFLINE POLICY EVALUATION
This work has the same flaw that Raghu, et al., has -- neither justifies the use of offline policy evaluation. Both simply apply Jiang, et al.'s doubly robust approach [3] "off the shelf" without commenting on its accuracy in practice or discussing potential pitfalls (neither even considers [4] which seems to be superior in practice, especially with limited data). As far as I can tell (I'm not an RL expert), the DR approach carries stronger consistency guarantees and reduced variance but is still only as good the data it is trained on, and clinical data is known to have significant bias, particularly with respect to treatment, where clinicians are often following formulaic guidelines. Can we trust the mortality estimates in Table 1? Why or why not? Why shouldn't I think that RL is basically guaranteed to outperform non-RL approaches under an evaluation that is itself an RL model learned from the same training data!
While I'm willing to accept that this is the best we can do in this setting (we can't just try the learned policy on new patients!), I think this paper (and similar works, like Raghu, et al.) *must* provide a sober and critical discussion of its results, rather than simply applaud itself for getting the best score among competing approaches.
REFERENCES
[1] Raghu, et al. "Continuous State-Space Models for Optimal Sepsis Treatment - a Deep Reinforcement Learning Approach." MLHC 2017.
[2] Futoma, et al. "An Improved Multi-Output Gaussian Process RNN with Real-Time Validation for Early Sepsis Detection." MLHC 2017.
[3] Jiang, et al. "Doubly robust off-policy value evaluation for reinforcement learning." ICML 2016.
[4] Thompson and Brunskill. "Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning." ICML 2016.


Published as a conference paper at ICLR 2018 DEEP LEARNING FOR PHYSICAL PROCESSES: INCORPORATING PRIOR SCIENTIFIC KNOWLEDGE
We consider the use of Deep Learning methods for modeling complex phenomena like those occurring in natural physical processes. With the large amount of data gathered on these phenomena the data intensive paradigm could begin to challenge more traditional approaches elaborated over the years in fields like maths or physics. However, despite considerable successes in a variety of application domains, the machine learning field is not yet ready to handle the level of complexity required by such problems. Using an example application, namely Sea Surface Temperature Prediction, we show how general background knowledge gained from the physics could be used as a guideline for designing efficient Deep Learning models. In order to motivate the approach and to assess its generality we demonstrate a formal link between the solution of a class of differential equations underlying a large family of physical phenomena and the proposed model. Experiments and comparison with series of baselines including a state of the art numerical approach is then provided.

The paper ‘Deep learning for Physical Process: incorporating prior physical knowledge’ proposes
to question the use of data-intensive strategies such as deep learning in solving physical
inverse problems that are traditionally solved through assimilation strategies. They notably show
how physical priors on a given phenomenon can be incorporated in the learning process and propose
an application on the problem of estimating sea surface temperature directly from a given
collection of satellite images.
All in all the paper is very clear and interesting. The results obtained on the considered problem
are clearly of great interest, especially when compared to state-of-the-art assimilation strategies
such as the one of Béréziat. While the learning architecture is not original in itself, it is
shown that a proper physical regularization greatly improves the performance. For these reasons I
believe the paper has sufficient merits to be published at ICLR. That being said, I believe that
some discussions could strengthen the paper:
 - Most classical variational assimilation schemes are stochastic in nature, notably by incorporating
uncertainties in the observation or physical evolution models. It is still unclear how those uncertainties
can be integrated in the model;
 - Assimilation methods are usually independent of the type of data at hand. It is not clear how the model
learnt on one particular type of data transpose to other data sequences. Notably, the question of transfer
and generalization is of high relevance here. Does the learnt model performs well on other dataset (for instance,
acquired on a different region or at a distant time). I believe this type of issue has to be examinated
for this type of approach to be widely use in inverse physical problems.


Recurrent Neural Networks (RNNs) are designed to handle sequential data but suffer from vanishing or exploding gradients. Recent work on Unitary Recurrent Neural Networks (uRNNs) have been used to address this issue and in some cases, exceed the capabilities of Long Short-Term Memory networks (LSTMs). We propose a simpler and novel update scheme to maintain orthogonal recurrent weight matrices without using complex valued matrices. This is done by parametrizing with a skew-symmetric matrix using the Cayley transform. Such a parametrization is unable to represent matrices with negative one eigenvalues, but this limitation is overcome by scaling the recurrent weight matrix by a diagonal matrix consisting of ones and negative ones. The proposed training scheme involves a straightforward gradient calculation and update step. In several experiments, the proposed scaled Cayley orthogonal recurrent neural network (scoRNN) achieves superior results with fewer trainable parameters than other unitary RNNs.

This manuscript introduce a scheme for learning the recurrent parameter matrix in a neural network that uses the Cayley transform and a scaling weight matrix. This scheme leads to good performance on sequential data tasks and requires fewer parameters than other techniques
Comments:
-- It’s not clear to me how D is determined for each test. Given the definition in Theorem 3.1 it seems like you would have to have some knowledge of how many eigenvalues in W you expect to be close to -1.
-- For the copying and adding problem test cases, it might be useful to clarify or cite something clarifying that the failure mode RNNs run into with temporal ordering problems is an exploding gradient, rather than any other pathological training condition, just to make it clear why these experiments are relevant.
-- The ylabel in Figure 1 is “Test Loss” which I didn’t see defined. Is this test loss the cross entropy? If so, I think it would be more effective to label the plot with that.
-- The plots in figure 1 and 2 have different colors to represent the same set of techniques. I would suggest keeping a  consistent color scheme
-- It looks like in Figure 1 the scoRNN is outperformed by the uRNN in the long run in spite of the scoRNN convergence being smoother, which should be clarified.
-- It looks like in Figure 2 the scoRNN is outperformed by the LSTM across the board, which should be clarified.
-- How is test set accuracy defined in section 5.3? Classifying digits? Recreating digits?
-- When discussing table 1, the manuscript mentions scoRNN and Restricted-capacity uRNN have similar performance for 16k parameters and then state that scoRNN has the best test accuracy at 96.2%. However, there is no example for restricted-capacity uRNN with 69k parameters to show that the performance of restricted-capacity uRNN doesn't also increase similarly with more parameters.
-- Overall it’s unclear to me how to completely determine the benefit of this technique over the others because, for each of the tests, different techniques may have superior performance. For instance, LSTM performs best in 5.2 and in 5.3 for the MNIST test accuracy. scoRNN and Restricted-capacity uRNN perform similarly for permuted MNIST Test Accuracy in 5.3. Finally, scoRNN seems to far outperform the other techniques in table 2 on the TIMIT speech dataset. I don’t understand the significance of each test and why the relative performance of the techniques vary from one to the other.
-- For example, the manuscript seems to be making the case that the scoRNN gradients are more stable than those of a uRNN, but all of the results are presented in terms of network accuracy and not gradient stability. You can sort of see that generally the convergence is more gradual for the scoRNN than the uRNN from the training graphs but it'd be nice if there was an actual comparison of the stability of the gradients during training (as in Figure 4 of the Arjovsky 2016 paper being compared to for instance) just to make it really clear.


We improve previous end-to-end differentiable neural networks (NNs) with fast weight memories. A gate mechanism updates fast weights at every time step of a sequence through two separate outer-product-based matrices generated by slow parts of the net. The system is trained on a complex sequence to sequence variation of the Associative Retrieval Problem with roughly 70 times more temporal memory (i.e. time-varying variables) than similar-sized standard recurrent NNs (RNNs). In terms of accuracy and number of parameters, our architecture outperforms a variety of RNNs, including Long Short-Term Memory, Hypernetworks, and related fast weight architectures.

Summary
The paper proposes a neural network architecture for associative retrieval based on fast weights with context-dependent gated updates. The architecture consists of a ‘slow’ network which provides weight updates for the ‘fast’ network which outputs the predictions of the system. The experiments show that the architecture outperforms a couple of related models on an associative retrieval problem.
Quality
The authors evaluate their architecture on an associative retrieval task which is similar to the variable assignment task used in Danihelka et al. (2016). The difference with the original task seems to be that the network is also trained to predict a ‘blank’ symbol which indicates that no prediction has been made. While this task is artificial, it does make sense in the context of what the authors want to show. The fact that the authors compare their results with three sensible baselines and perform some form of hyper-parameter search for all of the models, adds to the quality of the experiment. It is somewhat unfortunate that the paper doesn’t give more detail about the precise hyper-parameters involved and that there is no comparison with the associative LSTM from Danihelka et al. Did these hyper-parameters also include the sizes of the models? Otherwise it’s not very clear to me why the numbers of parameters are so much higher for the baseline models. While I think that this experiment is well done, it is unfortunate that it is the only experiment the authors carried out and the paper would be more impactful if there would have been results for a wider variety of tasks. It is commendable that the authors also discuss the memory requirements and increased wall clock time of the model.
Clarity
I found the paper hard to read at times and it is often not very clear what the most important differences are between the proposed methods and earlier ones in the literature. I’m not saying those differences aren’t there, but the paper simply didn’t emphasize them very well and I had to reread the paper from Ba et al. (2016) to get the full picture.
Originality/Significance
While the architecture is new, it is based on a combination of previous ideas about fast weights, hypernetworks and activation gating and I’d say that the novelty of the approach is average. The architecture does seem to work well on the associative retrieval task, but it is not clear yet if this will also be true for other types of tasks. Until that has been shown, the impact of this paper seems somewhat limited to me.
Pros
Experiments seem well done.
Good baselines.
Good results.
Cons
Hard to extract the most important changes from the text.
Only a single synthetic task is reported.


Learning a better representation with neural networks is a challenging problem, which has been tackled from different perspectives in the past few years. In this work, we focus on learning a representation that would be useful in a clustering task. We introduce two novel loss components that substantially improve the quality of produced clusters, are simple to apply to arbitrary models and cost functions, and do not require a complicated training procedure. We perform an extensive set of experiments, supervised and unsupervised, and evaluate the proposed loss components on two most common types of models, Recurrent Neural Networks and Convolutional Neural Networks, showing that the approach we propose consistently improves the quality of KMeans clustering in terms of mutual information scores and outperforms previously proposed methods.

Summary
This paper proposes two regularizers that are intended to make the
representations learned in the penultimate layer of a classifier more conforming
to inherent structure in the data, rather than just the class structure enforced
by the classifier. One regularizer encourages the weights feeding into the
penultimate layer to be dissimilar and the other encourages the activations
across samples (even if they belong to the same class) to be dissimilar.
Pros
- The proposed regularizers are able to separate out the classes inherent in the
  data, even if this information is not provided through class labels. This is
validated on several datasets using visualizations as well as quantitative
metrics based on mutual information.
Cons
- It is not explained why it makes sense to first convert the weight vectors
  into probability distributions by applying the softmax function, and then
measuring distances using KL divergence between the probability distributions.
It should be explained more clearly if there is there a natural interpretation
of the weight vectors as probability distributions. Otherwise it is not obvious
why the distance between the weight vectors is measured the way it is.
- Similarly, the ReLU activations are also first converted into probability
  distributions by applying a softmax. It should be explained why the model does
this, as opposed to simply using dot products to measure similarity.
- The model is not compared to simpler alternatives such as adding an
  orthogonality regularization on the weights, i.e., computing W^TW and making
the diagonals close to 1 and all other terms 0. Similar regularizers can be
applied for activation vectors as well.
- The objective of this paper seems to be to produce representations that are
  easy to separate into clusters. This topic has a wealth of previous work. Of
particular relevance are methods such as t-SNE [1], parametric t-SNE [2], and
DEC [3]. The losses introduced in this paper are fairly straight-forward.
Therefore it would be good to compare to these baselines to show that a simple
loss function is sufficient to achieve the objective.
- Disentangling usually refers to disentangling factors of variation, for
  example, lighting, pose, and object identity which affect the appearance of a
data point. This is different from separability, which is the property of a
representation that makes the presence of clusters evident. This paper seems to
be about learning separable representations, whereas the title suggests that it
is about disentangled ones.
Quality
The design choices made in the paper (such as the choice of distance function)
is not well explained. Also, given that the modifications introduced are quite
simple, it can be improved by doing more thorough comparisons to other
baselines.
Clarity
The paper is easy to follow.
Originality
The novel aspect of the paper is the way distance is measured by converting the
weights (and activations) to probability distributions and using KL divergence
to measure distance. However, it is not explained what motivated this choice.
Significance
The objective of this model is to produce representations that are separable, which
is of general interest. However, given the wealth of previous work done in
clustering, this paper would only be impactful if it compares to other hard
baselines and shows clear advantages.
[1] van der Maaten, Laurens and Hinton, Geoffrey. Visualizing
data using t-SNE. JMLR, 2008.
[2] van der Maaten, Laurens. Learning a parametric embedding
by preserving local structure. In International Conference
on Artificial Intelligence and Statistics, 2009.
[3] Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for
clustering analysis. ICML 2016.


We propose a neural clustering model that jointly learns both latent features and how they cluster. Unlike similar methods our model does not require a predefined number of clusters. Using a supervised approach, we agglomerate latent features towards randomly sampled targets within the same space whilst progressively removing the targets until we are left with only targets which represent cluster centroids. To show the behavior of our model across different modalities we apply our model on both text and image data and achieve very competitive results on MNIST against methods that require a predefined number of clusters. We also provide results against baseline models for fashion-MNIST, the 20 newsgroups dataset, and a Twitter dataset we ourselves create.
1 .

This paper proposes a neural clustering model following the "Noise as Target" technique. Combining with an reconstruction objective and "delete-and-copy" trick, it is able to cluster the data points into different groups and is shown to give competitive results on different benchmarks.
It is nice that the authors tried to extend the "noise as target" to the clustering problem, and proposed the simple "delete-and-copy" technique to group different data points into clusters. Even tough a little bit ad-hoc, it seems promising based on the experiment results. However, it is unclear to me why it is necessary to have the optimal matching here and why the simple nearest target would not work. After all, the cluster membership is found based on the nearest target in the test stage.
Also, the authors should provide more detailed description regarding the scheduling of the alpha and lambda values during training, and how sensitive it is to the final clustering performance. The authors cited the no requirement of "a predefined number of clusters" as one of the contributions, but the tuning of alpha seems more concerning.
I like the authors experimented with different benchmarks, but lack of comparisons with existing deep clustering techniques is definitely a weakness. The only baseline comparison provided is the k-means clustering, but the comparisons were somewhat unfair. For all the text datasets, there were no comparisons with k-means on the features learned from the auto-encoders or clusterings learned from similar number of clusters. The comparisons for the Twitter dataset were even based on character-level with word-level. It is more convincing to show the superiority of the proposed method than existing ones on the same ground.
Some other issues regarding quantitative results:
- In Table 1, there are 152 clusters for 10-d latent space after convergence, but there are 61 clusters for 10-d latent space in Table 2 for the same MNIST dataset. Are they based on different alpha and lambda values?
- Why does NATAC perform much better than NATAC-k? Would NATAC-k need a different number of clusters than the one from NATAC? The number of centroids learned from NATAC may not be good for k-means clustering.
- It seems like the performance of AE-k is increasing with increase of dimensionality of latent space for Fashion-MNIST. Would AE-k beat NATAC with a different dimensionality of latent space and k?


HIERARCHICAL SUBTASK DISCOVERY WITH NON-NEGATIVE MATRIX FACTORIZATION
Hierarchical reinforcement learning methods offer a powerful means of planning flexible behavior in complicated domains. However, learning an appropriate hierarchical decomposition of a domain into subtasks remains a substantial challenge. We present a novel algorithm for subtask discovery, based on the recently introduced multitask linearly-solvable Markov decision process (MLMDP) framework. The MLMDP can perform never-before-seen tasks by representing them as a linear combination of a previously learned basis set of tasks. In this setting, the subtask discovery problem can naturally be posed as finding an optimal low-rank approximation of the set of tasks the agent will face in a domain. We use non-negative matrix factorization to discover this minimal basis set of tasks, and show that the technique learns intuitive decompositions in a variety of domains. Our method has several qualitatively desirable features: it is not limited to learning subtasks with single goal states, instead learning distributed patterns of preferred states; it learns qualitatively different hierarchical decompositions in the same domain depending on the ensemble of tasks the agent will face; and it may be straightforwardly iterated to obtain deeper hierarchical decompositions.

The present paper extends a previous work by Saxe et al (2017) that considered multitask learning in RL and proposed a hierarchical learner based on concurrent execution of many actions in parallel. That framework made heavy use of the framework of linearly solvable Markov decision process (LMDP) proposed by Todorov, which allows for closed form solutions of the control due to the linearity of the Bellman optimality equations. The simple form of the solutions allow them to be composed naturally, and to form deep hierarchies through iteration. The framework is restricted to domains where the transitions are fixed but the rewards may change between tasks.  A key role is played in the formalism by the so-called ‘free dynamics’ that serves to regularize the action selected.
The present paper goes beyond Saxe et al. in several ways. First, it renders the process of deep hierarchy formation automatic, by letting the algorithm determine the new passive dynamics at each stage, as well as the subtasks themselves. The process of subtask discovery is done via non-negative matrix factorization, whereby the matrix of desirability functions, determined by the solution of the LMDPs with exponentiated reward. Since the matrix is non-negative, the authors propose a non-negative factorization into a product of non-negative low rank matrices that capture its structure at a more abstract level of detail. A family of optimization criteria for this process are suggested, based on a subclass if Bregman divergences. Interestingly, the subtasks discovered correspond to distributions over states, rather than single states as in many previous approaches. The authors present several demonstrations of the intuitive decomposition achieved. A nice feature of the present framework is that a fully autonomous scheme (given some assumed parameter values) is demonstrated for constructing the full hierarchical decomposition.
I found this to be an interesting approach to hierarchical multitask learning, augmenting a previous approach with several steps leading to increased autonomy, an essential agent for any learning agent. Both the intuition behind the construction and the application to test problem reveal novel insight. The utilization on the analytic framework of LMDP facilitates understanding and efficient algorithms.
I would appreciate the authors’ clarification of several issues. First, the LMDP does not seem to be completely general, so I would appreciate a description of the limitations of this framework. The description of the elbow-joint behavior around eq. (4) was not clear to me, please expand. The authors do not state any direct or indirect extensions – please do so. Please specify how many free parameters the algorithm requires, and what is a reasonable way to select them. Finally, it would be instructive to understand where the algorithm may fail.


Convolutional neural networks and recurrent neural networks are designed with network structures well suited to the nature of spacial and sequential data respectively. However, the structure of standard feed-forward neural networks (FNNs) is simply a stack of fully connected layers, regardless of the feature correlations in data. In addition, the number of layers and the number of neurons are manually tuned on validation data, which is time-consuming and may lead to suboptimal networks. In this paper, we propose an unsupervised structure learning method for learning parsimonious deep FNNs. Our method determines the number of layers, the number of neurons at each layer, and the sparse connectivity between adjacent layers automatically from data. The resulting models are called Backbone-Skippath Neural Networks (BSNNs). Experiments on 17 tasks show that, in comparison with FNNs, BSNNs can achieve better or comparable classification performance with much fewer parameters. The interpretability of BSNNs is also shown to be better than that of FNNs.

The main strengths of the paper are the supporting experimental results in comparison to plain feed-forward networks (FNNs).  The proposed method is focused on discovering sparse neural networks.  The experiments show that sparsity is achieved and still the discovered sparse networks have comparable or better performance compared to dense networks.
The main weakness of the paper is lack of cohesion in contributions and difficulty in delineating the scope of their proposed approach.
Below are some suggestions for improving the paper:
Can you enumerate the paper’s contributions and specify the scope of this work?  Where is this method most applicable and where is it not applicable?
Why is the paper focused on these specific contributions?  What problem does this particular set of contributions solve that is not solvable by the baselines?  There needs to be a cohesive story that puts the elements together.  For example, you explain how the algorithm for creating the backbone can use unsupervised data.  On the other hand, to distinguish this work from the baselines you mention that this work is the first to apply the method to supervised learning problems.
The motivation section in the beginning of the paper motivates using the backbone structure to get a sparse network.  However, it does not adequately motivate the skip-path connections or applications of the method to supervised tasks.
Is this work extending the applicability of baselines to new types of problems?  Or is this work focused on improving the performance of existing methods?  Answers to these questions can automatically determine suitable experiments to run as well.  It's not clear if Pruned FNNs are the most suitable baseline for evaluating the results.  Can your work be compared experimentally with any of the constructive methods from the related work section?  If not, why?
When contrasting this work with existing approaches, can you explain how existing work builds toward the same solution that you are focusing on?  It would be more informative to explain how the baselines contribute to the solution instead of just citing them and highlighting their differences.
Regarding the experimental results, is there any insight on why the dense networks are falling short?  For example, if it is due to overfitting, is there a correlation between performance and size of FNNs?  Do you observe a similar performance vs FNNs in existing methods?  Whether this good performance is due to your contributions or due to effectiveness of the baseline algorithm, proper analysis and discussion is required and counts as useful research contribution.


Deep networks have shown great performance in classification tasks. However, the parameters learned by the classifier networks usually discard stylistic information of the input, in favour of information strictly relevant to classification. We introduce a network that has the capacity to do both classification and reconstruction by adding a "style memory" to the output layer of the network. We also show how to train such a neural network as a deep multi-layer autoencoder, jointly minimizing both classification and reconstruction losses. The generative capacity of our network demonstrates that the combination of style-memory neurons with the classifier neurons yield good reconstructions of the inputs when the classification is correct. We further investigate the nature of the style memory, and how it relates to composing digits and letters.

This paper proposes to train a classifier neural network not just to classifier, but also to reconstruct a representation of its input, in order to factorize the class information from the appearance (or "style" as used in this paper). This is done by first using unsupervised pretraining and then fine-tuning using a weighted combination of the regular multinomial NLL loss and a reconstruction loss at the last hidden layer. Experiments on MNIST are provided to analyse what this approach learns.
Unfortunately, I fail to see a significantly valuable contribution from this work. First, the paper could do a better job at motivating the problem being addressed. Why is it important to separate class from style? Should it allow better classification performance? If so, it's never measured in this work. If that's not the motivation, then what is it?
Second, all experiments were conducted on the MNIST dataset. In 2017, most would expect experiments on at least one other, more complex dataset, to trust any claims on a method.
Finally, the results are not particularly impressive. I don't find the reconstructions demonstrated particularly compelling (they are generally pretty different from the original input). Also, that the "style" representation contain less (and I'd say slightly less, in Figure 7 b and d, we still see a lot of same class nearest neighbors) is not exactly a surprising result. And the results of figure 9, showing poor reconstructions when changing the class representation essentially demonstrates that the method isn't able to factorize class and style successfully. The interpolation results of Figure 11 are also underwhelming, though possibly mostly because the reconstructions are in general not great. But most importantly, none of these results are measured in a quantitative way: they are all qualitative, and thus subjective.
For all these reasons, I'm afraid I must recommend this paper be rejected.


Published as a conference paper at ICLR 2018 VARIATIONAL MESSAGE PASSING WITH STRUCTURED INFERENCE NETWORKS
Recent efforts on combining deep models with probabilistic graphical models are promising in providing flexible models that are also easy to interpret. We propose a variational message-passing algorithm for variational inference in such models. We make three contributions. First, we propose structured inference networks that incorporate the structure of the graphical model in the inference network of variational auto-encoders (VAE). Second, we establish conditions under which such inference networks enable fast amortized inference similar to VAE. Finally, we derive a variational message passing algorithm to perform efficient naturalgradient inference while retaining the efficiency of the amortized inference. By simultaneously enabling structured, amortized, and natural-gradient inference for deep structured models, our method simplifies and generalizes existing methods.

This paper presents a variational inference algorithm for models that contain
deep neural network components and probabilistic graphical model (PGM)
components.
The algorithm implements natural-gradient message-passing where the messages
automatically reduce to stochastic gradients for the non-conjugate neural
network components. The authors demonstrate the algorithm on a Gaussian mixture
model and linear dynamical system where they show that the proposed algorithm
outperforms previous algorithms. Overall, I think that the paper proposes some
interesting ideas, however, in its current form I do not think that the novelty
of the contributions are clearly presented and that they are not thoroughly
evaluated in the experiments.
The authors propose a new variational inference algorithm that handles models
with deep neural networks and PGM components. However, it appears that the
authors rely heavily on the work of (Khan & Lin, 2017) that actually provides
the algorithm. As far as I can tell this paper fits inference networks into
the algorithm proposed in (Khan & Lin, 2017) which boils down to i) using an
inference network to generate potentials for a conditionally-conjugate
distribution and ii) introducing new PGM parameters to decouple the inference
network from the model parameters. These ideas are a clever solution to work
inference networks into the message-passing algorithm of (Khan & Lin, 2017),
but I think the authors may be overselling these ideas as a brand new algorithm.
I think if the authors sold the paper as an alternative to (Johnson, et al., 2016)
that doesn't suffer from the implicit gradient problem the paper would fit into
the existing literature better.
Another concern that I have is that there are a lot of conditiona-conjugacy
assumptions baked into the algorithm that the authors only mention at the end
of the presentation of their algorithm. Additionally, the authors briefly state
that they can handle non-conjugate distributions in the model by just using
conjugate distributions in the variational approximation. Though one could do
this, the authors do not adequately show that one should, or that one can do this
without suffering a lot of error in the posterior approximation. I think that
without an experiment the small section on non-conjugacy should be removed.
Finally, I found the experimental evaluation to not thoroughly demonstrate the
advantages and disadvantages of the proposed algorithm. The algorithm was applied
to the two models originally considered in (Johnson, et al., 2016) and the
proposed algorithm was shown to attain lower mean-square errors for the two
models. The experiments do not however demonstrate why the algorithm is
performing better. For instance, is the (Johnson, et al., 2016) algorithm
suffering from the implicit gradient? It also would have been great to have
considered a model that the (Johnson, et. al., 2016) algorithm would not work
well on or could not be applied to show the added applicability of the proposed
algorithm.
I also have some minor comments on the paper:
- There are a lot of typos.
- The first two sentences of the abstract do not really contribute anything
  to the paper. What is a powerful model? What is a powerful algorithm?
- DNN was used in Section 2 without being defined.
- Using p() as an approximate distribution in Section 3 is confusing notation
  because p() was used for the distributions in the model.
- How is the covariance matrix parameterized that the inference network produces?
- The phrases "first term of the inference network" are not clear. Just use The
  DNN term and the PGM term of the inference networks, and better still throw
  in a reference to Eq. (4).
- The term "deterministic parameters" was used and never introduced.
- At the bottom of page 5 the extension to the non-conjugate case should be
  presented somewhere (probably the appendix) since the fact that you can do
  this is a part of your algorithm that's important.


In this paper, we turn our attention to the interworking between the activation functions and the batch normalization, which is virtually mandatory technique to train deep networks currently. We propose the activation function Displaced Rectifier Linear Unit (DReLU) by conjecturing that extending the identity function of ReLU to the third quadrant enhances compatibility with batch normalization. Moreover, we used statistical tests to compare the impact of using distinct activation functions (ReLU, LReLU, PReLU, ELU, and DReLU) on the learning speed and test accuracy performance of standardized VGG and Residual Networks state-of-the-art models. These convolutional neural networks were trained on CIFAR-100 and CIFAR-10, the most commonly used deep learning computer vision datasets. The results showed DReLU speeded up learning in all models and datasets. Besides, statistical significant performance assessments (p < 0.05) showed DReLU enhanced the test accuracy presented by ReLU in all scenarios. Furthermore, DReLU showed better test accuracy than any other tested activation function in all experiments with one exception, in which case it presented the second best performance. Therefore, this work demonstrates that it is possible to increase performance replacing ReLU by an enhanced activation function.

This paper proposes an activation function, called displaced ReLU (DReLU), to improve the performance of CNNs that use batch normalization. Compared to ReLU, DReLU cut the identity function at a negative value rather than the zero. As a result, the activations outputted by DReLU can have a mean closer to 0 and a variance closer to 1 than the standard ReLU. The DReLU is supposed to remedy the problem of covariate shift better.
The presentation of the paper is clear. The proposed method shows encouraging results in a controlled setting (i.e., all other units, like dropout, are removed). Statistical tests are performed for many of the experimental results, which is solid.
However, I have some concerns.
1) As DReLU(x) = max{-\delta, x}, what is the optimal strategy to determine \delta? If it is done by hyperparameter tuning with cross-validation, the training cost may be too high.
2) I believe the control experiments are encouraging, but I do not agree that other techniques like Dropouts are not useful. Using DReLU to improve the state-of-art neural network in an uncontrolled setting is important. The arguments for skipping this experiments are respectful, but not convincing enough.
3) Batch normalization is popular, especially for the convolutional neural networks. However, its application is not universal, which can limit the use of the proposed DReLU. It is a minor concern, anyway.


Recent efforts on training light-weight binary neural networks offer promising execution/memory efficiency. This paper introduces ResBinNet, which is a composition of two interlinked methodologies aiming to address the slow convergence speed and limited accuracy of binary convolutional neural networks. The first method, called residual binarization, learns a multi-level binary representation for the features within a certain neural network layer. The second method, called temperature adjustment, gradually binarizes the weights of a particular layer. The two methods jointly learn a set of soft-binarized parameters that improve the convergence rate and accuracy of binary neural networks. We corroborate the applicability and scalability of ResBinNet by implementing a prototype hardware accelerator. The accelerator is reconfigurable in terms of the numerical precision of the binarized features, offering a trade-off between runtime and inference accuracy.

This paper proposes ResBinNet, with residual binarization, and temperature adjustment. It is a reconfigurable binarization method for neural networks. It improves the convergence rate during training.
I appreciate a lot that the authors were able to validate their idea by building a prototype of an actual hardware accelerator.
I am wondering what are the values of \gamma’s in the residual binarization after learning? What is its advantage over having only one \gamma, and then the rest are just 1/2*\gamma, 1/4* \gamma, … , etc.? The latter is an important baseline for residual binarization because that corresponds to the widely used fixed point format for real numbers. If you can show some results that residual encoding is better than having {\gamma, 1/2*\gamma, 1/4* \gamma, …, } (which contains only one \gamma), it would validate the need of using this relatively complex binarization scheme. Otherwise, we can just use the l-bit fixed point multiplications, which is off-the-shelf and already highly optimized in many hardwares.
For the temperature adjustment, modifying the tanh() scale has already had a long history, for example, http://yann.lecun.com/exdb/publis/pdf/lecun-89.pdf page 7, which is exactly the same form as in this paper. Adjusting the slope during training has also been explored in some straight-through estimator approaches, such as https://arxiv.org/pdf/1609.01704.pdf. In addition, having this residual binarization and adjustable tanh(), is already adding extra computations for training. Could you provide some data for comparing the computations before and after adding residual binarization and temperature adjustment?
The authors claimed that ResBinNet converges faster during training, and in table 2 it shows that ResBinNet just needs 1/10 of the training epochs needed by BinaryNet. However, I don’t find it very fair. Given that the accuracy RBN gets is much lower than Binary Net, the readers might suspect that maybe the other two models already reach ResBinNet’s accuracy at an earlier training epochs (like epoch 50), and just take all the remaining epochs to reach a higher accuracy. On the other hand, this comparison is not fair for ResBinNet as well. The model size was much larger in BinaryNet than in ResBinNet. So it makes sense to train a BinaryNet or FINN, in the same size, and then compare the training curves. Lastly, in CIFAR-10 1-level case, it didn’t outperform FINN, which has the same size. Given these experiments, I can’t draw any convincing conclusion.
Apart from that, There is an error in Figure 7 (b), where the baseline has an accuracy of 80.1% but its corresponding bar is lower than RBN1, which has an accuracy of 76%.


Published as a conference paper at ICLR 2018 DCN+: MIXED OBJECTIVE AND DEEP RESIDUAL COATTENTION FOR QUESTION ANSWERING
Traditional models for question answering optimize using cross entropy loss, which encourages exact answers at the cost of penalizing nearby or overlapping answers that are sometimes equally accurate. We propose a mixed objective that combines cross entropy loss with self-critical policy learning. The objective uses rewards derived from word overlap to solve the misalignment between evaluation metric and optimization objective. In addition to the mixed objective, we improve dynamic coattention networks (DCN) with a deep residual coattention encoder that is inspired by recent work in deep self-attention and residual networks. Our proposals improve model performance across question types and input lengths, especially for long questions that requires the ability to capture long-term dependencies. On the Stanford Question Answering Dataset, our model achieves state-of-the-art results with 75.1% exact match accuracy and 83.1% F1, while the ensemble obtains 78.9% exact match accuracy and 86.0% F1.

The authors of this paper propose some extensions to the Dynamic Coattention Networks models presented last year at ICLR. First they modify the architecture of the answer selection model by adding an extra coattention layer to improve the capture of dependencies between question and answer descriptions. The other main modification is to train their DCN+ model using both cross entropy loss and F1 score (using RL supervision) in order to  reward the system for making partial matching predictions. Empirical evaluations conducted on the SQuAD dataset indicates that this architecture achieves an improvement of at least 3%, both on F1 and exact match accuracy, over other comparable systems. An ablation study clearly shows the contribution of the deep coattention mechanism and mixed objective training on the model performance.
The paper is well written, ideas are presented clearly and the experiments section provide interesting insights such as the impact of RL on system training or the capability of the model to handle long questions and/or answers. It seems to me that this paper is a significant contribution to the field of question answering systems.


Published as a conference paper at ICLR 2018 TRAINING WIDE RESIDUAL NETWORKS FOR DEPLOY- MENT USING A SINGLE BIT FOR EACH WEIGHT
For fast and energy-efficient deployment of trained deep neural networks on resource-constrained embedded hardware, each learned weight parameter should ideally be represented and stored using a single bit. Error-rates usually increase when this requirement is imposed. Here, we report large improvements in error rates on multiple datasets, for deep convolutional neural networks deployed with 1-bit-per-weight.
Using wide residual networks as our main baseline, our approach simplifies existing methods that binarize weights by applying the sign function in training; we apply scaling factors for each layer with constant unlearned values equal to the layer-specific standard deviations used for initialization. For CIFAR-10, CIFAR-100 and ImageNet, and models with 1-bitper-weight requiring less than 10 MB of parameter memory, we achieve error rates of 3.9%, 18.5% and 26.0% / 8.5% (Top-1 / Top-5) respectively. We also considered MNIST, SVHN and ImageNet32, achieving 1-bit-per-weight test results of 0.27%, 1.9%, and 41.3% / 19.1% respectively. For CIFAR, our error rates halve previously reported values, and are within about 1% of our errorrates for the same network with full-precision weights. For networks that overfit, we also show significant improvements in error rate by not learning batch normalization scale and offset parameters. This applies to both full precision and 1-bit-per-weight networks. Using a warm-restart learning-rate schedule, we found that training for 1-bit-per-weight is just as fast as full-precision networks, with better accuracy than standard schedules, and achieved about 98%-99% of peak performance in just 62 training epochs for CIFAR-10/100. For full training code and trained models in MATLAB, Keras and PyTorch see https://github.com/McDonnell-Lab/1-bit-per-weight/.

The authors propose to train neural networks with 1bit weights by storing and updating full precision weights in training, but using the reduced 1bit version of the network to compute predictions and gradients in training. They add a few tricks to keep the optimization numerically efficient. Since right now more and more neural networks are deployed to end users, the authors make an interesting contribution to a very relevant question.
The approach is precisely described although the text sometimes could be a bit clearer (for example, the text contains many important references to later sections).
The authors include a few other methods for comparision, but I think it would be very helpful to include also some methods that use a completely different approach to reduce the memory footprint. For example, weight pruning methods sometimes can give compression rates of around 100 while the 1bit methods by definition are limited to a compression rate of 32. Additionally, for practical applications, methods like weight pruning might be more promising since they reduce both the memory load and the computational load.
Side mark: the manuscript has quite a few typos.