Given an existing trained neural network, it is often desirable to learn new capabilities without hindering performance of those already learned. Existing approaches either learn sub-optimal solutions, require joint training, or incur a substantial increment in the number of parameters for each added task, typically as many as the original network. We propose a method called Deep Adaptation Networks (DAN) that constrains newly learned filters to be linear combinations of existing ones. DANs preserve performance on the original task, require a fraction (typically 13%) of the number of parameters compared to standard fine-tuning procedures and converge in less cycles of training to a comparable or better level of performance. When coupled with standard network quantization techniques, we further reduce the parameter cost to around 3% of the original with negligible or no loss in accuracy. The learned architecture can be controlled to switch between various learned representations, enabling a single network to solve a task from multiple different domains. We conduct extensive experiments showing the effectiveness of our method on a range of image classification tasks and explore different aspects of its behavior.

This paper proposes to adapt convnet representations to new tasks while avoiding catastrophic forgetting by learning a per-task “controller” specifying weightings of the convolution-al filters throughout the network while keeping the filters themselves fixed.
Pros
The proposed approach is novel and broadly applicable.  By definition it maintains the exact performance on the original task, and enables the network to transfer to new tasks using a controller with a small number of parameters (asymptotically smaller than that of the base network).
The method is tested on a number of datasets (each used as source and target) and shows good transfer learning performance on each one.  A number of different fine-tuning regimes are explored.
The paper is mostly clear and well-written (though with a few typos that should be fixed).
Cons/Questions/Suggestions
The distinction between the convolutional and fully-connected layers (called “classifiers”) in the approach description (sec 3) is somewhat arbitrary -- after all, convolutional layers are a generalization of fully-connected layers. (This is hinted at by the mention of fully convolutional networks.)  The method could just as easily be applied to learn a task-specific rotation of the fully-connected layer weights.  A more systematic set of experiments could compare learning the proposed weightings on the first K layers of the network (for K={0, 1, …, N}) and learning independent weights for the latter N-K layers, but I understand this would be a rather large experimental burden.
When discussing the controller initialization (sec 4.3), it’s stated that the diagonal init works the best, and that this means one only needs to learn the diagonals to get the best results.  Is this implying that the gradients wrt off-diagonal entries of the controller weight matrix are 0 under the diagonal initialization, hence the off-diagonal entries remain zero after learning?  It’s not immediately clear to me whether this is the case -- it could help to clarify this in the text.
If the off-diag gradients are indeed 0 under the diag init, it could also make sense to experiment with an “identity+noise” initialization of the controller matrix, which might give the best of both worlds in terms of flexibility and inductive bias to maintain the original representation. (Equivalently, one could treat the controller-weighted filters as a “residual” term on the original filters F with the controller weights W initialized to noise, with the final filters being F+(W\crossF) rather than just W\crossF.)
The dataset classifier (sec 4.3.4) could be learnt end-to-end by using a softmax output of the dataset classifier as the alpha weighting. It would be interesting to see how this compares with the hard thresholding method used here.  (As an intermediate step, the performance could also be measured with the dataset classifier trained in the same way but used as a soft weighting, rather than the hard version rounding alpha to 0 or 1.)
Overall, the paper is clear and the proposed method is sensible, novel, and evaluated reasonably thoroughly.


Building deep neural networks to control autonomous agents which have to interact in real-time with the physical world, such as robots or automotive vehicles, requires a seamless integration of time into a network's architecture. The central question of this work is, how the temporal nature of reality should be reflected in the execution of a deep neural network and its components. Most artificial deep neural networks are partitioned into a directed graph of connected modules or layers and the layers themselves consist of elemental building blocks, such as single units. For most deep neural networks, all units of a layer are processed synchronously and in parallel, but layers themselves are processed in a sequential manner. In contrast, all elements of a biological neural network are processed in parallel. In this paper, we define a class of networks between these two extreme cases. These networks are executed in a streaming or synchronous layerwise-parallel manner, unlocking the layers of such networks for parallel processing. Compared to the standard layerwise-sequential deep networks, these new layerwise-parallel networks show a fundamentally different temporal behavior and flow of information, especially for networks with skip or recurrent connections. We argue that layerwise-parallel deep networks are better suited for future challenges of deep neural network design, such as large functional modularized and/or recurrent architectures as well as networks allocating different network capacities dependent on current stimulus and/or task complexity. We layout basic properties and discuss major challenges for layerwise-parallel networks. Additionally, we provide a toolbox to design, train, evaluate, and online-interact with layerwiseparallel networks.

This paper introduces a new toolbox for deep neural networks learning and evaluation. The central idea is to include time in the processing of all the units in the network. For this, the authors propose a paradigm switch: form layerwise-sequential networks, where at every time frame the network is evaluated by updating each layer – from bottom to top – sequentially; to layerwise-parallel networks, where all the neurons are updated in parallel. The new paradigm implies that the layer update is achieved by using the stored previous state and the corresponding previous state of the previous layer. This has three consequences. First, every layer now use memory, a condition that already applies for RNNs in layerwise-sequential networks. Second, in order to have a consistent output, the information has to flow in the network for a number of time frames equal to the number of layers. In Neuroscience, this concept is known as reaction time. Third, since the network is not synchronized in terms of the information that is processed in a specific time frame, there are discrepancies w.r.t. the layerwise-sequential networks computation: all the techniques used to train deep NNs have to be reconsidered.
Overall, the concept is interesting and timely especially for the rising field of spiking neural networks or for large and distributed architectures. The paper, however, should probably provide more examples and results in terms of architectures that can been implemented with the toolbox in comparison with other toolboxes. The paper presents a single example in which either the accuracy and the training time are not reported. While I understand that the main result of this work is the toolbox itself, more examples and results would improve the clarity and the implications for such paradigm switch. Another concern comes from the choice to use Theano as back-end, since it's known that it is going to be discontinued. Finally I suggest to improve the clarity and description of Figure 2, which is messy and confusing especially if printed in B&W.


Published as a conference paper at ICLR 2018 EMERGENT COMPLEXITY VIA MULTI-AGENT COMPETITION
Reinforcement learning algorithms can train agents that solve problems in complex, interesting environments. Normally, the complexity of the trained agent is closely related to the complexity of the environment. This suggests that a highly capable agent requires a complex environment for training. In this paper, we point out that a competitive multi-agent environment trained with self-play can produce behaviors that are far more complex than the environment itself. We also point out that such environments come with a natural curriculum, because for any skill level, an environment full of agents of this level will have the right level of difficulty. This work introduces several competitive multi-agent environments where agents compete in a 3D world with simulated physics. The trained agents learn a wide variety of complex and interesting skills, even though the environment themselves are relatively simple. The skills include behaviors such as running, blocking, ducking, tackling, fooling opponents, kicking, and defending using both arms and legs. A highlight of the learned behaviors can be found here: https://goo.gl/eR7fbX.

In this paper, the authors produced quite cool videos showing the acquisition of highly complex skills, and they are happy about it. If you read the conclusion, this is the only message they put forward, and to me this is not a scientific message.
A more classical summary is that the authors use PPO, a state-of-the-art deep RL method, in a context where two agents are trained to perform competitive games against each other. They reuse a very recent "dense reward" technique to bootstrap the agent skills, and then anneal it to zero so that the competitive rewards obtained from defeating the opponent takes the lead. They study the effect of this annealing process (considered as a curriculum) and of various strategies for sampling the opponents. The main outcome is the acquisition of a large variety of useful skills, just observed from videos of the competitions.
The main issue with this paper is the lack of scientific analysis of the results, together with many local issues in the presentation of these results.
Below, I talk directly to the authors.
---------------------------------
The related work subsection is just a list of works, it should explain how the proposed work position itself with respect to these works.
In Section 5.2, you are just describing "cool" behaviors observed from your videos.
Science is about producing quantitative results, analyzing them and discussing them.
I would be glad to read more science about these cool behaviors. Can you define a repertoire of such behaviors?
Determine how often they are discovered? Study how the are represented in the networks?
Anything beyond "look, that's great!" would make the paper better...
By the end of Section 5.2, you allude to transfer learning phenomena.
It would be nice to study these transfer effects in your results with a quantitative methodology.
Section 5.3 is more scientific, but it has serious issues.
In all subfigures in Figure 3, the performance of opponents should be symmetric around 50%. This is not the case for subfigures (b) and (c-1). Why?
Do they correspond to non-zero sum game? The x-label is "version". Don't you mean "number of epochs", or something like this? Why do the last 2 images
share the same caption?
I had a hard time understanding the message from Table 1. It really needs a line before the last row and a more explicative caption.
Still in 5.3, "These results echo"...: can you characterize this echo? What is the relationship to this other work?
Again, "These results shed further light": further with respect to what? Can you be more explicit about what we learn?
Also, I find that annealing a kind of reward with respect to another is a weak form of curriculum learning. This should be further discussed.
In Section 5.4, the idea of using many opponents from many stages of learning in not new.
If I'm correct, the same was done in evolutionary method to escape the "arms race" dead-end in prey-predator races quite a while ago  (see e.g. "Coevolving predator and prey robots: Do “arms races” arise in artificial evolution?" Nolfi and Floreano, 1998)
Section 5.5.1 would deserve a more quantitative presentation of the effect of randomization.
Actually, in Fig5: the axes are not labelled. I don't believe it shows a win-rate. So probably the caption (or the image) is wrong.
In Section 5.5.2, you "suspect this is because...".
The role of a scientific paper is to clearly establish results and explanation from solid quantitative analysis.
-------------------------------------------
More local comments:
Abstract:
"Normally, the complexity of the trained agent is closely related to the complexity of the environment." Here you could cite Herbert Simon (1962).
"In this paper, we point out that a competitive multi-agent environment trained with self-play can produce behaviors that are far more complex than the environment itself."
Well, for an agent, the other agent(s) are part of its environment, aren't they? So I don't like this perspective that the environment itself is "simple".
Intro:
"RL is exciting because good RL exists." I don't believe this is a strong argument. There are many good things that exist which are not exciting.
"In general, training an agent to perform a highly complex task requires a highly complex environment, and these can be difficult to create." Well, the standard perspective is the other way round: in general, you face a complex problem, then you need to design a complex agent to solve it, and this is difficult.
"This happens because no matter how weak or strong an agent is, an environment populated with other agents of comparable strength provides the right challenge to the agent, facilitating maximally rapid learning and avoiding getting stuck." This is not always true. The literature is full of examples where two-players competition end-up with oscillations between to solutions rather than ever-increasing skill performance. See the prey-predator literature pointed above.
"in the domain of continuous control, where balance, dexterity, and manipulation are the key skills." In robotics, dexterity, and manipulation usually refer to using the robot's hand(s), a capability which is not shown here.
In preliminaries, notation, what you describe corresponds to the framework of Dec-POMDPs, you should position yourself with respect to this framework (see e.g. Memory-Bounded Dynamic Programming for DEC-POMDPs. S Seuken, S Zilberstein)
In PPO description : Let l_t(\theta) ... denote the likelihood ratio: of what?
p5:
would train on the dense reward for about 10-15% of the trainig epochs. So how much is \alpha_t? How did you tune it? Was it hard?
p6:
you give to the agent the mass: does the mass change over time???
In observations: Are both agents given different observations? Could you specify which is given what?
In Algorithms parameters: why do you have to anneal longer for kick-and-defend? What is the underlying phenomenon?
In Section 5, the text mentions Fig5 before Fig4.
-------------------------------------------------
Typos:
p4:
research(Andrychowicz => missing space
straight forward => straightforward
p5:
agent like humanoid(s)
from exi(s)ting work
p6:
eq. 1 => Eq. (1) (you should use \eqref{})
In section 4.1 => In Section 4.1 (same p7 for Section 4.2)
"One question that arises is the extent to which the outcome of learning is affected by this exploration reward and to explore the benefit of this exploration reward. As already argued, we found the exploration reward to be crucial for learning as otherwise the agents are unable to explore the sparse competition reward." => One question that arises is the extent to which the outcome of learning is affected by this exploration reward and to explore its benefit. As already argued, we found it to be crucial for learning as otherwise the agents are unable to explore the sparse competition reward.
p8:
in a local minima => minimum
p9:
in references, you have Jakob Foerster and Jakob N Foerster => try to be more consistent.
p10, In Laetitia Matignon et al.  ... markov => Markov
p11, I would rename C_{alive} as C_{standing}


According to parallel distributed processing (PDP) theory in psychology, neural networks (NN) learn distributed rather than interpretable localist representations. This view has been held so strongly that few researchers have analysed single units to determine if this assumption is correct. However, recent results from psychology, neuroscience and computer science have shown the occasional existence of local codes emerging in artificial and biological neural networks. In this paper, we undertake the first systematic survey of when local codes emerge in a feed-forward neural network, using generated input and output data with known qualities. We find that the number of local codes that emerge from a NN follows a well-defined distribution across the number of hidden layer neurons, with a peak determined by the size of input data, number of examples presented and the sparsity of input data. Using a 1-hot output code drastically decreases the number of local codes on the hidden layer. The number of emergent local codes increases with the percentage of dropout applied to the hidden layer, suggesting that the localist encoding may offer a resilience to noisy networks. This data suggests that localist coding can emerge from feed-forward PDP networks and suggests some of the conditions that may lead to interpretable localist representations in the cortex. The findings highlight how local codes should not be dismissed out of hand.

This paper studies the development of localist representations in the hidden layers
of feed-forward neural networks.
The idea is interesting and the findings are intriguing.  Local codes
increase understandability and could be important for better
understanding natural neural networks. Understanding how local codes
form and the factors that increase their likelihood is critically
important.  This is a good start in that direction, but still leaves
open many questions.  The issues raised in the Conclusions section are
also very interesting -- do the local codes increase with networks
that generalize better, or with overtrained networks?
A  weakness in this paper (admitted by the authors in the
Conclusions section) is the dependence of the results on the form of input
representation.  If we consider the Jennifer Aniston cells, they do
not receive as input as well separated inputs as modeled in this
paper.  In fact the input representation used in this study is already
a fairly localist representation as each 1 unit is fairly selectively
on for its own class and mostly off for the other classes.  It will be
very interesting to see the results of hidden layers in deep networks
operating on natural images.
Please give your equation for selectivity.  On Page 2 it is stated "We
use the word ‘selectivity’ as a quantitative measure of the difference
between activations for the two categories, A and not-A, where A is
the class a neuron is selective for (and not-A being all other
classes)."  However you state that neurons were counted as a local
code if the selectivity was above .05.  A difference between
activations for the two categories of .05 does not seem very
selective, so I'm thinking you used something other than the
mathematical difference.
What is the selectivity of units in the input codewords?  With no
perturbation, and S_x=.2, w_R=50, w_P=50, the units in the prototype
blocks have a high selectivity responding with 1 for all patterns in
their class and with 0 for 8/9 of the patterns in the other classes.
Could this explain the much higher selectivity for this case in the
hidden units?  I would like to see the selectivity of the input units
for each of the plots/curves.  This would be especially interesting
for Figure 5.
It is stated that LCs emerge with longer training and that ReLU
neurons may produce more LCs because they train quicker and all
experiments were stopped at 45,000 epochs.  Why not investigate this
by changing learning rates for one of ReLu or sigmoidal units to more
closely match their training speed?  It would be interesting to see if
the difference is simply due to learning rate, or something deeper
about the activation functions.
You found that very few local codes in the HLNs were found when a
1-hot ouput encoding was used and suggest that this means that
emergent local codes are highly unlikely to be found in the
penultimate layer of deep networks.  If your inputs are a local code
(e.g. for low w_R), you found local codes above the layer of local
codes but in this result not below the layer of local codes which
might also imply (as you say in the Conclusions) that more local
coding neurons may be found in the higher layers (though not the
penultimate one as you argue).  Could you analyze how the selectivity
of a hidden layer changes as a function of the selectivity in the
lower and higher layers?
Minor Note -- The Neural Network Design section looks like it still
has draft notes in it.


Words in natural language follow a Zipfian distribution whereby some words are frequent but most are rare. Learning representations for words in the "long tail" of this distribution requires enormous amounts of data. Representations of rare words trained directly on end tasks are usually poor, requiring us to pre-train embeddings on external data, or treat all rare words as out-of-vocabulary words with a unique representation. We provide a method for predicting embeddings of rare words on the fly from small amounts of auxiliary data with a network trained end-to-end for the downstream task. We show that this improves results against baselines where embeddings are trained on the end task for reading comprehension, recognizing textual entailment and language modeling.

This paper examines ways of producing word embeddings for rare words on demand. The key real-world use case is for domain specific terms, but here the techniques are demonstrated on rarer words in standard data sets. The strength of this paper is that it both gives a more systematic framework for and builds on existing ideas (character-based models, using dictionary definitions) to implement them as part of a model trained on the end task.
The contribution is clear but not huge. In general, for the scope of the paper, it seems like what is here could fairly easily have been made into a short paper for other conferences that have that category. The basic method easily fits within 3 pages, and while the presentation of the experiments would need to be much briefer, this seems quite possible. More things could have been considered. Some appear in the paper, and there are some fairly natural other ones such as mining some use contexts of a word (such as just from Google snippets) rather than only using textual definitions from wordnet. The contributions are showing that existing work using character-level models and definitions can be improved by optimizing representation learning in the context of the final task, and the idea of adding a learned linear transformation matrix inside the mean pooling model (p.3). However, it is not made very clear why this matrix is needed or what the qualitative effect of its addition is.
The paper is clearly written.
A paper that should be referred to is the (short) paper of Dhingra et al. (2017): A Comparative Study of Word Embeddings
for Reading Comprehension https://arxiv.org/pdf/1703.00993.pdf . While it in no way covers the same ground as this paper it is relevant as follows: This paper assumes a baseline that is also described in that paper of using a fixed vocab and mapping other words to UNK. However, they point out that at least for matching tasks like QA and NLI that one can do better by assigning random vectors on the fly to unknown words. That method could also be considered as a possible approach to compare against here.
Other comments:
 - The paper suggests a couple of times including at the end of the 2nd Intro paragraph that you can't really expect spelling models to perform well in representing the semantics of arbitrary words (which are not morphological derivations, etc.). While this argument has intuitive appeal, it seems to fly in the face of the fact that actually spelling models, including in this paper, seem to do surprisingly well at learning such arbitrary semantics.
 - p.2: You use pretrained GloVe vectors that you do not update. My impression is that people have had mixed results, sometimes better, sometimes worse with updating pretrained vectors or not. Did you try it both ways?
 - fn. 1: Perhaps slightly exaggerates the point being made, since people usually also get good results with the GloVe or word2vec model trained on "only" 6 billion words – 2 orders of magnitude less data.
 - p.4. When no definition is available, is making e_d(w) a zero vector worse than or about the same as using a trained UNK vector?
 - Table 1: The baseline seems reasonable (near enough to the quality of the original Salesforce model from 2016 (66 F1) but well below current best single models of around 76-78 F1. The difference between D1 and D3 does well illustrate that better definition learning is done with backprop from end objective. This model shows the rather strong performance of spelling models – at least on this task – which he again benefit from training in the context of the end objective.
 - Fig 2: It's weird that only the +dict (left) model learns to connect "In" and "where". The point made in the text between "Where" and "overseas" is perfectly reasonable, but it is a mystery why the base model on the right doesn't learn to associate the common words "where" and "in" both commonly expressing a location.
 - Table 2: These results are interestingly different. Dict is much more useful than spelling here. I guess that is because of the nature of NLI, but it isn't 100% clear why NLI benefits so much more than QA from definitional knowledge.
 - p.7: I was slightly surprised by how small vocabs (3k and 5k words) are said to be optimal for NLI (and similar remarks hold for SQuAD). My impression is that most papers on NLI use much larger vocabs, no?
 - Fig 3: This could really be drawn considerably better: make the dots bigger and their colors more distinct.
 - Table 3: The differences here are quite small and perhaps the least compelling, but the same trends hold.


Published as a conference paper at ICLR 2018 SENSITIVITY AND GENERALIZATION IN NEURAL NETWORKS: AN EMPIRICAL STUDY
In practice it is often found that large over-parameterized neural networks generalize better than their smaller counterparts, an observation that appears to conflict with classical notions of function complexity, which typically favor smaller models. In this work, we investigate this tension between complexity and generalization through an extensive empirical exploration of two natural metrics of complexity related to sensitivity to input perturbations. Our experiments survey thousands of models with various fully-connected architectures, optimizers, and other hyper-parameters, as well as four different image classification datasets.
We find that trained neural networks are more robust to input perturbations in the vicinity of the training data manifold, as measured by the norm of the input-output Jacobian of the network, and that it correlates well with generalization. We further establish that factors associated with poor generalization -such as full-batch training or using random labels -correspond to lower robustness, while factors associated with good generalization -such as data augmentation and ReLU non-linearities -give rise to more robust functions. Finally, we demonstrate how the input-output Jacobian norm can be predictive of generalization at the level of individual test points.

This paper proposes an analysis of the robustness of deep neural networks with respect to data perturbations.
*Quality*
The quality of exposition is not satisfactory. Actually, the paper is pretty difficult to evaluate at the present stage and it needs a drastic change in the writing style.
*Clarity*
The paper is not clear and highly unstructured.
*Originality*
The originality is limited for what regards Section 3: the proposed metrics are quite standard tools from differential geometry. Also, the idea of taking into account the data manifold is not brand new since already proposed in “Universal Adversarial Perturbation” at CVPR 2017.
*Significance*
Due to some flaws in the experimental settings, the relevance of the presented results is very limited. First, the authors essentially exploit a customized architecture, which has been broadly fine-tuned regarding hyper-parameters, gating functions and optimizers. Why not using well established architectures (such as DenseNets, ResNets, VGG, AlexNet)?
Moreover, despite having a complete portrait of the fine-tuning process is appreciable, this compromises the clarity of the figures which are pretty hard to interpret and absolutely not self-explanatory: probably it’s better to only consider the best configuration as opposed to all the possible ones.
Second, authors assume that circular interpolation is a viable way to traverse the data manifold. The reviewer believes that it is an over-simplistic assumption. In fact, it is not guaranteed a priori that such trajectories are geodesic curves so, a priori, it is not clear why this could be a sound technique to explore the data manifold.
CONS:
The paper is difficult to read and needs to be thoroughly re-organized. The problem is not stated in a clear manner, and paper’s contribution is not outlined. The proposed architectures should be explained in detail. The results of the sensitivity analysis should be discussed in detail. The authors should explain the approach of traversing the data manifold with ellipses (although the reviewer believes that such approach needs to be changed with something more principled). Figures and results are not clear.
The authors are kindly asked to shape their paper to match the suggested format of 8 pages + 1 of references (or similar). The work is definitely too long considered its quality. Additional plots and discussion can be moved to an appendix.
Despite the additional explanation in Footnote 6, the graphs are not clear. Probably authors should avoid to present the result for each possible configuration of the hyper-parameters, gatings and optimizers and just choose the best setting.
Apart from the customized architecture, authors should have considered established deep nets, such as DenseNets, ResNets, VGG, AlexNet.
The idea of considering the data manifold within the measurement of complexity is a nice claim, which unfortunately is paired with a not convincing experimental analysis. Why ellipses should be a proper way to explore the data manifold? In general, circular interpolation is not guaranteed to be geodesic curves which lie on the data manifold.
Minor Comments:
Sentence to rephrase: “We study common in the machine learning community ways to ...”
Please, put the footnotes in the corresponding page in which it is referred.
The reference to ReLU is trivially wrong and need to be changed with [Nair & Hinton ICML 2010]
**UPDATED EVALUATION AFTER AUTHORS' REBUTTAL**
We appreciated the effort in providing specific responses and we also inspected the updated version of the paper. Unfortunately, despite the authors' effort, the reviewer deems that the conceptual issues that have been highlighted are still present in the paper which, therefore, is not ready for acceptance yet.


Reinforcement learning methods have recently achieved impressive results on a wide range of control problems. However, especially with complex inputs, they still require an extensive amount of training data in order to converge to a meaningful solution. This limitation largely prohibits their usage for complex input spaces such as video signals, and it is still impossible to use it for a number of complex problems in a real world environments, including many of those for video based control. Supervised learning, on the contrary, is capable of learning on a relatively small number of samples, however it does not take into account rewardbased control policies and is not capable to provide independent control policies. In this article we propose a model-free control method, which uses a combination of reinforcement and supervised learning for autonomous control and paves the way towards policy based control in real world environments. We use SpeedDreams/TORCS video game to demonstrate that our approach requires much less samples (hundreds of thousands against millions or tens of millions) comparing to the state-of-the-art reinforcement learning techniques on similar data, and at the same time overcomes both supervised and reinforcement learning approaches in terms of quality. Additionally, we demonstrate the applicability of the method to MuJoCo control problems.

This paper proposes leveraging labelled controlled data to accelerate reinforcement-based learning of a control policy.  It provides two main contributions: pre-training the policy network of a DDPG agent in a supervised manner so that it begins in reasonable state-action distribution and regalurizing the Q-updates of the q-network to be biased towards existing actions.  The authors use the TORCS enviroment to demonstrate the performance of their method both in final cumulative return of the policy and speed of learning.
This paper is easy to understand but has a couple shortcomings and some fatal (but reparable) flaws:.
1) When using RL please try to standardize your notation to that used by the community, it makes things much easier to read.  I would strongly suggest avoiding your notation a(x|\Theta) and using \pi(x) (subscripting theta or making conditional is somewhat less important).  Your a(.) function seems to be the policy here, which is invariable denoted \pi in the RL literature.  There has been recent effort to clean up RL notation which is presented here: https://sites.ualberta.ca/~szepesva/papers/RLAlgsInMDPs.pdf. You have no obligation to use this notation but it does make reading of your paper much easier on others in the community.  This is more of a shortcoming than a fundamental issue.
2) More fatally, you have failed to compare your algorithm's performance against benchline implementations of similar algorithms.  It is almost trivial to run DDPG on Torcs using the openAI baselines package [https://github.com/openai/baselines].  I would have loved, for example, to see the effects of simply pre-training the DDPG actor on supervised data, vs. adding your mixture loss on the critic.  Using the baselines would have (maybe) made a very compelling graph showing DDPG, DDPG + actor pre-training, and then your complete method.
3) And finally, perhaps complementary to point 2), you really need to provide examples on more than one environment.  Each of these simulated environments has its own pathologies linked to determenism, reward structure, and other environment particularities.  Almost every algorithm I've seen published will often beat baselines on one environment and then fail to improve or even be wors on others, so it is important to at least run on a series of these.  Mujoco + AI Gym should make this really easy to do (for reference, I have no relatinship with OpenAI).  Running at least cartpole (which is a very well understood control task), and then perhaps reacher, swimmer, half-cheetah etc. using a known contoller as your behavior policy (behavior policy is a good term for your data-generating policy.)
4) In terms of state of the art you are very close to Todd Hester et. al's paper on imitation learning, and although you cite it, you should contrast your approach more clearly with the one in that paper.  Please also have a look at some more recent work my Matej Vecerik, Todd Hester & Jon Scholz: 'Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards' for an approach that is pretty similar to yours.
Overall I think your intuitions and ideas are good, but the paper does not do a good enough job justifying empirically that your approach provides any advantages over existing methods.  The idea of pre-training the policy net has been tried before (although I can't find a published reference) and in my experience will help on certain problems, and hinder on others, primarily because the policy network is already 'overfit' somewhat to the expert, and may have a hard time moving to a more optimal space.  Because of this experience I would need more supporting evidence that your method actually generalizes to more than one RL environment.


DECOUPLING THE LAYERS IN RESIDUAL NETWORKS
We propose a Warped Residual Network (WarpNet) using a parallelizable warp operator for forward and backward propagation to distant layers that trains faster than the original residual neural network. We apply a perturbation theory on residual networks and decouple the interactions between residual units. The resulting warp operator is a first order approximation of the output over multiple layers. The first order perturbation theory exhibits properties such as binomial path lengths and exponential gradient scaling found experimentally by Veit et al. (2016). We demonstrate through an extensive performance study that the proposed network achieves comparable predictive performance to the original residual network with the same number of parameters, while achieving a significant speed-up on the total training time. As WarpNet performs model parallelism in residual network training in which weights are distributed over different GPUs, it offers speed-up and capability to train larger networks compared to original residual networks.

The main contribution of this paper is a particular Taylor expansion of the outputs of a ResNet which is shown to be exact at almost all points in the input space.  This expression is used to develop a new layer called a “warp layer” which essentially tries to compute several layers of the residual network using the Taylor expansion expression — however in this expression, things can be done in parallel, and interestingly, the authors show that the gradients also decouple when the (ResNet) model is close to a local minimum in a certain sense, which may motivate the decoupling of layers to begin with.  Finally the authors stack these warp layers to create a “warped resnet” which they show does about as well as an ordinary ResNet but has better parallelization properties.
To me the analytical parts of the paper are the most interesting, particularly in showing how the gradients approximately decouple.  However there are several weaknesses to the paper (or maybe just things I didn’t understand).  First,  a major part of the paper tries to make the case that there is a symmetry breaking property of the proposed model, which I am afraid I simply was not able to follow.  Some of the notation is confusing here — for example, presumably the rotations refer to image level rotations rather than literally multiplying the inputs by an orthogonal matrix, which the notation suggests to be the case.  It is also never precisely spelled out what the final theoretical guarantee is (preferably the authors would do this in the form of a proposition or theorem).
Throughout, the authors write out equations as if the weights in all layers are equal, but this is confusing even if the authors say that this is what they are doing, since their explanation is not very clear.  The confusion is particularly acute in places where derivatives are taken, because the derivatives continue to be taken as if the weights were untied, but then written as if they happened to be the same.
Finally the experimental results are okay but perhaps a bit preliminary.  I have a few recommendations here:
* It would be stronger to evaluate results on a larger dataset like ILSVRC.
* The relative speed-up of WarpNet compared to ResNet needs to be better explained — the authors break the computation of the WarpNet onto two GPUs, but it’s not clear if they do this for the (vanilla) ResNet as well.  In batch mode, the easiest way to parallelize is to have each GPU evaluate half the batch.  Even in a streaming mode where images need to be evaluated one by one, there are ways to pipeline execution of the residual blocks, and I do not see any discussion of these alternatives in the paper.
* In the experimental results, K is set to be 2, and the authors only mention in passing that they have tried larger K in the conclusion.  It would be good to have a more thorough experimental evaluation of the trade-offs of setting K to be higher values.
A few remaining questions for the authors:
* There is a parallel submission (presumably by different authors called “Residual Connections Encourage Iterative Inference”) which contains some related insights.  I wonder what are the differences between the two Taylor expansions, and whether the insights of this paper could be used to help the other paper and vice versa?
* On implementation - the authors mention using Tensorflow’s auto-differentiation.  My question here is — are gradients being re-used intelligently as suggested in Section 3.1?
* I notice that the analysis about the vanishing Hessian could be applied to most of the popular neural network architectures available now.  How much of the ideas offered in this paper would then generalize to non-resnet settings?


When deep learning is applied to sensitive data sets, many privacy-related implementation issues arise. These issues are especially evident in the healthcare, finance, law and government industries. Homomorphic encryption could allow a server to make inferences on inputs encrypted by a client, but to our best knowledge, there has been no complete implementation of common deep learning operations, for arbitrary model depths, using homomorphic encryption. This paper demonstrates a novel approach, efficiently implementing many deep learning functions with bootstrapped homomorphic encryption. As part of our implementation, we demonstrate Single and Multi-Layer Neural Networks, for the Wisconsin Breast Cancer dataset, as well as a Convolutional Neural Network for MNIST. Our results give promising directions for privacy-preserving representation learning, and the return of data control to users.

The paper presents a means of evaluating a neural network securely using homomorphic encryption. A neural network is already trained, and its weights are public. The network is to be evaluated over a private input, so that only the final outcome of the computation-and nothing but that-is finally learned.
The authors take a binary-circuit approach: they represent numbers via a fixed point binary representation, and construct circuits of secure adders and multipliers, based on homomorphic encryption as a building block for secure gates. This allows them to perform the vector products needed per layer; two's complement representation also allows for an "easy" implementation of the ReLU activation function, by "checking" (multiplying by) the complement of the sign bit. The fact that multiplication often involves public weights is used to speed up computations, wherever appropriate. A rudimentary  experimental evaluation with small networks is provided.
All of this is somewhat straightforward; a penalty is paid by representing numbers via fixed point arithmetic, which is used to deal with ReLU mostly. This is somewhat odd: it is not clear why, e.g., garbled circuits where not used for something like this, as it would have been considerably faster than FHE.
There is also a work in this area that the authors do not cite or contrast to, bringing the novelty into question; please see the following papers and references therein:
GILAD-BACHRACH, R., DOWLIN, N., LAINE, K., LAUTER, K., NAEHRIG, M., AND WERNSING, J. Cryptonets: Applying neural networks to encrypted data with high throughput and accuracy. In Proceedings of The 33rd International Conference on Machine Learning (2016), pp. 201–210.
SecureML: A System for Scalable Privacy-Preserving Machine Learning
Payman Mohassel and Yupeng Zhang
SHOKRI, R., AND SHMATIKOV, V. Privacy-preserving deep learning. In
Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security (2015), ACM, pp. 1310–1321.
The first paper is the most related, also using homomorphic encryption, and seems to cover a superset of the functionalities presented here (more activation functions, a more extensive analysis, and faster decryption times). The second paper uses arithmetic circuits rather than HE, but actually implements training an entire neural network securely.
 Minor details:
The problem scenario states that the model/weights is private, but later on it ceases to be so (weights are not encrypted).
"Both deep learning and FHE are relatively recent paradigms". Deep learning is certainly not recent, while Gentry's paper is now 7 years old.
"In theory, this system alone could be used to compute anything securely." This is informal and incorrect. Can it solve the halting problem?
"However in practice the operations were incredibly slow, taking up to 30 minutes in some cases." It is unclear what operations are referred to here.


Neural networks make mistakes. The reason why a mistake is made often remains a mystery. It would be useful to have a method that can give an explanation that is intuitive to a user as to why an image is misclassified. In this paper we develop a method for explaining the mistakes of an image classification model by visually showing what must be added to an image such that it is correctly classified. Our work combines the fields of adversarial examples, generative modeling and a correction technique based on difference target propagation to create an technique that creates explanations of why an image is misclassified. In this paper we explain our method and demonstrate it on MNIST and CelebA. This approach could aid in demystifying neural networks for a user.

In this paper, the authors aim to better understand the classification of neural networks. The authors explore the latent space of a variational auto encoder and consider the perturbations of the latent space in order to obtain the correct classification. They evaluate their method on CelebA and MNIST datasets.
Pros:
 1) The paper explores an alternate methodology that uses perturbation in latent spaces to better understand neural networks
2) It takes inspiration from adversarial examples and uses the explicit classifier loss to better perturb the $z$ in the latent space
3) The method is quite simple and captures the essence of the problem well
Cons:
The main drawback of the paper is it claims to understand working of neural networks, however, actually what the authors end up doing are perturbations of the encoded latent space. This would evidently not explain why a deep network generates misclassifications for instance understanding the failure modes of ResNet or DenseNet cannot be obtained through this method. Other drawbacks include:
1)     They do not show how their method would perform against standard adversarial attack techniques, since by explaining a neural network they should be able to guard against attacks, or at-least explain why they work well.
2) The paper reports results on 2 datasets, out of which on 1 of them it does not perform well and gets stuck in a local minima therefore implying that it is not able to capture the diversity in the data well.
3) The authors provide limited evaluation on a few attributes of CelebA. Extensive evaluation that would show on a larger scale with more attributes is not performed.
4) The authors also have claimed that the added parts should be interpretable and visible. However, the perturbations of the latent space would yield small $\epsilon$ variation in the image and it need not actually explain why the modification is yielding a correct classification, the same way an imperceptible adversarial attack yields a misclassification. Therefore there is no guarantee that the added parts would be interpretable. What would be more reasonable to claim would be that the latent transformations that yield correct classifications are projected into the original image space. Some of these yield interpretations that are semantically meaningful and some of these do not yield semantically meaningful interpretations.
5)  Solving mis-classification does not seem to equate with explaining the neural network, but rather only suggest where it makes mistakes. That is not equal to an explanation about how it is making a classification decision. That would rather be done by using the same input and perturbing the weights of the classifier network.
In conclusion, the paper in its current form provides a direction in terms of using latent space exploration to understand classification errors and corrections to them in terms of perturbations of the latent space. However, these are not conclusive yet and actually verifying this would need a more thorough evaluation.


Episodic memory is a psychology term which refers to the ability to recall specific events from the past. We suggest one advantage of this particular type of memory is the ability to easily assign credit to a specific state when remembered information is found to be useful. Inspired by this idea, and the increasing popularity of external memory mechanisms to handle long-term dependencies in deep learning systems, we propose a novel algorithm which uses a reservoir sampling procedure to maintain an external memory consisting of a fixed number of past states. The algorithm allows a deep reinforcement learning agent to learn online to preferentially remember those states which are found to be useful to recall later on. Critically this method allows for efficient online computation of gradient estimates with respect to the write process of the external memory. Thus unlike most prior mechanisms for external memory it is feasible to use in an online reinforcement learning setting.
Much of reinforcement learning (RL) theory is based on the assumption that the environment has the Markov property, meaning that future states are independent of past states given the present state. This implies the agent has all the information it needs to make an optimal decision at each time and therefore has no need to remember the past. This is however not realistic in general, realistic problems often require significant information from the past to make an informed decision in the present, and there is often no obvious way to incorporate the relevant information into an expanded present state. It is thus desirable to establish techniques for learning a representation of the relevant details of the past (e.g. a memory, or learned state) to facilitate decision making in the present.
A popular approach to integrate information from the past into present decision making is to use some variant of a recurrent neural network, possibly coupled to some form of external memory, trained with backpropagation through time. This can work well for many tasks, but generally requires backpropagating many steps into the past which is not practical in an online RL setting. In purely recurrent architectures one way to make online training practical is to simply truncate gradients after a fixed number of steps. In architectures which include some form of external memory however it is not clear that this is a viable option as the intent of the external memory is generally to capture long term dependencies which would be difficult for a recurrent architecture alone to handle, especially when trained with truncated gradients. Truncating gradients to the external memory would likely greatly hinder this capability.
In this work we explore a method for adding external memory to a reinforcement learning architecture which can be efficiently trained online. We liken our method to the idea of episodic memory from psychology. In this approach the information stored in memory is constrained to consist of a finite set of past states experienced by the agent. In this work, by states we mean observations explicitly provided by the environment. In general, states could be more abstract, such as the internal state of an RNN or predictions generated by something like the Horde architecture of Sutton  et al. (2011). By storing states explicitly we enforce that the information recorded also provides the context in which it was recorded. We can therefore assign credit to the recorded state without explicitly backpropagating through time between when the information proves useful and when it was recorded. If a recorded state is found to be useful we train the agent to preferentially remember similar states in the future.

The paper proposes a modified approach to RL, where an additional "episodic memory" is kept by the agent. What this means is that the agent has a reservoir of n "states" in which states encountered in the past can be stored. There are then of course two main questions to address (i) which states should be stored and how (ii) how to make use of the episodic memory when deciding what action to take.
For the latter question, the authors propose using a "query network" that based on the current state, pulls out one state from the memory according to certain probability distribution. This network has many tunable parameters, but the main point is that the policy then can condition on this state drawn from the memory. Intuitively, one can see why this may be advantageous as one gets some information from the past. (As an aside, the authors of course acknowledge that recurrent neural networks have been used for this purpose with varying degrees of success.)
The first question, had a quite an interesting and cute answer. There is a (non-negative) importance weight associated with each state and a collection of states has weight that is simply the product of the weights. The authors claim (with some degree of mathematical backing) that sampling a memory of n states where the distribution over the subsets of past states of size n is proportional to the product of the weights is desired. And they give a cute online algorithm for this purpose. However, the weights themselves are given by a network and so weights may change (even for states that have been observed in the past). There is no easy way to fix this and for the purpose of sampling the paper simply treats the weights as immutable.
There is also a toy example created to show that this approach works well compared to the RNN based approaches.
Positives:
- An interesting new idea that has potential to be useful in RL
- An elegant algorithm to solve at least part of the problem properly (the rest of course relies on standard SGD methods to train the various networks)
Negatives:
- The math is fudged around quite a bit with approximations that are not always justified
- While overall the writing is clear, in some places I feel it could be improved. I had a very hard time understanding the set-up of the problem in Figure 2. [In general, I also recommend against using figure captions to describe the setup.]
- The experiments only demonstrate the superiority of this method on an example chosen artificially to work well with this approach.


Hyperparameter tuning is one of the most time-consuming workloads in deep learning. State-of-the-art optimizers, such as AdaGrad, RMSProp and Adam, reduce this labor by adaptively tuning an individual learning rate for each variable. Recently researchers have shown renewed interest in simpler methods like momentum SGD as they may yield better results. Motivated by this trend, we ask: can simple adaptive methods, based on SGD perform as well or better? We revisit the momentum SGD algorithm and show that hand-tuning a single learning rate and momentum makes it competitive with Adam. We then analyze its robustness to learning rate misspecification and objective curvature variation. Based on these insights, we design YELLOWFIN, an automatic tuner for momentum and learning rate in SGD. YELLOWFIN optionally uses a negative-feedback loop to compensate for the momentum dynamics in asynchronous settings on the fly. We empirically show YELLOWFIN can converge in fewer iterations than Adam on ResNets and LSTMs for image recognition, language modeling and constituency parsing, with a speedup of up to 3.28x in synchronous and up to 2.69x in asynchronous settings.

This paper proposes a method to automatically tuning the momentum parameter in momentum SGD methods, which achieves better results and fast convergence speed than state-of-the-art Adam algorithm.
Although the results are promising, I found the presentation of this paper almost inaccessible to me.
First, though a minor point, but where does the name *YellowFin* come from?
For the presentation, the motivation in introduction is fine, but the following section about momentum operator is hard to follow. There are a lot of undefined notation. For example, what does the *convergence rate* mean (what is the measurement for convergence)? And is the *optimal accelerated rate* the same as *convergence rate* mentioned above? Also, what do you mean by *all directions* in the sentence below eq.2?
Then the paper talks about robustness properties of the momentum operator. But: first, I am not sure why the derivative of f(x) is defined as in eq.3, how is that related to the original definition of derivative?
In the following paragraph, what is *contraction*? Does it have anything to do with the paper as I didn't see it in the remaining text?
Lemma 2 seems to use the spectral radius of the momentum operator as the *robustness*. But how can it describe the robustness? More details are needed to understand this.
What it comes to Section 3, it seems to me that the authors try to use a local quadratic approximation for the original function f(x), and use the results in last section to find the optimal momentum parameter. I got confused in this section because eq.9 defines f(x) as a quadratic function. Is this f(x) the original function (non quadratic) or just the local quadratic approximation? If it is the local quadratic approximation, how is it correlated to the original function? It seems to me that the authors try to say if h and C are calculated from the original function, then this f(x) is a local quadratic approximation? If what I think is correct, I think it would be important to show this.
Also, the objective function in SingleStep algorithm seems to come from eq.13, but I failed to get the exact reasoning.
Overall, I think this is an interesting paper, but the presentation is too fuzzy to get it evaluated.


With the increasing interest in deeper understanding of the loss surface of many non-convex deep models, this paper presents a unifying framework to study the local/global optima equivalence of the optimization problems arising from training of such non-convex models. Using the local openness property of the underlying training models, we provide simple sufficient conditions under which any local optimum of the resulting optimization problem is globally optimal. We first completely characterize the local openness of matrix multiplication mapping in its range. Then we use our characterization to: 1) show that every local optimum of two layer linear networks is globally optimal. Unlike many existing results in the literature, our result requires no assumption on the target data matrix Y , and input data matrix X. 2) develop almost complete characterization of the local/global optima equivalence of multi-layer linear neural networks. We provide various counterexamples to show the necessity of each of our assumptions. 3) show global/local optima equivalence of non-linear deep models having certain pyramidal structure. Unlike some existing works, our result requires no assumption on the differentiability of the activation functions and can go beyond "full-rank" cases.

Summary: The paper focuses on the characterization of the landscape of deep neural networks; i.e., when and why local minima are global, what are the conditions for saddle critical points, etc. The paper covers a somewhat wide range of deep nets (from shallow with linear activation to deeper with non-linear activation); it focuces only on feed forward neural networks.
As the authors state, this paper provides a unifying perspective to the subject (it justifies the results of others through this unifying theory, but also provides new results; e.g., there are results that do not depend on assumptions on the target data matrix Y).
Originality: The paper provides similar results to previous work, while removing some of the assumptions made in previous work. In that sense, the originality of the results is weak, but definitely there is some novelty in the methodology used to get to these results. Thus, I would say original.
Importance: The paper deals with the important problem of when and why training algorithms might get to global/local/saddle critical points. While there are no direct connections with generalization properties, characterizing the landscape of neural networks is an important topic to make further steps into better understanding of deep learning. It will attract some attention at the conference.
Clarity: The paper is well-written - some parts need improvement, but overall I'm satisfied with the current version.
Comments:
1. If problem (4) is not considered at all in this paper (in its full generality that considers matrix completion and matrix sensing as special cases), then the authors could just start with the model in (5).
2. Remark 1 has a nice example - could this example be shown with Y not being the all-zeros vector?
3. In section 5, the authors make a connection with the work of Ge et al. 2016. They state that the problems in (10)-(11) constitute generalizations of the symmetric matrix completion case, considered in Ge et al. 2016. However, in that work, the main difficulty of proving global optimality comes from the randomness of the sampling mask operator (which introduces the notion of incoherence and requires results in expectation). It is not clear, and maybe it is an overstatement, that the results in section 5 generalize that work. If that is the case, could the authors describe this a bit further?


Published as a conference paper at ICLR 2018 DEBIASING EVIDENCE APPROXIMATIONS: ON IMPORTANCE-WEIGHTED AUTOENCODERS AND JACKKNIFE VARIATIONAL INFERENCE
The importance-weighted autoencoder (IWAE) approach of Burda et al. (2015) defines a sequence of increasingly tighter bounds on the marginal likelihood of latent variable models. Recently, Cremer et al. (2017)  reinterpreted the IWAE bounds as ordinary variational evidence lower bounds (ELBO) applied to increasingly accurate variational distributions. In this work, we provide yet another perspective on the IWAE bounds. We interpret each IWAE bound as a biased estimator of the true marginal likelihood where for the bound defined on K samples we show the bias to be of order O(K −1 ). In our theoretical analysis of the IWAE objective we derive asymptotic bias and variance expressions. Based on this analysis we develop jackknife variational inference (JVI), a family of bias-reduced estimators reducing the bias to O(K −(m+1) ) for any given m < K while retaining computational efficiency. Finally, we demonstrate that JVI leads to improved evidence estimates in variational autoencoders. We also report first results on applying JVI to learning variational autoencoders.

[After author feedback]
I think this is an interesting paper and recommend acceptance. My remaining main comments are described in the response to author feedback below.
[Original review]
The authors introduce jackknife variational inference (JVI), a method for debiasing Monte Carlo objectives such as the importance weighted auto-encoder. Starting by studying the bias of the IWAE bound for approximating log-marginal likelihood, the authors propose to make use of debiasing techniques to improve the approximation. For the binarized MNIST the authors show improved approximations given the same number of samples from the auxiliary distribution q(z|x).
JVI seems to be an interesting extension of, and perspective on, the IWAE bound (and other Monte Carlo objectives). Some questions and comments:
* The Cremer et al. (2017) paper contains some errors when interpreting the IWAE bound as a standard ELBO with a more flexible variational approximation distribution. For example eq. (1) in their paper does not correspond to an actual distribution, it is not properly normalized. This makes the connection in their section 2.1 unclear. I would suggest citing the following paper instead for this connection and the relation to importance sampling (IS):
Naesseth, Linderman, Ranganath, Blei, "Variational Sequential Monte Carlo", 2017.
* Regarding the analysis of the IWAE bound the paper by Rainforth et al. (2017) mentioned in the comments seems very relevant. Also, because of the strong connection between IWAE and IS detailed in the Naesseth et al. (2017) paper it is possible to make use of a standard Taylor approximation/delta methods to derive Prop. 1 and Prop. 2, see e.g. Robert & Casella, "Monte Carlo Statistical Methods" or Liu's "Monte Carlo Strategies for Scientific Computing".
* It could be worth mentioning that the JVI objective function is now no longer (I think?) a lower bound to the log-evidence.
* Could the surprising issue (IWAE-learned, JV1-evaluated being better than JV1-learned, JV1-evaluated) in Table 1 be because of different local optima?
* Also, we can easily get unbiased estimates of the evidence p(x) using IS and optimize this objective wrt to model parameters. The proposal parameters can be optimized to minimize variance, how do you think this compares to the proposed method?
Minor comments:
* p(x) -> p_\theta(x)
* In the last paragraph of section 1 it seems like you claim that the expressiveness of p_\theta(x|z) is a limitation of VAE. It was a bit unclear to me what was actually a general limitation of maximum likelihood versus the approximation based on VAEs.
* Last paragraph of section 1, "strong bound" -> "tight bound"
* Last paragraph of section 2, citation missing for DVI


While existing graph embedding models can generate useful embedding vectors for graph-related tasks, what valuable information can be jointly learned from a graph embedding model is less discussed. In this paper, we consider the possibility of detecting critical structures by a graph embedding model. We propose Ego-CNN to embed graphs, which works in a local-to-global manner to take advantages of CNNs that gradually expands the detectable local regions on the graph as the network depth increases. Critical structures can be detected if Ego-CNN is combined with a supervised task model. We show that Ego-CNN is (1) competitive to state-of-the-art graph embeddings models, (2) can work nicely with CNNs visualization techniques to show the detected structures, and (3) is efficient and can incorporate with scale-free priors, which commonly occurs in social network datasets, to further improve the training efficiency.

Dear authors,
Thank you for your contribution to ICLR. The problem you are addressing with your work is important. Your paper is well-motivated. Detecting and exploiting "critical structures" in graphs for graph classification is indeed something that is missing in previous work.
After the introduction you discuss some related work. While I really appreciate the effort you put into this section (including the figures etc.) there are several inaccuracies in the portrayal of existing methods. Especially the comparison to Patchy-san is somewhat vague. Please make sure that you clearly state the differences between patchy-san and Ego-CNNs. What exactly is it that Patchy cannot achieve that you can. I believe I understood what the advantages of the proposed method are but it took a while to get there. Just one example to show you what I mean; you write:
"The reason why the idea of Patchy-San fails to generalize into multiple layers is that its definition of
neighborhood, which is based on adjacency matrix, is not static and may not corresponding to local
regions in the graph. "
It is very difficult to understand what it is that you want to express with the above sentence. Its definition of neighborhood is based on adjacency matrix - what does that mean? A neighborhood is a set of nodes, no? Why is it that their definition of neighborhood might not correspond to local regions? In general, you should try to be more precise and concise when discussing related work.
Section 3, the most important section in the paper that describes the proposed Ego-CNN approach, should also be written more clearly. For instance, it would be good if you could define the notion of an "Ego-Convolution layer." You use that term without properly defining it and it is difficult to make sense of the approach without understanding it. Also, you contrast your approach with patchy and write that "Our main idea is to use the egocentric design, i.e. the neighborhood at next
layer is defined on the same node." Unfortunately, I find it difficult to understand what this means. In general, this section is very verbose and needs a lot more work. This is at the moment also the crucial shortcoming of the paper. You should spent more time on section 3 and formally and more didactically introduce your approach. In my opinion, without a substantial improvement of this section, the paper should not be accepted.
The experiments are standard and compare to numerous existing state of the art methods. The data sets are also rather standard. The one thing I would add to the results are the standard deviations. It is common to report those. Also, in the learning for graph structured data, the variance can be quite high and providing the stddev would at least indicate how significant the improvements are.
I also like the visualizations and the discussion of the critical structures found in some of the graphs.
Overall, I think this is an interesting paper that has a lot of potential. The problem, however, is that the presentation of the proposed approach is verbose and partially incomprehensible. What exactly is different to existing approaches? What exactly is the formal definition of the method? All of this is not well presented and, in my opinion, requires another round of editing and reviews.


Published as a conference paper at ICLR 2018 IMPROVING GANS USING OPTIMAL TRANSPORT
We present Optimal Transport GAN (OT-GAN), a variant of generative adversarial nets minimizing a new metric measuring the distance between the generator distribution and the data distribution. This metric, which we call mini-batch energy distance, combines optimal transport in primal form with an energy distance defined in an adversarially learned feature space, resulting in a highly discriminative distance function with unbiased mini-batch gradients. Experimentally we show OT-GAN to be highly stable when trained with large mini-batches, and we present state-of-the-art results on several popular benchmark problems for image generation.

There have recently been a set of interesting papers on adapting optimal transport to GANs. This makes a lot of sense. The paper makes some very good connections to the state of the art and those competing approaches. The proposal makes sense from the generative standpoint and it is clear from the paper that the key contribution is the design of the transport cost. I have two main remarks and questions.
* Regarding the transport cost, the authors say that the Euclidean distance does not work well. Did they try to use normalised vectors with the squared Euclidean distance ? I am asking this question because solving the OT problem with cost defined as in c_eta is equivalent to using a *normalized squared* Euclidean distance in the feature space defined by v_eta. If the answer is yes and it did not work, then there is indeed a real contribution to using the DNN. Otherwise, the contribution has to be balanced. In either case, I would have been happy to see numbers for comparison.
* The square mini batch energy distance looks very much like a maximum mean discrepancy criterion (see the work of A. Gretton), up to the sign, and also to regularised approached to MMD optimisation (see the paper of Kim, NIPS'16 and references therein). The MMD is the solution of an optimisation problem which, I suppose, has lots of connections with the dual Wasserstein GAN. The authors should elaborate on the relationships, and eventually discuss regularisation in this context.


Deep learning on graphs has become a popular research topic with many applications. However, past work has concentrated on learning graph embedding tasks only, which is in contrast with advances in generative models for images and text. Is it possible to transfer this progress to the domain of graphs? We propose to sidestep hurdles associated with linearization of such discrete structures by having a decoder output a probabilistic fully-connected graph of a predefined maximum size directly at once. Our method is formulated as a variational autoencoder. We evaluate on the challenging task of conditional molecule generation.

This paper studies the problem of learning to generate graphs using deep learning methods. The main challenges of generating graphs as opposed to text or images are said to be the following:
(a) Graphs are discrete structures, and incrementally constructing them would lead to non-differentiability (I don't agree with this; see below)
(b) It's not clear how to linearize the construction of graphs due to their symmetries. Based on this motivation, the paper decides to generate a graph in "one shot", directly  outputting node and edge existence probabilities, and node attribute vectors.
A graph is represented by a soft adjacency matrix A (entries are probability of existence of an edge), an edge attribute tensor E (entries are probability of each edge being one of d_e discrete types), and a node attribute matrix F, which has a node vector for each  potential node. A cross entropy loss is developed to measure the loss between generated A, E, and F and corresponding targets.
The main issue with training models in this formulation is the alignment of the generated graph to the ground truth graph. To handle this, the paper proposes to use a simple graph  matching algorithm (Max Pooling Matching) to align nodes and edges. A downside to the algorithm is that it has complexity O(k^4) for graphs with k nodes, but the authors argue that this is not a problem when generating small graphs. Once the best correspondence is found, it is treated as constant and gradients are propagated appropriately.
Experimentally, generative models of chemical graphs are trained on two datasets. Qualitative results and ELBO values are reported as the dimensionality of the embeddings is varied. No baseline results are presented. A further small set of experiments evaluates the quality of the matching algorithm on a synthetic setup.
Strengths:
- Generating graphs is an interesting problem, and the proposed approach seems like an easy-to-implement, mostly reasonable way of approaching the problem.
- The exposition is clear (although a bit more detail on MPM matching would be appreciated)
However, there are some significant weaknesses. First, the motivation for one-shot graph construction is not very strong:
- I don't understand why the non-differentiability argued in (a) above is an issue. If training uses a maximum likelihood objective, then we should be able to decompose the generation of a graph into a sequence of decisions and maximize the sum of the logprobs of the conditionals. People do this all the time with sequence data and non-differentiability is not an issue.
- I also don't agree that the one shot graph construction sidesteps the issue of how to linearize the construction of a graph. Even after doing so, the authors need to solve a matching problem to resolve the alignment issue. I see this as equivalent to choosing an order in which to linearize the order of nodes and edges in the graph.
Second, the experiments are quite weak. No baselines are presented to back up the claims motivating the formulation. I don't know how to interpret whether the results are good or bad. I would have at least liked to see a comparison to a method that generated SMILES format in an autoregressive manner (similar to previous work on chemical graph generation), and would ideally have liked to see an attempt at solving the alignment problem within an autoregressive formulation (e.g., by greedily constructing the alignment as the graph was generated). If one is willing to spend O(k^4) computation to solve the alignment problem, then there seem like many possibilities that could be easily applied to the autoregressive formulation. The authors might also be interested in a concurrent ICLR submission that approaches the problem from an autoregressive angle (https://openreview.net/pdf?id=Hy1d-ebAb).
Finally, I would have expected to see a discussion and comparison to "Learning Graphical State Transitions" (Johnson, 2017). Please also don't make statements like "To the best of our knowledge, we are the first to address graph generation using deep learning." This is very clearly not true. Even disregarding Johnson (2017), which the authors claim to be unaware of, I would consider approaches that generate SMILES format (like Gomez-Bombarelli et al) to be doing graph generation using deep learning.
Overall, the paper is about an interesting subject, but in my opinion the execution isn't strong enough to warrant publication at this point.


Deep neural networks (DNNs) have been found to be vulnerable to adversarial examples resulting from adding small-magnitude perturbations to inputs. Such adversarial examples can mislead DNNs to produce adversary-selected results. Different attack strategies have been proposed to generate adversarial examples, but how to produce them with high perceptual quality and more efficiently requires more research efforts. In this paper, we propose AdvGAN to generate adversarial examples with generative adversarial networks (GANs), which can learn and approximate the distribution of original instances. For AdvGAN, once the generator is trained, it can generate adversarial perturbations efficiently for any instance, so as to potentially accelerate adversarial training as defenses. We apply AdvGAN in both semi-whitebox and black-box attack settings. In semi-whitebox attacks, there is no need to access the original target model after the generator is trained, in contrast to traditional white-box attacks. In black-box attacks, we dynamically train a distilled model for the black-box model and optimize the generator accordingly. Adversarial examples generated by AdvGAN on different target models have high attack success rate under state-of-the-art defenses compared to other attacks. Our attack has placed the first with 92.76% accuracy on a public MNIST black-box attack challenge (Mądry et al., 2017b).

I thank the authors for the thoughtful response and rebuttal. The authors have substantially updated their manuscript and improved the presentation.
Re: Speed. I brought up this point because this was a bulleted item in the Introduction in the earlier version of the manuscript. In the revised manuscript, this bullet point is now removed. I will take this point to be moot.
Re: High resolution. The authors point to recent GAN literature that provides some first results with high resolution GANs but I do not see quantitative evidence in the high resolution setting for this paper. (Figure 4 provides qualitative examples from ImageNet but no quantitative assessment.)
Because the authors improved the manuscript, I upwardly revised my score to 'Ok but not good enough - rejection'. I am not able to accept this paper because of the latter point.
==========================
The authors present an interesting new method for generating adversarial examples. Namely, the author train a generative adversarial network (GAN) to adversarial examples for a target network. The authors demonstrate that the network works well in the semi-white box and black box settings.
The authors wrote a clear paper with great references and clear descriptions.
My primary concern is that this work has limited practical benefit in a realistic setting. Addressing each and every concern is quite important:
1) Speed. The authors suggest that training a GAN provides a speed benefit with respect to other attack techniques. The FGSM method (Goodfellow et al, 2015) is basically 1 inference operation and 1 backward operation. The GAN is 1 forward operation. Granted this results in a small difference in timing 0.06s versus 0.01s, however it would seem that avoiding a backward pass is a somewhat small speed gain.

Furthermore, I would want to question the practical usage of having an 'even faster' method for generating adversarial examples. What is the reason that we need to run adversarial attacks 'even faster'? I am not aware of any use-cases, but if there are some, the authors should describe the rationales at length in their paper.
2) High spatial resolution images. Previous methods, e.g. FGSM, may work on arbitrarily sized images. At best, GANs generate reasonable images that are lower resolutions (e.g. < 128x128). Building GAN's that operate above-and-beyond moderate spatial resolution is an open research topic. The best GAN models for generating high resolution images are  difficult to train and it is not clear if they would work in this setting. Furthermore, images with even higher resolutions, e.g. 512x512, which is quite common in ImageNet, are difficult to synthesizes using current techniques.
3) Controlling the amount of distortion. A feature of previous optimization based methods is that a user may specify the amount of perturbation (epsilon). This is a key feature if not requirement in an adversarial perturbation because a user might want to examine the performance of a given model as a function of epsilon. Performing such an analysis with this model is challenging (i.e. retraining a GAN) and it is not clear if a given image generated by a GAN will always achieve a given epsilon perturbation/
On a more minor note, the authors suggest that generating a *diversity* of adversarial images is of practical import. I do not see the utility of being able to generate a diversity of adversarial images. The authors need to provide more justification for this motivation.


Published as a conference paper at ICLR 2018 DEEP SENSING: ACTIVE SENSING USING MULTI- DIRECTIONAL RECURRENT NEURAL NETWORKS
For every prediction we might wish to make, we must decide what to observe (what source of information) and when to observe it. Because making observations is costly, this decision must trade off the value of information against the cost of observation. Making observations (sensing) should be an active choice. To solve the problem of active sensing we develop a novel deep learning architecture: Deep Sensing. At training time, Deep Sensing learns how to issue predictions at various cost-performance points. To do this, it creates a different presentation at each of a variety of different performance levels, each associated with a particular set of measurement rates (costs). This requires learning how to estimate the value of real measurements vs. inferred measurements, which in turn requires learning how to infer missing (unobserved) measurements. To infer missing measurements, we develop a Multi-directional Recurrent Neural Network (M-RNN). An M-RNN differs from a bi-directional RNN in that it sequentially operates across streams in addition to within streams, and because the timing of inputs into the hidden layers is both lagged and advanced. At runtime, the operator prescribes a performance level or a cost constraint, and Deep Sensing determines what measurements to take and what to infer from those measurements, and then issues predictions. To demonstrate the power of our method, we apply it to two real-world medical datasets with significantly improved performance.

This is a very interesting submission that takes an interesting angle on clinical time series modeling, namely, actively choosing when to measure while simultaneously attempting to impute missing measurements and predict outcomes of interest. The proposed solution formulates everything as a giant learning problem that involves learning (a) an interpolation function that predicts a missing measurement from its past and present, (b) an imputation function that predicts a missing measurement from other variables at the same time step, (c) a prediction function that predicts outcomes of interest, including forecasting future measurements, (d) an error estimation function that estimates error of the forecasts in (c). These four pieces are then used in combination with a heuristic to decide when certain variables should be measured. This framework is used with a GRU-RNN architecture and in experiments with two datasets, outperforms a number of strong baselines.
I am inclined toward accepting this paper due to the significance of the problem, the ingenuity of their proposed approach, and the strength of the empirical results. However, I think that there is a lot of room for improvement in the current manuscript, which is difficult to read and fully grasp. This will lessen its impact in the long run, so I encourage the authors to strive to make it clearer. If they succeed in improving it during the review period, I will gladly raise my score.
NOTE: please do a thorough editorial pass for the next version -- I found at least one typo in the references (Yu, et al. "Active sensin.")
QUALITY
This is solid research, and I have few complaints about the work itself (most of my feedback will focus on clarity). I will list some strengths (+) and weaknesses (-) below and try to provide actionable feedback:
+ Very important problem that receives limited attention from the community
+ I like the formulation of active sensing as a prediction loss optimization problem
+ The learning problem is pretty intuitive and is well-suited to deep learning architectures since it yields a differentiable (albeit complex) loss function
+ The results speak for themselves -- for adverse event prediction in the MIMIC-III task, DS improves upon the nearest baseline by almost 9 points in AUC! More interestingly, using Deep Sensing to create a "resampled" version of the data set improves the performance of the baselines. It also achieves much more accurate imputation than standard approaches.
- The proposed approach is pretty complex, and it's unclear what is the relative contribution of each component. I think it is incumbent to do an ablation study where different components are removed to see how performance degrades, if at all. For example, how would the model perform with interpolation but not imputation? Is bidirectional interpolation necessary, or would forward interpolation work sufficiently well (the obvious disadvantage of the bidirectional approach is the need to rerun inference at each new time step). Is it necessary to use both the actual AND predicted measurements as inputs (what if we instead used actual measurements when available and predicted otherwise)?
- The experiments are thorough with a nice selection of baselines, but I wonder if perhaps Futoma, et al. [1] would be a stronger baseline than Choi, Che, or Lipton. They showed improvements over similar magnitude over baselines for predicting sepsis, and their approach (a differentiable GP-approximating layer) is conceptually simpler and has other benefits. I think it could be combined with the active sensing framework in this paper.
- The one question this framework appears incapable of answering in a straightforward manner is WHEN the next set of measurements should be made. One could imagine a heuristic in which predictive loss/gain are assessed at different points in the future, but the search space will be huge, particularly if one wants to optimize over measurements at different points, e.g., maybe the optimal strategy is to take roughly hourly vitals but no labs until 12 hours from now. Indeed, it might be impossible to train such a model properly since the sampling times in the available training data are highly biased.
- One thing potentially missing from this paper is a theoretical analysis to understand and analyze its behavior and performance. My very superficial analysis is that the prediction loss/gain framework is related to minimizing entropy and that the heuristic for choosing which variables to measure is a greedy search. A theoretical treatment to understand whether and how this approach might be sub-optimal would be very desirable.
- Are the measurement and prediction "confidence intervals" proper confidence intervals (in the formal statistical sense)? I don't think so -- I wonder if there are alternatives for measuring uncertainty (formal CIs or maybe a Bayesian approach?).
CLARITY
My main complaint about this paper is clarity -- it is not difficult to read per se, but it is difficult to fully grok the details of the approach and the experimental setup. From the current manuscript, I do not feel confident that I could re-implement Deep Sensing or reproduce the experiments. This is especially important in healthcare research, where there is a minor reproducibility crisis, even for resarch using MIMIC (see [2]). Of course, this can be alleviated by publishing the code and using a public benchmark [3], but it can't hurt to clarify these details in the paper itself (and to add an appendix if length is an issue).
Here are some potential areas for improvement:
- The structure of the paper is a bit weird. In particular section 2 (pages 2-4) seems to be a grab bag of miscellaneous topics, at least by the headers. I think the content is fine -- perhaps section 2 can be renamed as "Background," subsection 2.1 renamed as "Notation," and subsection 2.2 renamed as "Problem Formulation" (or similar). I'd just combine subsection 2.3 with the previous one and explain that Figure 1 illustrates the problem formulation.
- The active sensing procedure (subsection 2.2, page 3, equation 1 and the equations just above) is unclear. How are the minimization and maximization performed (gradient descent, line search, etc.)? How is the search for the subset of measurement variables performed (greedy search)? The latter is a discrete search, and I doubt it's, e.g., submodular, so it must be a nontrivial optimization.
- Related, I'm a little confused about equation 1: C_T is the set of variables that should be measured, but C_T is being used to index prediction targets -- is this a typo?
- The related work section is pretty extensive, but I wonder if it should also include work on active learning (Bayesian active learning, in particular, has been applied to sensing), submodular optimization (for sensor placement, which can be thought of as a spatial version of active sensing), and reinforcement learning.
- I don't understand how the training data for the interpolation and imputation functions are constructed. I *think* that is what is described in the Adaptive Sampling subsection on page 8, but that is unclear. The word "representations" is used here, but that's an overloaded term in machine learning, and its meaning here is unclear from context. It appears that maybe there's an iterative procedure in which we alternate between training a model and then resampling the data using the model -- starting with the full data set.
- The distinction between training and inference is not clear to me, at least with respect to the active sensing component. Is selective sampling performed during training? If so, what happens if the model elects to sample a variable at time t that is not actually measured in the data?
- I don't follow subsection 4.2 (pages 8-9) at all -- what is it describing? If by "runtime" the authors refer to the computational complexity of the algorithm, then I would expect a Big-O analysis (none is provided -- it's just a rather vague discussion of what happens). I'd recommend removing this entire subsection and replacing it with, e.g., an Algorithm figure with pseudocode, as a more succinct description.
- For the experiments, the authors provide insufficient detail about the data and task setup. Since MIMIC is publicly available, then readers ought (hypothetically) to be able to reproduce the experiments, but that is not currently possible. As an example, what adverse events are being predicted? How are they defined?
- Figure 4 is nice, but it's not immediately obvious what the connection between observation rate and sampling cost. The authors should explain how a given observation rate is encoded as cost in the loss function.
ORIGINALITY
While active sensing is not a new research topic per se, there has been very limited research into the specific question of choosing what clinical variables to measure when in the context of a given prediction problem. This is a topic that (in my experience) is frequently discussed but rarely studied in clinical informatics circles. Hence, this is a very original line of inquiry, and the prediction loss/gain framing is a unique angle.
SIGNIFICANCE
I anticipate this paper will generate significant interest and follow-up work, at least among clinical informaticists and machine learning + health researchers. The main blockers to a significant impact are the clarity of writing issues listed above -- and if the authors fail to publish their code.
REFERENCES
[1] Futoma, et al. An Improved Multi-Output Gaussian Process RNN with Real-Time Validation for Early Sepsis Detection. MLHC 2017.
[2] Johnson, et al. Reproducibility in critical care: a mortality prediction case study. MLHC 2017
[3] Harutyunyan, et al. Multitask Learning and Benchmarking with Clinical Time Series Data. arXiv.


Published as a conference paper at ICLR 2018 COUNTERING ADVERSARIAL IMAGES USING INPUT TRANSFORMATIONS
This paper investigates strategies that defend against adversarial-example attacks on image-classification systems by transforming the inputs before feeding them to the system. Specifically, we study applying image transformations such as bit-depth reduction, JPEG compression, total variance minimization, and image quilting before feeding the image to a convolutional network classifier. Our experiments on ImageNet show that total variance minimization and image quilting are very effective defenses in practice, in particular, when the network is trained on transformed images. The strength of those defenses lies in their non-differentiable nature and their inherent randomness, which makes it difficult for an adversary to circumvent the defenses. Our best defense eliminates 60% of strong gray-box and 90% of strong black-box attacks by a variety of major attack methods.

To increase robustness to adversarial attacks, the paper fundamentally proposes to transform an input image before feeding it to a convolutional network classifier. The purpose of the transformation is to erase the high-frequency signals potentially embedded by an adversarial attack.
Strong points:
* To my knowledge, the proposed defense strategy is novel (even if the idea of transformation has been introduced at https://arxiv.org/abs/1612.01401).
* The writing is reasonably clear (up to the terminology issues discussed among the weak points), and introduces properly the adversarial attacks considered in the work.
* The proposed approach really helps in a black-box scenario (Figure 4). As explained below, the presented investigation is however insufficient to assess whether the proposed defense helps in a true white-box scenario.
Weak points:
* The black-box versus white-box terminology is not appropriate, and confusing. In general, black-box means that the adversary ignores everything from the decision process. Hence, in this case, the adversary does not know about the classification model, nor the defensive method, when used. This corresponds to Figure 3. On the contrary, white-box means that the adversary knows everything about the classification method, including the transformation implemented to make it more robust to attacks. Assimilating the parameters of the transform to a secret key is not correct because those parameters could be inferred by presenting many image samples to the transform and looking at the outcome of the transformation (which is supposed to be available in a 'white-box' paradigm) for those samples.
* Using block diagrams would definitely help in presenting the training/testing and attack/defense schemes investigated in Figure 3, 4, and 5.
* The paper does not discuss the impact of the denfense strategy on the classification performance in absence of adversity.
* The paper lacks of positioning with respect to recent related works, e.g. 'Adversary Resistant Deep Neural Networks with an Application to Malware Detection' in KDD 2017, or 'Building Adversary-Resistant Deep Neural Networks without
Security through Obscurity' at https://arxiv.org/abs/1612.01401.
* In a white-box scenario, the adversary knows about the transformation and the classification model. Hence, an effective and realistic attack should exploit this knowledge. Designing an attack in case of a non differentiable transformation is obviously not trivial since back-propagation can not be used. However, since the proposed transformation primarily aim at removing the high frequency pattern induced by the attack, one could for example design an attack that account for a (linear and differentiable) low-pass filter transformation. Another example of attack that account for transformation knowledge (and would hopefully be more robust than the attacks considered in the manuscript) could be one that alternates between a conventional attack and the transformation.
* If I understand correctly, the classification model considered in Figure 3 has been trained on original images, while the one in Figure 4 has been trained on transformed images. However, in absence of attack, they both achieve 76% accuracy. Is it correct? Does it mean that the transformation does not affect the classification accuracy at all?
Overall, the works investigates an interesting idea, but lacks maturity to be accepted. Therefore, I would only recommend acceptation if room.
Minor issues:
Typo on p7: to change*s*
Clarify poor formulations:
* p1: 'enforce model-specific strategies that enforce model properties such as invariance and smoothness via the learning algorithm or regularization schemes'.
* p1: 'too simple to remove adversarial perturbations from input images sufficiently'