Batch Normalization (BatchNorm) is an extremely useful component of modern neural network architectures, enabling optimization using higher learning rates and achieving faster convergence. In this paper, we use mean-field theory to analytically quantify the impact of BatchNorm on the geometry of the loss landscape for multi-layer networks consisting of fully-connected and convolutional layers. We show that it has a flattening effect on the loss landscape, as quantified by the maximum eigenvalue of the Fisher Information Matrix. These findings are then used to justify the use of larger learning rates for networks that use BatchNorm, and we provide quantitative characterization of the maximal allowable learning rate to ensure convergence. Experiments support our theoretically predicted maximum learning rate, and furthermore suggest that networks with smaller values of the BatchNorm parameter γ achieve lower loss after the same number of epochs of training.

Interesting application of MFT on FIM to understand Batch Normalization
This paper applies mean field analysis to networks with batch normalization layers. Analyzing maximum eigenvalue of the Fisher Information Matrix, the authors provide theoretical evidence of allowing higher learning rates and faster convergence of networks with batch normalization.
The analysis reduces to providing lower bound for maximum eigenvalue of FIM using mean-field approximation. Authors provide lower bound of the maximum eigenvalue in the case of fully-connected and convolutional networks with batch normalization layers. Lastly authors observe empirical correlation between smaller \gamma and lower test loss.
Pro:
 - Clear result providing theoretical ground for commonly observed effects.
 - Experiments are simple but illustrative. It is quite surprising how well the maximum learning rate prediction matches with actual training performance curve.

Con:
 - While mean field analysis a-priori works in the limit where networks width goes to infinity for fixed dataset size, the analysis of Fisher and Batch normalization need asymptotic limit of dataset size.
 - Although some interesting results are provided. The content could be expanded further for conference submission. The prediction on maximum learning rate is interesting and the concrete result from mean field analysis
 - While correlation between batch norm \gamma parameter and test loss is also interesting, the provided theory does not seem to provide good intuition about the phenomenon.
Comments:
- The theory provides the means to compute lower bound of maximum eigenvalue of FIM using mean-field theory. In Figure 1, is \bar \lambda_{max} computed using the theory or empirically computed on the actual network? It would be nice to make this clear.
- In Figure 2, the observed \eta_*/2 of dark bands in heatmap is interesting. While most of networks without Batch Norm, performance is maximized using learning rates very close to maximal value, often networks using batch norm the learning rate with maximal performance is not the maximal one and it would be interesting to provide theoretical
- I feel like section 3.2 should cite Xiao et al (2018). Although this paper is cited in the intro, the mean field analysis of convolutional layers was first worked out in this paper and should be credited.


ADAPTIVE INPUT REPRESENTATIONS FOR NEURAL LANGUAGE MODELING
We introduce adaptive input representations for neural language modeling which extend the adaptive softmax of Grave et al. (2017) to input representations of variable capacity. There are several choices on how to factorize the input and output layers, and whether to model words, characters or sub-word units. We perform a systematic comparison of popular choices for a self-attentional architecture. Our experiments show that models equipped with adaptive embeddings are more than twice as fast to train than the popular character input CNN while having a lower number of parameters. On the WIKITEXT-103 benchmark we achieve 18.7 perplexity, an improvement of 10.5 perplexity compared to the previously best published result and on the BILLION WORD benchmark, we achieve 23.02 perplexity.

The authors extend an existing approach to adaptive softmax classifiers used for the output component of neural language models into the input component, once again allowing tying between the embedding and softmax. This fills a significant gap in the language modeling architecture space, and the perplexity results bear out the advantages of combining adaptively-sized representations with weight tying. While the advance is in some sense fairly incremental, the centrality of unsupervised language modeling to modern deep NLP (ELMo, BERT, etc.) implies that perplexity improvements as large as this one may have meaningful downstream effects on performance on other tasks. Some things I noticed:
- One comparison that I believe is missing (I could be misreading the tables) is comparing directly to Merity et al.'s approach (adaptive softmax but fixed embedding/softmax dimension among the bands). Presumably you're faster, but is there a perplexity trade-off?
- The discussion/explanation of the differing performance of tying or not tying each part of the embedding weights for the different datasets is confusing; I think it could benefit from tightening up the wording but mostly I just had to read it a couple times. Perhaps all that's complicated is the distinction between embedding and projection weights; it would definitely be helpful to be as explicit about that as possible upfront.
- The loss by frequency-bin plots are really fantastic. You could also try a scatterplot of log freq vs. average loss by individual word/BPE token.
- Do you have thoughts as to why full-softmax BPE is worse than adaptive softmax word level? That goes against the current (industry) conventional wisdom in machine translation and large-scale language modeling that BPE is solidly better than word-level approaches because it tackles the softmax bottleneck while also sharing morphological information between words.


The correct interpretation of quantifier statements in the context of a visual scene requires non-trivial inference mechanisms. For the example of "most", we discuss two strategies which rely on fundamentally different cognitive concepts. Our aim is to identify what strategy deep learning models for visual question answering learn when trained on such questions. To this end, we carefully design data to replicate experiments from psycholinguistics where the same question was investigated for humans. Focusing on the FiLM visual question answering model, our experiments indicate that a form of approximate number system emerges whose performance declines with more difficult scenes as predicted by Weber's law. Moreover, we identify confounding factors, like spatial arrangement of the scene, which impede the effectiveness of this system.

The paper analyzes the strategy that a visual question answering model (FiLM) uses to verify statements containing the quantifier "most" ("most of the dots are red"). It finds that the model is sensitive to the ratio of objects that satisfy the predicate (that are red) to objects that do not; as the ratio decreases (e.g. 10 red dots compared to 9 blue dots), the model's performance decreases too. This is consistent with human behavior.
Strengths:
* The introduction lays out an ambitious program of comparing humans to deep neural networks.
* The experimental results are interesting (although of modest scope) and support the hypothesis that the network is not counting the objects but rather is using an approximation that is sensitive to the ratio between the red and non-red items.
Weaknesses:
* The architecture of the particular model is described very briefly, and at multiple points there’s an implication that this is an investigation of “deep learning models” more generally, even though those models may vary widely. While the authors are using an existing model, they shouldn't assume that the reader has read the paper describing that model. I would like to see more discussion of whether it is at all plausible for this model to acquire the pairing strategy, compared to alternative VQA models (e.g., using relation networks).
* I found it difficult to follow the theoretical motivation for performing the work. The goal seems to be to test whether the network is performing the task in way that "if not human-like, at least is cognitively plausible". I don't understand what is meant by cognitively plausible but not human-like; perhaps an example of a cognitively implausible mechanism would help clarify this issue. Later in the same paragraph, the authors argue that "in the case of a human-centered domain like natural language, ultimately, some degree of comparability to human performance is indispensable". This assertion is not justified, and seems surprising to me; we have very useful natural language processing systems that do not perform in a way that is comparable to humans (the hedge "some degree of" is really neither here nor there). In general, I don't understand why we would want a visual question answering system that returns approximate answers -- isn't it better to have it count exactly how many red dots there are compared to non-red dots?
* The authors assume that explicit counting is not "likely to be learned by the 'one-glance' feed-forward-style neural network" evaluated in the paper. What is this statement based on? Why would a "one-glance" network have trouble counting objects? (What is a “one-glance network”?)
* Another vague concept that is used without clarification: it is argued that if the network implements something like the Approximate Number System, that shows that it can "learn and utilize higher-level concepts than mere pattern matching". What is "pattern matching" and how does it differ from "higher-level concepts"?
* Why would the pairing strategy in a neural network be affected by the clustering of the objects? I understand why a human who needs to saccade back and forth between the two groups of objects might lose track of the objects that have been paired so far, but I don't understand why that would affect the architecture in question.
Minor comments:
* Is the definition of "most" really a central piece of evidence for "the apparent importance of a cardinality concept to human cognition"? Our ability to count seems sufficient to me. Perhaps I'm not understanding what the authors have in mind here.
* Please use the terms "interpretation" and "verification" consistently.
* "One over the other strategy" -> "one strategy over the other".
* The paper is almost 9 pages long, but the contribution does not appear more substantial than a standard 8-page submission.


SPARSE DICTIONARY LEARNING BY DYNAMICAL NEURAL NETWORKS
A dynamical neural network consists of a set of interconnected neurons that interact over time continuously. It can exhibit computational properties in the sense that the dynamical system's evolution and/or limit points in the associated state space can correspond to numerical solutions to certain mathematical optimization or learning problems. Such a computational system is particularly attractive in that it can be mapped to a massively parallel computer architecture for power and throughput efficiency, especially if each neuron can rely solely on local information (i.e., local memory). Deriving gradients from the dynamical network's various states while conforming to this last constraint, however, is challenging. We show that by combining ideas of top-down feedback and contrastive learning, a dynamical network for solving the 1 -minimizing dictionary learning problem can be constructed, and the true gradients for learning are provably computable by individual neurons. Using spiking neurons to construct our dynamical network, we present a learning process, its rigorous mathematical analysis, and numerical results on several dictionary learning problems.

The authors study sparse coding models in which unit activations minimize a cost that combines: 1) the error between a linear generative model and the input data; an 2) the L1 norm of unit activations themselves. They seek models in which both the inference procedure -- generating unit activations in response to each input data example -- and the learning procedure -- updating network connections so that the inferences minimize the cost function -- are local. By "local" they mean that the update to each unit's activation, and the updates to the connection weights, rely only on information about the inputs and outputs from that unit / connection. In a biological neural network, these are the variables represented by pre- and post-synaptic action potentials and voltages, and in hardware implementations, operations on these variables can be performed without substantially coordinating between different parts of the chip, providing strong motivation for the locality constraint(s).
The authors achieve a local algorithm that approximately optimizes the sparse coding objective function by using feedback: they send the sparse coding units' activities "back" to the input layer through feedback connections. In the case where the feedback connection matrix is the transpose of the sparse coding dictionary matrix (D), the elementwise errors in the linear generative model (e.g., the non-local part of the sparse coding learning rule obtained by gradient descent) are represented by the difference between the inputs and this feedback to the input layer: that difference can be computed locally at the input units and then sent back to the coding layer to implement the updates. The feedback connections B are updated in another local process that keeps them symmetric with the feedforward weights: B= D= F^T throughout the learning process.
The authors provide several theorems showing that this setup approximately solves the sparse coding problem (again, using local information), and show via simulation that their setup shows similar evolution of the loss function during training, as does SGD on the sparse coding cost function.
I think that the paper presents a neat idea --  feedback connections are too often ignored in computational models of the nervous system, and correspondingly in machine learning. At the same time, I have some concerns about the novelty and the presentation. Those are described below:
1. The paper is unnecessarily hard to read, at least in part due to a lack of notational consistency. As just one example, with B=D, why use two different symbols for this matrix? This just makes is so that your reader needs to keep track mentally of which variable is actually which other variable, and that quickly becomes confusing. I strongly recommend choosing the simplest and most consistent notation that you can throughout the paper.
2. Other recent studies also showed that feedback connections can lead to local updates successfully training neural networks: three such papers are cited below. The first two papers do supervised learning, while the third does unsupervised learning. It would be helpful for the authors to explain the key points of novelty of their paper: application of these feedback connection ideas to sparse coding. Otherwise, readers may mistakenly get the impression that this work is the first to use feedback connections in training neural networks.
Guerguiev, J., Lillicrap, T.P. and Richards, B.A., 2017. Towards deep learning with segregated dendrites. ELife, 6, p.e22901.
Sacramento, J., Costa, R.P., Bengio, Y. and Senn, W., 2018. Dendritic cortical microcircuits approximate the backpropagation algorithm. arXiv preprint arXiv:1810.11393.
Federer, C. and Zylberberg, J., 2018. A self-organizing short-term dynamical memory network. Neural Networks.
3. Given that the performance gains of the locality (vs something like SparseNet) are given such emphasis in the paper, those should be shown from the numerical experiments. This could be quantified by runtime, or some other measure.
4. The discussion of prior work is a little misleading -- although I'm sure this is unintentional. For example, at the top of p. 3, it is mentioned that the previous local sparse coding models do not have rigorous learning objectives. But then the appendix describes the learning objectives, and the approximations, made in the prior work. I think that the introduction should have a more transparent discussion of what was, and was not, in the prior papers, and how the current work advances the field.
5. The paper -- and especially appendix C2 -- makes strong emphasis of the importance finding local implementations of true gradient descent, as opposed to the approximations made by prior authors. I'm not sure that's such a big deal, given that Lillicrap et al. showed nicely in the paper cited below that any learning rule that is within 90 degrees of true gradient descent will still minimize the cost function: even if an algorithm doesn't move down the steepest path, it can still have updates that always move "downhill", and hence minimize the loss function. Consequently, I think that some justification is needed showing that the current model, being closer to true gradient descent, really outperforms the previous ones.
Lillicrap, T.P., Cownden, D., Tweed, D.B. and Akerman, C.J., 2016. Random synaptic feedback weights support error backpropagation for deep learning. Nature communications, 7, p.13276.


We prove, under two sufficient conditions, that idealised models can have no adversarial examples. We discuss which idealised models satisfy our conditions, and show that idealised Bayesian neural networks (BNNs) satisfy these. We continue by studying near-idealised BNNs using HMC inference, demonstrating the theoretical ideas in practice. We experiment with HMC on synthetic data derived from MNIST for which we know the ground-truth image density, showing that nearperfect epistemic uncertainty correlates to density under image manifold, and that adversarial images lie off the manifold in our setting. This suggests why MC dropout, which can be seen as performing approximate inference, has been observed to be an effective defence against adversarial examples in practice; We highlight failure-cases of non-idealised BNNs relying on dropout, suggesting a new attack for dropout models and a new defence as well. Lastly, we demonstrate the defence on a cats-vs-dogs image classification task with a VGG13 variant.

The paper studies the adversarial robustness of Bayesian classifiers. The authors state two conditions that they show are provably sufficient for "idealised models" on "idealised datasets" to not have adversarial examples. (In the context of this paper, adversarial examples can either be nearby points that are classified differently with high confidence or points that are "far" from training data and classified with high confidence.) They complement their results with experiments.
I believe that studying Bayesian models and their adversarial robustness is an interesting and promising direction. However I find the current paper lacking both in terms of conceptual and technical contributions.
They consider "idealized Bayesian Neural Networks (BNNs)" to be continuous models with a confidence of 1.0 on the training set. Since these models are continuous, there exists an L2 ball of radius delta_x around each point x, where the classifier has high confidence (say 0.9). This in turn defines a region D' (the training examples plus the L2 balls around them) where the classifier has high confidence. By assuming that an "idealized BNN" has low certainty on all points outside D' they argue that these idealized models do not have adversarial examples. In my opinion, this statement follows directly from definitions and assumptions, hence having little technical depth or value. From a conceptual point of view, I don't see how this argument "explains" anything. It is fairly clear that classifiers only predicting confidently on points _very_ close to training examples will not have high-confidence adversarial examples. How do these results guide our design of ML models? How do they help us understand the shortcomings of our current models?
Moreover, this argument is not directly connected to the accuracy of the model. The idealized models described are essentially only confident in regions very close to the training examples and are thus unlikely to confidently generalize to new, unseen inputs. In order to escape this issue, the authors propose an additional assumption. Namely that idealized models are invariant to a set of transformations T that we expect the model to be also invariant to. Hence by assuming that the "idealized" training set contains at least one input from each "equivalence class", the model will have good "coverage". As far as I understand, this assumption is not connected to the main theorem at all and is mostly a hand-wavy argument. Additionally, I don't see how this assumption is justified. Formally describing the set of invariances we expect natural data to have or even building models that are perfectly encoding these invariances by design is a very challenging problem that is unlikely to have a definite solution. Also, is it natural to expect that for each test input we will have a training input that is close to L2 norm to some transformation of the test input?
Another major issue is that the value of delta_x (the L2 distance around training point x  where the model assigns high confidence) is never discussed. This value is _very_ small for standard NN classifiers (this is what causes adversarial examples in the first place!). How do we expect models to deal with this issue?
The experimental results of the paper are essentially for a toy setting. The dataset considered ("ManifoldMNIST") is essentially synthetic with access to the ground-truth probability of each sample. Moreover, the results on real datasets are unreliable. When evaluating the robustness of a model utilizing dropout, using a single gradient estimation query is not enough. Since the model is randomized, it is necessary to estimate the gradient using multiple queries. By using first-order attacks on these more reliable gradient estimates, an adversary can completely bypass a dropout "defense" (https://arxiv.org/abs/1802.00420).
Overall, I find the contributions of the paper limited both technically and conceptually. I thus recommend rejection.
[UPDATE]: Given the discussion with the authors, I agree that the paper outlines a potentially interesting research direction. As such, I have increased my score from 3 to 5 (and updated the review title). I still do not find the contribution of the paper significant enough to cross the ICLR bar.
Comments to the authors:
-- You cite certain detention methods for adversarial examples (Grosse et al. (2017), Feinman et al. (2017)) that have been shown to be ineffective (that is they can be bypassed by an adaptive attacker) by Carlini and Wagner (https://arxiv.org/abs/1705.07263)
-- The organization of the paper could be improved. I didn't understand what the main contribution was until reading Theorem 1 (this is 6 pages into the paper). The introduction is fairly vague about your contributions. You should consider moving related work later in the paper (most of the discussion there is not directly related to your approach) and potentially shortening your background section.
-- How is the discussion about Gaussian Processes connected with your results?
-- Consider making the two conditions more prominent in the text.
-- In Definition 5, the wording is confusing "We define an idealized BNN to be a Bayesian idealized NN..."


NEURAL LOGIC MACHINES
We propose the Neural Logic Machine (NLM), a neural-symbolic architecture for both inductive learning and logic reasoning. NLMs exploit the power of both neural networks-as function approximators, and logic programming-as a symbolic processor for objects with properties, relations, logic connectives, and quantifiers. After being trained on small-scale tasks (such as sorting short arrays), NLMs can recover lifted rules, and generalize to large-scale tasks (such as sorting longer arrays). In our experiments, NLMs achieve perfect generalization in a number of tasks, from relational reasoning tasks on the family tree and general graphs, to decision making tasks including sorting arrays, finding shortest paths, and playing the blocks world. Most of these tasks are hard to accomplish for neural networks or inductive logic programming alone.

The paper introduces Neural Logic Machines, a particular way to combine neural networks and first order but finite logic.
The paper is very well written and structured. However, there are also some downsides.
First of all, Section 2.1 is rather simple from a logical perspective and hence it is not clear what this gets a special term. Moreover, why do mix Boolean logic (propostional logic) and first order logic? Any how to you deal with the free variables, i.e., the variables that are not bounded by a quantifier? The semantics you define later actually assumes that all free variables (in your notation) are bounded by all quantifiers since you apply the same rule to all ground instances. Given that you argue that you want a neural extension of symbolic logic ("NLM is a neural realization of (symbolic) logic machines") this has to be clarified as it would not be an extension otherwise.
Furthermore, Section 2.2 argues that we can use a MLP with a sigmoid output to encode any joint distribution. This should be proven. It particular, given that the input to the network are the marginals of the ground atoms. So this is more like a conditional distribution? Moreover, it is not clear how this is different to other approaches that encode the weight of weighted logical rule (e.g. in a MLN) using neural networks, see
e.g.
Marco Lippi, Paolo Frasconi:
Prediction of protein beta-residue contacts by Markov logic networks with grounding-specific weights.
Bioinformatics 25(18): 2326-2333 (2009)
Now of course, and this is the nice part of the present paper, by stacking several of the rules, we could directly specify that we may need a certain number of latent predicates.
This is nice but it is not argued that this is highly novel. Consider again the work by Lippi and Frasconi. We unroll a given NN-parameterized MLN for s fixed number of forward chaining steps. This gives us essentially a computational graph that could also be made differentiable and hence we could also have end2end training. The major difference seems to be that now objects are directly attached with vector encodings, which are not present in Lippi and Frasconi's approach. This is nice but also follows from Rocktaeschel and Riedel's differentiable Prolog work (when combined with Lippi and Frasconi's approach).
Moreover, there have been other combinations of tensors and logic, see e.g.
Ivan Donadello, Luciano Serafini, Artur S. d'Avila Garcez:
Logic Tensor Networks for Semantic Image Interpretation.
IJCAI 2017: 1596-1602

Here you can also have vector encodings of constants. This also holds for
Robin Manhaeve, Sebastijan Dumancic, Angelika Kimmig, Thomas Demeester, Luc De Raedt:
DeepProbLog: Neural Probabilistic Logic Programming. CoRR abs/1805.10872 (2018)
The authors should really discuss this missing related work. This should also involve
a clarification of the "ILP systems do not scale" statement. At least if one views statistical relational learning methods as an extension of ILP, this is not true. Probabilistic ILP aka statistical relational learning has been used to learn models on electronic health records, see e.g., the papers collectively discussed in
Sriraam Natarajan, Kristian Kersting, Tushar Khot, Jude W. Shavlik:
Boosted Statistical Relational Learners - From Benchmarks to Data-Driven Medicine. Springer Briefs in Computer Science, Springer 2014, ISBN 978-3-319-13643-1, pp. 1-68
So the authors should either discuss SRL and its successes, separating SRL from ILP, or they cannot argue that ILP does not scale. In the related work section, they decided to view both as ILP, and, in turn, the statement that ILP does not scale is not true. Moreover, many of the learning tasks considered have been solved with ILP, too, of course in the ILP setting. Any ILP systems have been shown to scale beyond those toy domains.
This also includes the blocks world. Here relational MDP solvers can deal e.g. with BW worlds composed of 10 blocks, resulting in MDPs with several million states. And the can compute relational policies that solve e.g. the goal on(a,b) for arbitrary number of blocks. This should be incorporated in the discussion of the introduction in order to avoid the wrong impression that existing methods just work for toy examples.
Coming back to scaling, the current examples are on rather small datasets, too, namely <12 training instances. Moreover, given that we learn a continuous approximation with a limit depth of reasoning, it is also very likely that the models to not generate well to larger test instances. So the scaling issue has to be qualified to avoid to give the wrong impression that the present paper solves this issue.
Finally, the BW experiments should indicate some more information on the goal configuration. This would help to understand whether an average number of moves of 84 is good or bad. Moreover, some hints about the MDP formulation should be provided, given that there have been relational MDPs that solve many of the probabilistic planning competition tasks. And, given that the conclusions argue that NLMs can learn the "underlying logical rules", the learned rules should actually be shown.
Nevertheless, the direction is really interesting but there several downsides that have to be addressed.


Deep learning for object classification relies heavily on convolutional models. While effective, CNNs are rarely interpretable after the fact. An attention mechanism can be used to highlight the area of the image that the model focuses on thus offering a narrow view into the mechanism of classification. We expand on this idea by forcing the method to explicitly align images to be classified to reference images representing the classes. The mechanism of alignment is learned and therefore does not require that the reference objects are anything like those being classified. Beyond explanation, our exemplar based cross-alignment method enables classification with only a single example per category (one-shot). Our model cuts the 5-way, 1-shot error rate in Omniglot from 2.1% to 1.4% and in MiniImageNet from 53.5% to 46.5% while simultaneously providing point-wise alignment information providing some understanding on what the network is capturing. This method of alignment also enables the recognition of an unsupported class (open-set) in the one-shot setting while maintaining an F1-score of above 0.5 for Omniglot even with 19 other distracting classes while baselines completely fail to separate the open-set class in the one-shot setting.

Authors argue that using average (independent) greedy matching of pixel embedding (based on 4-6 layer cnn hypercolumns) is a better metric for one-shot learning than just using final layer embedding of a 4-6 layer cnn for the whole image.  Their argument is backed by outperforming their baseline and getting competitive results on few shot learning tasks. Their method is much more computationally heavy than the baseline matching networks. In order to make training feasible, in practice they train with 90% dropout of test pixels embedding & 80% dropout of reference pixels embedding.
The caveats:
-> Using hyper-columns is related to adding residual connections. The question remains how much performance can be gained by just adding residual connections (with dropout) to the matching networks and letting the network automatically (or with a probability) choose to embed higher layers or lower ones. Adding the residual connection and just comparing the final layer embeddings is a cleaner method than ABM which  provides a richer embedding than baseline and could potentially close the performance gap between ABM and final layer matching.
->It is strictly designed for one-shot learning. It does not benefit from few shots (extra shots) and the fact that these different shots are getting classified as the same label. Vinyal et al mitigates this shortcoming by adding the FCE. However FCE is not directly applicable anymore. Author’s don’t suggest any alternatives either. Their smaller gains (or even worse than baseline without self-regularization) in the 5-shot cases is an evidence of this shortcoming.
The fact that SNAIL (TCML Mishra et al. (2017)) consistently outperforms this method puts a question mark on the significance of this work. If it was computationally feasible, authors could have used SNAIL and replaced the 64 dimensional embedding of each picture with the 10-20% hypercolumns. Essentially due to computational costs authors are sacrificing a more thorough matching system (non-greedy) for a richer embedding and they don’t get better results.
On the other hand, authors may argue that the hyper-column matching is not just about performance, whereas it also adds interpretability to why two images are categorized the same. Illustrations like fig. 3 for example shows that the model is not matching semantically similar points and can be used to debug & improve the model. While understanding why a blackbox matching network is making a mistake and improving, is  harder.
It would have been nice if authors used this added interpretability in some manner. Such as getting an idea about a regularizer, a prior, a mask, etc. and improved the performance.
I would argue for accepting this paper for two reasons.
-> Given that they beat their baseline and  they get comparable performance to sota even with a greedy matching (min-pooling followed by average pooling), is impressive. Furthermore, it is orthogonal to methods like SNAIL if the computational cost could be resolved.
-> They not only provide which image is a match but how they are matched, which could be interesting for one-shot detection as well as classification.
Question: At test/validation: do you still only categorize with 10,20% samples or do you average the full attention map for all test pixels?
Nit: The manuscript needs several passes of proofreading, spell & grammar checking. A few examples in the first couple of pages:
-> The citing format needs to be fixed (like: LSTMsRavi, there should be () around citations).
-> are not incompatible: are compatible
->incomprehensible sentence with two whiles: ABM networks outperforms these other state-of-the-art models on the open-set recognition task while in the one-shot setting by achieving a high accuracy of matching the non-open-set classes while maintaining a high F1 score for identifying samples in the open-set.
-> add dots to the end of contribution list items.
-> we first look pairwise matchings: we first look at the pairwise matchings


A Synaptic Neural Network (SynaNN) consists of synapses and neurons. Inspired by the synapse research of neuroscience, we built a synapse model with a nonlinear synapse function of excitatory and inhibitory channel probabilities. Introduced the concept of surprisal space and constructed a commutative diagram, we proved that the inhibitory probability function -log(1-exp(-x)) in surprisal space is the topologically conjugate function of the inhibitory complementary probability 1-x in probability space. Furthermore, we found that the derivative of the synapse over the parameter in the surprisal space is equal to the negative Bose-Einstein distribution. In addition, we constructed a fully connected synapse graph (tensor) as a synapse block of a synaptic neural network. Moreover, we proved the gradient formula of a cross-entropy loss function over parameters, so synapse learning can work with the gradient descent and backpropagation algorithms. In the proof-of-concept experiment, we performed an MNIST training and testing on the MLP model with synapse network as hidden layers.

Thanks for submitting your paper. It takes a lot of effort and courage to put your ideas out into the world. Sometimes the hardest work for researchers is conveying their thoughts to others in a manner in which those ideas can be understood.
With that in mind, I had an extremely difficult time following your arguments.
I noticed several things:
 - There are numerous places in the text that lack proper citation, or are cited improperly.
 - Why was there not a related methods section? I find it hard to believe that all of your ideas have no precursor.
 - When there are citations, there is usually only one text and it is quite old. For example, all of your neuroscience citations reference a work that is almost 40 years old. There have been quite a few improvements in our biological understanding as well as theoretical understanding since then. ( I make this point as a common justification used in the manuscript is that the method describes how synapses function in biology. )
- There is a claim regarding how this can be used in fintech. This statement doesn't belong in this work.
 - There are many different equations given throughout the text. Some of these equations come from areas like physics or information theory, and others seem to be of your own design. Regarding the latter, there is no justification or explanation for the origin of the equations. Regarding the former, if you are using equations from lots of different fields, or even field you think part of your audience might not be familiar with, you should, at the very least, include a some description of the algorithm or intuition as to why it is being leveraged.
 - It wasn't clear from your diagrams or your descriptions what the difference between a synapse and a neuron was in your architecture. It seemed like the name was used interchangeably in some areas, but then had a strict definition in others.
 - I was also not able to understand how the excitatory and the inhibitory connections that were to enter each neuron were connected to the previous layer of the network. Is a link between neurons in Figure 2 actually two links? If this is the case, then it is a direct violation of Dale's law. Again, I only mention this because most arguments seem to be of the form "this is correct because it is how it is done biologically".
 - There were a few claims made in the paper that were completely unsubstantiated. A good example of this was in the conclusion section part ii) where it was stated that "using a large number of synapses and neurons SynaNN can solve the complex problems in the real world."
- Also, the last sentence of the conclusion was not discussed anywhere in the rest of the paper. Nor was the statement itself supported except with a single citation and no description.
Regarding the empirical testing of your algorithm, I was very dissapointed to see that the only dataset it was tested against was MNIST. Furthermore there was absolutely no benchmarking against other comparative algorithms. At the very least I would have expected a comparison to the perceptron algorithm that you use as inspiration, but that would also still not have been enough.
This paper needs heavy amounts of work to make it understandable. Once it is understandable an attempt to evaluate the merit of the scientific contribution would then be possible.


PREFERENCES IMPLICIT IN THE STATE OF THE WORLD
Reinforcement learning (RL) agents optimize only the features specified in a reward function and are indifferent to anything left out inadvertently. This means that we must not only specify what to do, but also the much larger space of what not to do. It is easy to forget these preferences, since these preferences are already satisfied in our environment. This motivates our key insight: when a robot is deployed in an environment that humans act in, the state of the environment is already optimized for what humans want. We can therefore use this implicit preference information from the state to fill in the blanks. We develop an algorithm based on Maximum Causal Entropy IRL and use it to evaluate the idea in a suite of proof-of-concept environments designed to show its properties. We find that information from the initial state can be used to infer both side effects that should be avoided as well as preferences for how the environment should be organized. Our code can be found at https://github.com/HumanCompatibleAI/rlsp.

This paper considers the problem of inferring unspecified costs in an RL problem (e.g., inferring that vases in a room should not be broken). The primary insight is that the initial state of the environment conveys rich information about such unspecified costs since environments are often optimized for humans. The paper frames the problem of inferring unspecified costs from the initial condition as an inverse reinforcement learning (IRL) problem and applies the Maximum Causal Entropy IRL framework to solve this problem. Two methods are proposed for combining the inferred unspecified costs with specified costs. The efficacy of the proposed approach is demonstrated on a number of simulated examples.
Overall, I was impressed by this paper and I believe that it makes a strong contribution. The paper presents an interesting perspective on a relatively old problem (the frame problem in AI). The primary intuition of the paper (that the initial state conveys information about unspecified costs) and the framing of this problem in terms of IRL is novel. The simulated examples (while relatively simple in terms of the number of states and actions) are informative and demonstrate the strengths of the approach (and also some of the weaknesses; the paper is explicit about the current challenges). The paper is very clearly written and is easy to read.
My concerns are relatively minor:
- Perhaps the weakest bit of the paper is Section 5 (combining the specified reward with the inferred reward). As presented, the Additive method is somewhat hard to justify. However, the simulated results suggest that the Additive method performs slightly better than the Bayesian method. I would suggest either presenting a bit more intuition and justification for the Additive method or getting rid of this method altogether (since the results are not too different from the Bayesian method, which seems a bit more justifiable).
- One practical (and potentially important) question that the paper does not directly address is the problem of choosing the time horizon T (i.e., the time horizon for the past). In the standard IRL setting, it is reasonable to assume that the time horizon is given (since the demonstrations have an associated horizon). However, it is not entirely clear how to choose T in the setting considered in this paper. It is possible that if one chooses T to be too small, the inferred rewards will not be accurate (and one may have to look further back in the past to correctly infer rewards). A discussion of this issue and possible ways to choose T would be helpful.
- In Section 6.1 (baselines), the paper mentions that "while relative reachability makes use of known dynamics, it does not benefit from our handcoded featurization". Is it possible to modify the relative reachability method to also take advantage of the handcoded features, perhaps by considering dynamics over the feature space? If not, a sentence explaining that this is not straightforward would be helpful.
- In the related work section (and also in the introduction), I would recommend being more explicit about precisely what the differences are between the presented work and the approaches presented in (Krakovna et al. 2018) and (Turner, 2018). The paper is currently slightly vague about the differences.
- Currently, the title of the paper is a bit uninformative. On first reading the title, I expected a paper on control theory; the title makes no mention of unspecified costs, or reinforcement learning, or humans, etc. I believe that this is a good paper and that the paper would have more readers if the title was more inline with the content of the paper. Of course, this is at the discretion of the authors. My suggestion would be something along the lines of "Inferring Unspecified Rewards in RL from the Initial State".
Typos:
- Pg. 1, second paragraph, 3rd line: there is a placeholder for citations.
- Periods are missing at the end of equations.


We show that simple spatial transformations, namely translations and rotations alone, suffice to fool neural networks on a significant fraction of their inputs in multiple image classification tasks. Our results are in sharp contrast to previous work in adversarial robustness that relied on more complicated optimization approaches unlikely to appear outside a truly adversarial context. Moreover, the misclassifying rotations and translations are easy to find and require only a few black-box queries to the target model. Overall, our findings emphasize the need to design robust classifiers even for natural input transformations in benign settings.

Summary
The authors study robustness of neural networks for image recognition tasks with respect to geometric transformations in input space. The question is posed in an adversarial setting, where the authors exploit that Conv/ResNets are not fully translation and rotation invariant. The authors propose three untargeted attacks to increase the classification error of the network: a first-order method, an attack involving random transformations and a grid search of allowed transformations. For the random and grid search the worst prediction is considered the outcome of the attack. The authors observe that first-order attacks are not very successful in fooling the network compared to the grid search. Data augmentation as a counter measure is found to be not sufficient and adversarial (robust) training with respect to the random search attack is proposed in addition.
Evaluation
The paper is well written and particularly the empirical part is interesting. However, novelty is limited, the best approach boils down to a grid search that tests multiple hypotheses instead of a single one. There are some conceptual problems and important aspects like confidences of the classification are not addressed.
Novelty:
Many claims and observations appear trivial and well-known. E.g.,
- the research question has already been addressed by related work, leaving the proposed attacks trivial given that the attack space (allowed transformations) is specified ad hoc and without a proper measure.
- that data augmentation and training with the adversarial loss function (i.e. with the attack scheme in mind) is helpful is straight forward and not surprising
Detailed comments:
The authors study whether neural networks are robust to transformations in input space and resort to a benign adversarial setting. I'm wondering whether this allows for an answer regarding general robustness? That is, the experiments are conducted wrt the worst case, while the training does not account for an attack setting. E.g, it is unclear why the classifier would not involve a pre-processing step to counter transformations in input space, see Rowley et al. (1998).
Translation and rotation invariance of neural networks has been addressed by many authors, e.g., see Jaderberg et al. (2015) and Marcos et al. (2017).
Adversarial examples are defined to be similar and misclassified with high confidence.
The similarity of the transformation is not addressed properly. E.g., if the goal of the adversary is to force errors, why not allow for rotations of 180 degrees? Pixel-based attacks (Goodfellow et al., 2014) are more rigorous in this regard while the cited transformation-based attacks (Kanbak et al. 2017; Xiao et al., 2018) are virtually indistinguishable from the real test cases.
The effectiveness of the grid search attack seems to be connected to performing $5 * 5 * 31 = 775$ individual tests for each test case where only the worst outcome would count. The sheer number should render a misclassification more likely compared to the competitors. This is supported by empirical findings showing that only a small subset of transformations per test case accounts for the misclassification on CIFAR10 and ImageNet (Fig. 10 in the Appendix).
Regarding the padding experiments, I wonder whether the network architecture is appropriate for the new input. Here, more experimentation is necessary. The conclusion with respect to the first-order method remains a conjecture.
References:
- Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
- Max Jaderberg, Karen Simonyan, and Andrew Zisserman. Spatial transformer networks. In Advances in neural information processing systems, pp. 2017–2025, 2015.
- Can Kanbak, Seyed-Mohsen Moosavi-Dezfooli, and Pascal Frossard. Geometric robustness of deep networks: analysis and improvement. arXiv preprint arXiv:1711.09115, 2017.
- Diego Marcos, Michele Volpi, Nikos Komodakis, and Devis Tuia. Rotation equivariant vector field networks. In The IEEE International Conference on Computer Vision (ICCV), Oct 2017.
- Henry Rowley, Shumeet Baluja, and Takeo Kanade. Rotation invariant neural network-based face detection. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 38. sn, 1998.
- Chaowei Xiao, Jun-Yan Zhu, Bo Li, Warren He, Mingyan Liu, and Dawn Song. Spatially transformed adversarial examples. arXiv preprint arXiv:1801.02612, 2018.


Machine learning models including traditional models and neural networks can be easily fooled by adversarial examples which are generated from the natural examples with small perturbations. This poses a critical challenge to machine learning security, and impedes the wide application of machine learning in many important domains such as computer vision and malware detection. Unfortunately, even state-of-the-art defense approaches such as adversarial training and defensive distillation still suffer from major limitations and can be circumvented. From a unique angle, we propose to investigate two important research questions in this paper: Are adversarial examples distinguishable from natural examples? Are adversarial examples generated by different methods distinguishable from each other? These two questions concern the distinguishability of adversarial examples. Answering them will potentially lead to a simple yet effective approach, termed as defensive distinction in this paper under the formulation of multi-label classification, for protecting against adversarial examples. We design and perform experiments using the MNIST dataset to investigate these two questions, and obtain highly positive results demonstrating the strong distinguishability of adversarial examples. We recommend that this unique defensive distinction approach should be seriously considered to complement other defense approaches.

Summary: The authors propose two research questions: (1) Are adversarial examples distinguishable from natural examples? And (2) are adversarial examples generated by different methods distinguishable from each other? They find positive answers to both questions according to their experiments, and propose a method for detecting adversarial examples.
The authors take the viewpoint of varying how much the defender knows about its attackers. How they define whether a defender “knows” an attackers’ model, source examples, or adversarial generation parameters, is through keeping characteristics of various test sets the same with the training set. For example, to test the effectiveness of when the defender “knows” the adversarial generation parameters, they will have a test set where the adversarial generation parameters are the same with the training set, but will possibly vary other characteristics. They do all their experiments on MNIST.
In the first experiment (Section 4.2), the authors find that a deep neural network binary classifier for detecting adversarially-tainted images does well when the adversarial generation parameters are known, and not as well when unknown. Thus, the author’s conclude “it is always beneficial for defenders to train a DDP-Model by using adversarial examples generated based on a variety of parameters. Meanwhile, the exact model architecture, and the exact natural examples used by attackers are not influential in the accuracy of the defenders’ models.”
In the second experiment (Section 4.3), the authors test whether a neural network is able to classify an image as adversarial if images from a particular adversarial generation method is left-out of the training samples, but all others are included. They conclude that the network has the hardest time when samples from L-BFGS and JSMA are left-out of the training sample.
In the last experiment (Section 4.4), the authors test whether a deep neural network can classify adversarially-generated images according the the generation method. The answer is affirmative, and they conclude, “Similar to what is observed for a DDP-Model in Section 4.2 and 4.3, it is also beneficial for defenders to train a DDS-Model by using adversarial examples generated based on a variety of parameters; meanwhile, the exact model architecture and the exact natural examples used by attackers are not influential in the accuracy of the defender’s models.”
Stengths: The authors’ research questions are interesting and worthy of more investigation, namely whether we can detect adversarial examples. They also have nice experiments and make nice heuristic conclusions.
Weaknesses: The main complaint I have is that the authors only use the MNIST dataset. And we know that the MNIST dataset is special, so I would have liked to see the same tests on different datasets, and possibly different model architectures. I think this will be a much better contribution to the field with these additions.
Other comments:
The paper is clearly written and their experimental methodology seems original, and examining whether adversarial examples can be distinguished from untainted examples is important. But only using MNIST currently severely lowers the significance of this work. I think with more datasets and perhaps different model architectures, this can become a nice contribution to the field.
Perhaps a minor point, but their terminology of “natural” might not be the best, as MNIST is not usually considered as a “natural image,” although I am aware that what the author say is “natural” means “original,” or “untainted”. I would maybe suggest the authors change this terminology.


SLIMMABLE NEURAL NETWORKS
We present a simple and general method to train a single neural network executable at different widths 1 , permitting instant and adaptive accuracy-efficiency trade-offs at runtime. Instead of training individual networks with different width configurations, we train a shared network with switchable batch normalization. At runtime, the network can adjust its width on the fly according to on-device benchmarks and resource constraints, rather than downloading and offloading different models. Our trained networks, named slimmable neural networks, achieve similar (and in many cases better) ImageNet classification accuracy than individually trained models of MobileNet v1, MobileNet v2, ShuffleNet and ResNet-50 at different widths respectively. We also demonstrate better performance of slimmable models compared with individual ones across a wide range of applications including COCO bounding-box object detection, instance segmentation and person keypoint detection without tuning hyper-parameters. Lastly we visualize and discuss the learned features of slimmable networks. Code and models are available at: https://github.com/JiahuiYu/slimmable_networks.

This paper trains a single network executable at different widths. This is implemented by maintaining separate BN parameter and statistics for different width. The problem is well-motivated and the proposed method can be very helpful for deployment of deep models to devices with varying capacity and computational ability.

This paper is well-written and the experiments are performed on various structures. Still I have several concerns regarding the algorithm.
1. In algo 1, while gradients for convolutional and fully-connected layers are accumulated for all switches before update, how are the parameters for different switches updated?
2. In algo 1, the gradients of all switches are accumulated before the update. This may result in implicit unbalanced gradient information, e.g. the connections in 0.25x model in Figure 1 has gradient flows on all four different switches,  while the right-most 0.25x connections in 1.0x model has only one gradient flow from the 1.0x switch, will this unbalanced gradient information increase optimization difficulty and how is it solved?
3.  In the original ResNet paper, https://arxiv.org/pdf/1512.03385.pdf, the top-1 error of RestNet-50 is <21% in Table 4. The number reported in this paper (Table 3) is 23.9. Where does the difference come from?


Knowledge distillation (KD) is a popular method for reducing the computational overhead of deep network inference, in which the output of a teacher model is used to train a smaller, faster student model. Hint training (i.e., FitNets) extends KD by regressing a student model's intermediate representation to a teacher model's intermediate representation. In this work, we introduce bLock-wise Intermediate representation Training (LIT), a novel model compression technique that extends the use of intermediate representations in deep network compression, outperforming KD and hint training. LIT has two key ideas: 1) LIT trains a student of the same width (but shallower depth) as the teacher by directly comparing the intermediate representations, and 2) LIT uses the intermediate representation from the previous block in the teacher model as an input to the current student block during training, avoiding unstable intermediate representations in the student network. We show that LIT provides substantial reductions in network depth without loss in accuracy -for example, LIT can compress a ResNeXt-110 to a ResNeXt-20 (5.5×) on CIFAR10 and a VDCNN-29 to a VDCNN-9 (3.2×) on Amazon Reviews without loss in accuracy, outperforming KD and hint training in network size for a given accuracy. We also show that applying LIT to identical student/teacher architectures increases the accuracy of the student model above the teacher model, outperforming the recently-proposed Born Again Networks procedure on ResNet, ResNeXt, and VDCNN. Finally, we show that LIT can effectively compress GAN generators.

This paper introduces LIT, a network compression framework, which uses multiple intermediate representations from a teacher network to guide the training of a student network. Experiments are designed such that student networks are shallower than teacher networks, while maintaining their width. The method is validated on CIFAR-10 and 100 as well as on Amazon Reviews.
The paper is clearly written and easy to follow. The main novelty of the paper is essentially using the teacher intermediate representations as input to the student network to stabilize the training, and applying the strategy to recent networks and tasks.
The authors claim that they are only concerned with knowledge transfer between layers of the same width, that is teacher and student network been designed (by model construction) to have the same number of downsampling operations, while maintaining the same number of stages (referred to as sections in the paper). However, resnet-based architectures have been shown to perform iterative refinement of their features between downsampling operations (see e.g. https://arxiv.org/pdf/1612.07771.pdf and https://arxiv.org/pdf/1710.04773.pdf ). Moreover, these models were also shown to be good regularizers, since they can reduce their model capacity as needed (see https://arxiv.org/pdf/1804.11332.pdf).  Therefore, having experiments skipping stages would be interesting, and may allow to further compress the networks (by skipping layers or stages which do not incorporate much transformation). Following https://arxiv.org/pdf/1804.11332.pdf, for the sake of completeness, it might also be interesting to compare LIT results to the ones obtained by just removing layers in the teacher network which have small weight norms.
In method, the last sentence before "knowledge distillation loss" suggests the training of student networks might not be done end-to-end. Could the authors clarify this?
It seems there might be a typo in the KD loss of "knowledge distillation loss", equation (2). Shouldn't the second term of the equation be a function of p^T and q^T (with temperature)?
I would suggest changing "sections" to stages, as previously introduced in https://arxiv.org/pdf/1612.07771.pdf .
As for the experiments, it would be more interesting to see this kind of analysis on ImageNet (pretained resnet models are readily available).
Figure 3, why not add hint training as well?
Figure 4, what's the dataset used here?
In Section 4.2, it seems that the choice of the IR layer in the analysis could have a significant impact. How was the layer chosen for the ablation study experiments?
There are a few overstatements in the paper:
- page 5, paragraph 2: FitNets proposes a general framework to transfer knowledge from a teacher network to a student network through intermediate layers. Thus, the framework itself does not require the student networks to be deeper and thinner than the teacher network.
- page 6, "LIT can compress GANs": authors claim to overcome limitations of KD when it comes to applying knowledge transfer to pixel-wise architecture that do not output distributions. It seems that changing the loss and using a l2 loss instead is a rather minor change, especially since performing knowledge transfer by means of l2 (although at intermediate layers) has already been explored in FitNets.
Please add references for inception and FID scores.
Please fix references format in page 10.


We propose a new approach to combine Restricted Boltzmann Machines (RBMs) that can be used to solve combinatorial optimization problems. This allows synthesis of larger models from smaller RBMs that have been pretrained, thus effectively bypassing the problem of learning in large RBMs, and creating a system able to model a large, complex multi-modal space. We validate this approach by using learned representations to create "invertible boolean logic", where we can use Markov chain Monte Carlo (MCMC) approaches to find the solution to large scale boolean satisfiability problems and show viability towards other combinatorial optimization problems. Using this method, we are able to solve 64 bit addition based problems, as well as factorize 16 bit numbers. We find that these combined representations can provide a more accurate result for the same sample size as compared to a fully trained model.

The paper proposes to combine several smaller, pretrained RBMs into a larger model as a way to solve combinatorial optimization problems. Results are presented on RBMs trained to implement binary addition, multiplication, and factorization, where the proposed approach is compared with the baseline of training a full model from scratch.
I found the paper confusing at times. It is well-written from a syntactical and grammatical point of view, but some key concepts are stated without being explained, which gives the impression that the authors have a clear understanding of the material presented in the paper but communicate only part of the full picture to the reader.
For instance, there’s a brief exposition of the connection between Boltzmann machines and combinatorial optimization problems: the latter is mapped onto the former by expressing constraints as a fixed set of Boltzmann machine weights and biases, and low-energy states (i.e. more optimal solutions) are found by sampling from the model, which involves no training. What’s less clear to me is what kinds of combinatorial optimization problems can be mapped onto the RBM *training* problem. The paper states that the problem of training "large modules" is "equivalent to solving the optimization problem", but does not explain how.
Similarly, the paper mentions that the "general approach to solving these combinatorial optimization problems is to recognize the atomic unit necessary to solve the problem", but at that point the reader has no concrete example of what combinatorial optimization problem would be mapped onto training and inference in RBMS.
A concrete example is provided in the Experiments section: the authors propose to implement invertible (reversible?) boolean logic circuits by combining smaller pre-trained RBMs which implement certain logical operations into larger circuits. I have two issues with the chosen example: 1) the connection with combinatorial optimization is not clear to me, and 2) it’s not very well explained. As far as I understand, these reversible boolean logic operations are expressed as sampling a subset of the RBM’s inputs conditioned on another subset of its inputs. An example is presented in Figure 3 but is not expanded upon in the main text. I’d like the authors to validate my understanding:
An RBM is trained to implement a complete binary adder circuit by having it model the joint distribution of the adder’s inputs and outputs [A, B, Cin, S, Cout] (A is the first input bit, B is the second input bit, Cin is the input carry bit, S is the output sum bit, and Cout is the output carry bit), where (I assume) the distribution over [A, B, Cin] is uniform, and where S and Cout follow deterministically from [A, B, Cin]. After training, the output of the circuit is computed from [A, B, Cin] by clamping [A, B, Cin] and sampling [S, Cout] given [A, B, Cin] using Gibbs sampling.
The alternative to this, which is examined in the paper, is to train individual XOR, AND, and OR gates in the same way and compose them into a complete binary adder circuit as prescribed by Section 3.
I think the paper has the potential to be a lot more transparent to the reader in explaining these concepts, which would avoid them spending quite a bit of time inferring meaning from figures.
I’m also confused by the presentation of the results. For instance, I don’t know what "log", "FA1", "FA2", etc. refer to in Figure 6. Also, Figure 6 is referenced in the text in the context of binary multiplication ("[...] is able to outperform a multiplier created just by training, as can be seen in Figure 6"), but presents results for addition and factorization only.
The way I see it, implementing reversible boolean logic circuits using RBMs is an artificial problem, and the key idea of the paper -- which I find interesting -- is that in some cases it appears to be possible to combine RBMs trained for sub-problems into larger RBMs without needing to fine-tune the model. I think there are interesting large-scale applications of this, such as building an autoregressive RBM for image generation by training a smaller RBM on a more restricted inpainting task. The connection to combinatorial optimization, however, is much less clear to me.


TOWARDS THE FIRST ADVERSARIALLY ROBUST NEURAL NETWORK MODEL ON MNIST
Despite much effort, deep neural networks remain highly susceptible to tiny input perturbations and even for MNIST, one of the most common toy datasets in computer vision, no neural network model exists for which adversarial perturbations are large and make semantic sense to humans. We show that even the widely recognized and by far most successful L ∞ defense by Madry et al.
(1) has lower L 0 robustness than undefended networks and is still highly susceptible to L 2 perturbations, (2) classifies unrecognizable images with high certainty, (3) performs not much better than simple input binarization and (4) features adversarial perturbations that make little sense to humans. These results suggest that MNIST is far from being solved in terms of adversarial robustness. We present a novel robust classification model that performs analysis by synthesis using learned class-conditional data distributions. We derive bounds on the robustness and go to great length to empirically evaluate our model using maximally effective adversarial attacks by (a) applying decision-based, score-based, gradient-based and transfer-based attacks for several different L p norms, (b) by designing a new attack that exploits the structure of our defended model and (c) by devising a novel decision-based attack that seeks to minimize the number of perturbed pixels (L 0 ). The results suggest that our approach yields state-of-the-art robustness on MNIST against L 0 , L 2 and L ∞ perturbations and we demonstrate that most adversarial examples are strongly perturbed towards the perceptual boundary between the original and the adversarial class.

This paper shows that the problem of defending MNIST is still unsuccessful. It hereby proposes a model that is robust by design specifically for the MNIST classification task. Unlike conventional classifiers, the proposal learns a class-dependent data distribution using VAEs, and conducts variational inference by optimizing over the latent space to estimate the classification logits.
Some extensive experiments verify the model robustness with respect to different distance measure, with most state-of-the-art attacking schemes, and compared against several baselines. The added experiments with rotation and translation further consolidate the value of the work.
Overall I think this is a nice paper. Although being lack of some good intuition, the proposed model indeed show superior robustness to previous defending approaches. Also, the model has some other benefits that are shown in Figure 3 and 4. The results show that the model has indeed learned the data distribution rather than roughly determining the decision boundary of the input space as most existing models do.
However, I have the following comments that might help to improve the paper:
1. It would be more interesting to add more intuition on why the proposed model is already robust by design.
2. Although the paper is designed for MNIST specifically, the proposed scheme should apply to other classification tasks. Have you tried the models on other datasets like CIFAR10/100? It would be interesting to see whether the proposal would work for more complicated tasks. When the training data for each label is unbalanced, namely, some class has very few samples, would you expect the model to fail?
3. Equation (8) is complicated and still model-dependent. Without further relaxation and simplification, it’s not easy to see if this value is small or large, or to understand what kind of message this section is trying to pass.
4. Although the main contribution of the paper is to propose a model that is robust without further defending, the proposed model could still benefit from adversarial training. Have you tried to retrain your model using the adversarial examples you have got and see if it helps?


RNNS IMPLICITLY IMPLEMENT TENSOR-PRODUCT REPRESENTATIONS
Recurrent neural networks (RNNs) can learn continuous vector representations of symbolic structures such as sequences and sentences; these representations often exhibit linear regularities (analogies). Such regularities motivate our hypothesis that RNNs that show such regularities implicitly compile symbolic structures into tensor product representations (TPRs; Smolensky, 1990), which additively combine tensor products of vectors representing roles (e.g., sequence positions) and vectors representing fillers (e.g., particular words). To test this hypothesis, we introduce Tensor Product Decomposition Networks (TPDNs), which use TPRs to approximate existing vector representations. We demonstrate using synthetic data that TPDNs can successfully approximate linear and tree-based RNN autoencoder representations, suggesting that these representations exhibit interpretable compositional structure; we explore the settings that lead RNNs to induce such structuresensitive representations. By contrast, further TPDN experiments show that the representations of four models trained to encode naturally-occurring sentences can be largely approximated with a bag of words, with only marginal improvements from more sophisticated structures. We conclude that TPDNs provide a powerful method for interpreting vector representations, and that standard RNNs can induce compositional sequence representations that are remarkably well approximated by TPRs; at the same time, existing training tasks for sentence representation learning may not be sufficient for inducing robust structural representations.

This paper presents an analysis of popularly-use RNN model for structure modeling abilities by designing Tensor Product Decomposition Networks to approximate the encoder. The results show that the representations exhibit interpretable compositional structure. To provide better understanding, the paper evaluates the performance on synthesized digit sequence data as well as several sentence-encoding tasks.
Pros:
1. The paper is well-written and easy to follow. The design of the TPDN and corresponding settings (including what an filler is and what roles are included) for experiments are understandable. It makes good point at the end of the paper (section 4) on how these analysis contribute to further design of RNN models, which seems useful.
2. The experiments are extensive to support their claims. Not only synthetic data but also several popularly-used data and models are being conducted and compared. An addition of analogy dataset further demonstrate the effect of TPDN on modeling structural regularities.
Cons:
1. More detailed and extensive discussion on the contribution of the paper should be included in the introduction part to help readers understand what's the point of investigating TPDN on RNN models.
2. Some details are missing to better understand the construction. For example, on page 4, Evaluation, it is unclear of how TPDN encoder is trained, specifically, which parameters are updated? What's the objective for training? It is also unclear of whether the models in Figure 3(c) use bidirectional or unidirectional or tree decoder? In Section 3, it could be better to roughly introduce each of the existing 4 models. How do TPDN trained for these 4 sentence encoding models need to be further illustrated. More reasons should be discussed for the results in Table 2 (why bag-of-words role seem to be ok, why skip-thought cannot be approximated well).
3. It could be better to provide the actual performance (accuracy) given by TPDN on the 4 existing tasks.
4. Further thoughts: have you considered applying these analysis on other models besides RNN?


We study the problem of deep reinforcement learning where the agent's action sequences are constrained, e.g., prohibition of dithering or overactuating action sequences that might damage a robot, drone, or other physical device. Our model focuses on constraints that can be described by automata such as DFAs or PDAs. We then propose multiple approaches to augment the state descriptions of the Markov decision process (MDP) with summaries of recent action histories. We empirically evaluate these methods applying DQN to three Atari games, training with reward shaping. We found that our approaches are effective in significantly reducing, and even eliminating, constraint violations while maintaining high reward. We also observed that the total reward achieved by an agent can be highly sensitive to how much the constraints encourage or discourage exploration of potentially effective actions during training, and, in addition to helping ensure safe policies, the use of constraints can enhance exploration during training.

This work aims to use formal languages to add a reward shaping signal in the form of a penalty on the system when constraints are violated. There is also an interesting notion of using an embedding based on the action history to aid the agent in avoiding violations. However, I do not believe this paper did a good enough job in situating this work in the context of prior work — in particular (Camacho 2017). There is a significant related work section that does an ok job of describing many other works, but to my knowledge (Camacho 2017) is the most similar to this one (minus the embedding), yet is not mentioned here. It is difficult to find all related work of course, so I would encourage revision with detailed description of the novelty of this work in comparison with that one. I would also encourage an more thoughtful examination of the theoretical ramifications of the reward shaping signal with respect to the optimal policy as (Camacho 2017) do and as is modeled in the (Ng 1999) paper. As of this revision, however, I'm not sure I would recommend it for publication. Additionally, I suggest that the authors describe the reward shaping mechanism a bit more formally, it was unclear whether it fits into Ng's potential function methodology at first pass.
Comments:
+ It would be nice to explain to the reader in intuitive terms what “no-1D-dithering” means near this text. I understand that later on this is explained, but for clarity it would be good to have a short explanation during the first mentioning of this term as well.
+ It would be good to clarify in Figure 1 what . * (lr)^2 is since in the main text near the figure is is just (lr)^2 and the .* is only explained several pages ahead
+ An interesting connection that might be made is that Ng et al.’s reward shaping mechanism, if the shaping function is based on a state-dependent potential then the optimal policy under the new MDP is still optimal for the old MDP. It would be interesting to see how well this holds under this holds under this schema. In fact, this seems like analysis that several other works have done for a very similar problem (see below).
+ I have concerns about the novelty of this method. It seems rather similar to
Camacho, Alberto, Oscar Chen, Scott Sanner, and Sheila A. McIlraith. "Decision-making with non-markovian rewards: From LTL to automata-based reward shaping." In Proceedings of the Multi-disciplinary Conference on Reinforcement Learning and Decision Making (RLDM), pp. 279-283. 2017.
Camacho, Alberto, Oscar Chen, Scott Sanner, and Sheila A. McIlraith. "Non-Markovian Rewards Expressed in LTL: Guiding Search Via Reward Shaping." In Proceedings of the Tenth International Symposium on Combinatorial Search (SoCS), pp. 159-160. 2017.
However, that work proposes a similar framework in a much more formal way. In fact, in that work also a DFA is used as a reward shaping signal -- from what I can tell for the same purpose through a similar mechanism. It is possible, however, that I missed something which contrasts the two works.
Another work that can be referenced:
De Giacomo, Giuseppe, Luca Iocchi, Marco Favorito, and Fabio Patrizi. "Reinforcement Learning for LTLf/LDLf Goals." arXiv preprint arXiv:1807.06333 (2018).
I think it is particularly important to situate this work within the context of those others.
+ General the structure of the paper was a bit all over the place, crucial details were spread throughout and it took me a couple of passes to put things together. For example, it wasn't quite clear what the reward shaping mechanism was until I saw the -1000 and then had to go back to figure out that basically -1000 is added to the reward if the constraint is violated. I would suggest putting relevant details all in one place. For example, "Our reward shaping function F(x) was  { -1000, constraint violation, 0 otherwise}".


STROKENET: A NEURAL PAINTING ENVIRONMENT
We've seen tremendous success of image generating models these years. Generating images through a neural network is usually pixel-based, which is fundamentally different from how humans create artwork using brushes. To imitate human drawing, interactions between the environment and the agent is required to allow trials. However, the environment is usually non-differentiable, leading to slow convergence and massive computation. In this paper we try to address the discrete nature of software environment with an intermediate, differentiable simulation. We present StrokeNet, a novel model where the agent is trained upon a wellcrafted neural approximation of the painting environment. With this approach, our agent was able to learn to write characters such as MNIST digits faster than reinforcement learning approaches in an unsupervised manner. Our primary contribution is the neural simulation of a real-world environment. Furthermore, the agent trained with the emulated environment is able to directly transfer its skills to real-world software.

Revision:
The authors have taken my advice and addressed my concerns wholeheartedly. It is clear to me that they have taken efforts to make notable progress during the rebuttal period. Summary of their improvements:
- They have extended their methodology to handle multiple strokes
- The model has been converted to a latent-space generative model (similar to Sketch-RNN, where the latent space is from a seq2seq VAE, and SPIRAL where the latent space is used by an adversarial framework)
- They have ran addition experiments on a diverse set of datasets (now includes Kanji and QuickDraw), in addition to omniglot and mnist.
- Newer version is better written, and I like how they are also honest to admit limitations of their model rather than hide them.
I think this work is a great companion to existing work such as Sketch-RNN and SPIRAL. As mentioned in my original review, the main advantage of this is the ability to train with very limited compute resources, due to the model-based learning inspired by model-based RL work (they cited some work on world models). Taking important concepts from various different (sub) research areas and synthesizing them into this nice work should be an inspiration to the broader community. The release of their code to reproduce results of all the experiments will also facilitate future research into this exciting topic of vector-drawing models.
I have revised my score to 8, since I believe this to be at least in the better half of accepted papers at ICLR based on my experience of publishing and attending the conference in the past few years. I hope the other reviewers can have some time to reevaluate the revision.
Original review:
Summary: they propose a differentiable learning algorithm that can output a brush stroke that can approximate a pixel image input, such as MNIST or Omniglot. Unlike sketch-pix2seq[3] (which is a pixel input -> sketch output model based on sketch-rnn[2]), their method trains in an unsupervised manner and does not require paired image/stroke data. They do this via training a "world model" to approximate brush painting software and emulate it. Since this emulation model is differentiable, they can easily train an algorithm to output a stroke to approximate the drawing via back propagation, and avoid using RL and costly compute such in earlier works such as [1].
The main strength of this paper is the original thought that went into it. From reading the paper, my guess is the authors came from a background that is not pure ML research (for instance, they are experts in Javascript, WebGL, and their writing style is easy to read), and it's great to see new ideas into our field. While research from big labs [1] have the advantage of having access to massive compute so that they can run large scale RL experiments to train an agent to "sketch" something that looks like MNIST or Omniglot, the authors probably had limited resources, and had to be more creative to come up with a solution to do the same thing that trains in a couple of hours using a single P40 GPU. Unlike [1] that used an actual software rendering package that is controlled by a stroke-drawing agent, their creative approach here is to train a generator network to learn to approximate a painting package they had built, and then freeze the weights of this generator to efficiently train an agent to draw. The results for MNIST and Omniglot look comparable to [1] but achieved with much fewer resources. I find this work refreshing, and I think it can be potentially much more impactful than [1] since people can actually use it with limited compute resources, and without using RL.
That being said, things are not all rosy, and I feel there are things that need to be done for this work to be ready for publication in a good venue like ICLR. Below are a few of my suggestions that I hope will help the authors improve their work, for either this conference, or if it gets rejected, I encourage the authors to try the next conference with these improvements:
1) multiple strokes, longe strokes. I don't think having a model that can output only a single stroke is scalable to other (simple) datasets such pixel versions of KangiVG [4] or QuickDraw [5]. The authors mentioned the need for an RNN, but couldn't the encoder just output the stroke in a format that contains the pen-down / pen-up event, like the stroke format suggested in [2]? Maybe, maybe not, but in either case, for this work to matter, multiple stroke generation is needed. Most datasets are also longer than 16 points, so you will need to show that your method works for say 80-120 points for this method to be comparable to existing work. If you can't scale up 16 points, would like to see a detailed discussion as to why.
2) While I like this method and approach, to play devil's advocate, what if I simply use an off the shelf bmp-to-svg converter that is fast and efficient (like [6]), and just build a set of stroke data from a dataset of pixel data, and train a sketch-rnn type model described in [3] to convert from pixel to stroke? What does this method offer that my description fails to offer? Would like to see some discussion there.
3) I'll give a hint for as to what I think for (2). I think the value in this method is that it can be converted to a full generative model with latent variables (like a VAE, GAN, sketch-rnn) where you can feed in a random vector (gaussian or uniform), and get a sketch as an output, and do things like interpolate between two sketches. Correct me if I'm wrong, but I don't think the encoder here in the first figure outputs an embedding that has a Gaussian prior (like a VAE), so it fails to be a generative model (check out [1], even that is a latent variable model). I think the model can be easily converted to one though to address this issue, and I strongly encourage the authors to try enforcing a Gaussian prior to an embedding space (that can fit right between the 16x16x128 average pooling op to the fully connected 1024 sized layer), and show results where we can interpolate between two latent variables and see how the vector sketches are interpolated. This has also been done in [2]. If the authors need space, I suggest putting the loss diagrams near the end into the appendix, since those are not too interesting to look at.
4) As mentioned earlier, I would love to see experimental results on [4] KangiVG and [5] QuickDraw datasets, even subsets of them. An interesting result would be to compare the stroke order of this algorithm with the natural stroke order for human doodles / Chinese characters.
Minor points:
a) The figures look like they are bitmap, pixel images, but for a paper advocating stroke/vector images, I recommend exporting the diagrams in SVG format and convert them to PDF so they like crisp in the paper.
b) Write style: There are some terms like "huge" dataset that is subjective and relative. While I'm happy about the writing style of this paper, maybe some reviewers who are more academic types might not like it and have a negative bias against this work. If things don't work out this time, I recommend the authors asking some friends who have published (successfully) at good ML conferences to proof read this paper for content and style.
c) It's great to see that the implementation is open sourced, and put it on github. Next time, I recommend uploading it to an anonymous github profile/repo, although personally (and for the record, in case area chairs are looking), I don't mind at all in this case, and I don't think the author's github address revealed any real identity (I haven't tried digging deeper). Some other reviewers / area chairs might not like to see a github link that is not anonymized though.
So in the end, even though I really like this paper, I can only give a score of 6 (edit: this has since been revised upward to 8). If the authors are able to address points 1-4, please do what you can in the next few weeks and give it your best shot. I'll look at the paper again and will revise the score upwards by a point or two if I think the improvements are there. If not, and this work ends up getting rejected, please consider improving the work later on and submitting to the next venue. Good luck!
[1] SPIRAL https://arxiv.org/abs/1804.01118
[2] sketch-rnn https://arxiv.org/abs/1704.03477
[3] sketch-pix2seq https://arxiv.org/abs/1709.04121
[4] http://kanjivg.tagaini.net/
[5] https://quickdraw.withgoogle.com/data
[6] https://vectormagic.com/


Over the past four years, neural networks have been proven vulnerable to adversarial images: targeted but imperceptible image perturbations lead to drastically different predictions. We show that adversarial vulnerability increases with the gradients of the training objective when viewed as a function of the inputs. For most current network architectures, we prove that the 1 -norm of these gradients grows as the square root of the input size. These nets therefore become increasingly vulnerable with growing image size. Our proofs rely on the network's weight distribution at initialization, but extensive experiments confirm that our conclusions still hold after usual training.

This paper argues that adversarial vulnerability of neural networks increases with input dimension. Theoretical and empirical evidence are given which connect the l_p norm of the gradient of the training objective with the existence of small-worst case l_q perturbations. This connection is made by assuming that the learned function is well approximated by a linear function local to the sampled input x. By making assumptions on the initialization scheme for some simple architectures, the authors show that the l_p norm of the gradient for randomly initialized network will be large, and provide empirical evidence that these assumptions hold after training. These assumptions imply bounds on the typical magnitude of the gradient of the loss with respect to a single input coordinate, this then implies that the overall gradient norm will depend on the input dimension.
I found this paper well written. The mathematical assumptions are presented in a clear, easy to understand manner. Also high level intuition is given around their main theorems which help the reader understand the main ideas. However, I have a  number of concerns about this work.
The first is, I do not buy the motivation for studying the "phenomenon" of small worst-case l_p perturbations. I realize this statement applies to a large body of literature, but since the publication of  [1] we are still lacking concrete motivating scenarios for the l_p action space. I would encourage the authors instead to ask the closely related but more general question of how we can improve model generalization outside the natural distribution of images, such as generalization in the presence of commonly occurring image corruptions [2]. It's possible that the analysis in this work could better our understanding model generalization in the presence of different image corruptions, indeed by making similar linearity assumptions as considered in this work, test error in additive Gaussian noise can be linked with distance to the decision boundary [3,4]. However, this particular question was not explored in this work.
Second, the work is one of many to relate the norm of the gradient with adversarial robustness (for example, this has been proposed as a defense mechanism in [5,6]). I also suspect that the main theorem relating gradient norm to initialization should easily follow for more general settings using the mean field theory developed by [7,8] (this would be particularly useful for removing assumption H1, which assumes the ReLU activation is a random variable independent of the weights). Overall, I don't see how gradient norms explain why statistical classifiers make mistakes, particularly for more realistic attacker action spaces [9]. Even for "small" l_p adversarial examples there seem to be limitations as to how much gradient norms can explain the phenomenon --- for example even max margin classifiers such as SVM's have "adversarial examples". Furthermore, adversarial training has been shown to reach a point where the model is "robust" locally to training points but this robustness does not generalize to the points in the test set [10]. In fact, for the synthetic data distributions considered in [10], it's proven that no learning algorithm can achieve robustness given insufficient training data.
Finally, the main conclusion of this work "adversarial vulnerability of neural networks increases with input dimension" is an overly general statement which needs a much more nuanced view. While experiments shown in [11] support this conclusion for naturally trained networks, it is shown that when adversarial training is applied the model is more robust when the input dimension is higher (see Figure 4 a. and b.). Perhaps the assumptions for Theorem 4 are violated for these adversarially trained models.
1. https://arxiv.org/abs/1807.06732
2. https://arxiv.org/abs/1807.01697
3. https://arxiv.org/abs/1608.08967
4. https://openreview.net/forum?id=S1xoy3CcYX&noteId=BklKxJBF57.
5. https://arxiv.org/abs/1704.08847
6. https://arxiv.org/abs/1608.07690
7. https://arxiv.org/abs/1611.01232
8. https://arxiv.org/abs/1806.05393
9. https://arxiv.org/abs/1712.09665
10. https://arxiv.org/abs/1804.11285
11. https://arxiv.org/pdf/1809.02104.pdf