iclr2018_conference.pdf
In this paper we present a thrust in three directions of visual development using supervised and semi-supervised techniques. The first is an implementation of semi-supervised object detection and recognition using the principles of Soft Attention and Generative Adversarial Networks (GANs). The second and the third are supervised networks that learn basic concepts of spatial locality and quantity respectively using Convolutional Neural Networks (CNNs). The three thrusts together are based on the approach of Experiential Robot Learning, introduced in previous publication. While the results are unripe for implementation, we believe they constitute a stepping stone towards autonomous development of robotic visual modules.

This work explores some approaches in the object detection field of computer vision: (a) a soft attention map based on the activations on convolutional layers, (b) a classification regarding the location of an object in a 3x3 grid over the image, (c) an autoencoder that the authors claim to be aware of the multiple object instances in the image. These three proposals are presented in a framework of a robot vision module, although neither the experiments nor the dataset correspond to this domain.
From my perspective, the work is very immature and seems away from current state of the art on object detection, both in the vocabulary, performance or challenges. The proposed techniques are assessed in a dataset which is not described and whose results are not compared with any other technique. This important flaw in the evaluation prevents any fair comparison with the state of the art.
The text is also difficult to follow. The three contributions seem disconnected and could have been presented in separate works with a more deeper discussion. In particular, I have serious problems understanding:
1. What is exactly the contribution of the CNN pre-trained with IMageNet when learning the soft-attention maps ? The reference to a GAN architecture seems very forced and out of the scope.
2. What is the interest of the localization network ? The task it addresses seems very simple and in any case it requires a manual annotation of a dataset of objects in each of the predefined locations in the 3x3 grid.
3. The authors talk about an autoencoder architecture, but also on a classification network where the labels correspond to the object count. I could not undertstand what is exactly assessed in this section.
Finally, the authors violate the double-bind review policy by clearly referring to their previous work on Experiental Robot Learning.
I would encourage the authors to focus in one of the research lines they point in the paper and go deeper into it, with a clear understanding of the state of the art and the specific challenges these state of the art techniques may encounter in the case of robotic vision.


Due to the substantial computational cost, training state-of-the-art deep neural networks for large-scale datasets often requires distributed training using multiple computation workers. However, by nature, workers need to frequently communicate gradients, causing severe bottlenecks, especially on lower bandwidth connections. A few methods have been proposed to compress gradient for efficient communication, but they either suffer a low compression ratio or significantly harm the resulting model accuracy, particularly when applied to convolutional neural networks. To address these issues, we propose a method to reduce the communication overhead of distributed deep learning. Our key observation is that gradient updates can be delayed until an unambiguous (high amplitude, low variance) gradient has been calculated. We also present an efficient algorithm to compute the variance and prove that it can be obtained with negligible additional cost. We experimentally show that our method can achieve very high compression ratio while maintaining the result model accuracy. We also analyze the efficiency using computation and communication cost models and provide the evidence that this method enables distributed deep learning for many scenarios with commodity environments.

The paper proposes a novel way of compressing gradient updates for distributed SGD, in order to speed up overall execution. While the technique is novel as far as I know (eq. (1) in particular), many details in the paper are poorly explained (I am unable to understand) and experimental results do not demonstrate that the problem targeted is actually alleviated.
More detailed remarks:
1: Motivating with ImageNet taking over a week to train seems misplaced when we have papers claiming to train ImageNet in 1 hour, 24 mins, 15 mins...
4.1: Lemma 4.1 seems like you want B > 1, or clarify definition of V_B.
4.2: This section is not fully comprehensible to me.
- It seems you are confusingly overloading the term gradient and words derived (also in other parts or the paper). What is "maximum value of gradients in a matrix"? Make sure to use something else, when talking about individual elements of a vector (which is constructed as an average of gradients), etc.
- Rounding: do you use deterministic or random rounding? Do you then again store the inaccuracy?
- I don't understand definition of d. It seems you subtract logarithm of a gradient from a scalar.
- In total, I really don't know what is the object that actually gets communicated, and consequently when you remark that this can be combined with QSGD and the more below it, I don't understand it. This section has to be thoroughly explained, perhaps with some illustrative examples.
4.3: allgatherv remark: does that mean that this approach would not scale well to higher number of workers?
4.4: Remarks about quantization and mantissa manipulation are not clear to me again, or what is the point in doing so. Possible because the problems above.
5: I think this section is not too useful unless you can accompany it with actual efficient implementation and contrast the practical performance.
6: Given that I don't understand how you compress the information being communicated, it is hard to believe the utility of the method. The objective was to speed up training time because communication is bottleneck. If you provide 12,000x compression, is it any more practically useful than providing 120x compression? What would be the difference in runtime? Such questions are never discussed. Further, if in the implementation you discuss masking mantissa, I have serious concern about whether the compression protocol is feasible to implement efficiently, without writing some extremely low-level code. I think the soundness of work addressing this particular problem is damaged if not implemented properly (compared to other kinds of works in current ML related research). Therefore I highly recommend including proper time comparison with a baseline in the future.
Further, I don't understand 2 things about the Tables. a) how do you combine the proposed method with Momentum in SGD? This is not discussed as far as I can see. b) What is "QSGD, 2bit" If I remember QSGD protocol correctly, there's no natural mapping of 2bit to its parameters.


Reinforcement Learning (RL) can model complex behavior policies for goaldirected sequential decision making tasks. A hallmark of RL algorithms is Temporal Difference (TD) learning: value function for the current state is moved towards a bootstrapped target that is estimated using the next state's value function. λ-returns define the target of the RL agent as a weighted combination of rewards estimated by using multiple many-step look-aheads. Although mathematically tractable, the use of exponentially decaying weighting of n-step returns based targets in λ-returns is a rather ad-hoc design choice. Our major contribution is that we propose a generalization of λ-returns called Confidence-based Autodidactic Returns (CAR), wherein the RL agent learns the weighting of the n-step returns in an end-to-end manner. In contrast to λ-returns wherein the RL agent is restricted to use an exponentially decaying weighting scheme, CAR allows the agent to learn to decide how much it wants to weigh the n-step returns based targets. Our experiments, in addition to showing the efficacy of CAR, also empirically demonstrate that using sophisticated weighted mixtures of multi-step returns (like CAR and λ-returns) considerably outperforms the use of n-step returns. We perform our experiments on the Asynchronous Advantage Actor Critic (A3C) algorithm in the Atari 2600 domain.

This paper revisits the idea of exponentially weighted lambda-returns at the heart of TD algorithms. The basic idea is that instead of geometrically weighting the n-step returns we should instead weight them according to the agent's own estimate of its confidence in it's learned value function. The paper empirically evaluates this idea on Atari games with deep non-linear state representations, compared to state-of-the-art baselines.
This paper is below the threshold because there are issues with the : 1) motivation, 2) the technical details, and (3) the empirical results.
The paper begins by stating that the exponential weighting of lambda returns is ad-hoc and unjustified. I would say the idea is well justified in several ways. First the lambda return definition lends itself to online approximations that achieve a fully incremental online form with linear computation and nearly as good performance of the off-line version. Second, decades of empirical results illustrating good performance of TD compared with MC methods. And an extensive literature of theoretical results. The paper claims that the exponential has been noted to be ad-hoc, please provide a reference for this.
There have been several works that have noted that lambda can and perhaps should be changed as a function of state (Sutton and Barto, White and White [1], TD-Gammon). In fact, such works even not that lambda should be related to confidence. The paper should work harder to motivate why adapting lambda as a function of state---which has been studied---is not sufficient.
I don't completely understand the objective. Returns with higher confidence should be weighted higher, according to the confidence estimate around the value function estimate as a function of state? With longer returns, n>>1, the role of the value function in the target is down-weighted by gamma^n---meaning its accuracy is of little relevance to the target. How does your formalism take this into account? The basic idea of the lambda return assumes TD targets are better than MC targets due to variance, which place more weight on shorter returns.
I addition I don't understand how learning confidence of the value function has a realizable target. We do not get supervised targets of the confidence of our value estimates. What is your network updating toward?
The work of Konidaris et al [1] is a more appropriate reference for this work (rather than the Thomas reference provided). Your paper does not very clearly different itself from Konidaris's work here. Please expand on this.
The experiments have some issues. One issue is that basic baselines could more clearly illustrate what is going on. There are two such baselines: random fixed weightings of the n-step returns, and persisting with the usual weighting but changing lambda on each time step (either randomly or according to some decay schedule). The first baseline is a sanity check to ensure that you are not observing some random effect. The second checks to see if your alternative weighting is simply approximating the benefits of changing lambda with time or state.
I would say the current results indicate the conventional approach to TD is working well if not better than the new one. Looking at fig 3, its clear the kangaroo is skewing the results, and that overall the new method is performing worse. This is further conflated by fig7 which attempts to illustrate the quality of the learned value functions. In Kangaroo, the domain where your method does best, the l2 error is worse. On the other hand in sea quest and space invaders, where your method does worse, the l2 error is better. These results seem conflicting, or at least raise more questions than they answer.
[1] A Greedy Approach to Adapting the Trace Parameter for Temporal Difference Learning . Adam White and Martha White. Autonomous Agents and Multi-agent Systems (AAMAS), 2016
[2] G. D. Konidaris, S. Niekum, and P. S. Thomas. TDγ: Re-evaluating complex backups in temporal difference learning. In Advances in Neural Information Processing Systems 24, pages 2402–2410. 2011.


Published as a conference paper at ICLR 2018 SIMULATING ACTION DYNAMICS WITH NEURAL PROCESS NETWORKS
Understanding procedural language requires anticipating the causal effects of actions, even when they are not explicitly stated. In this work, we introduce Neural Process Networks to understand procedural text through (neural) simulation of action dynamics. Our model complements existing memory architectures with dynamic entity tracking by explicitly modeling actions as state transformers. The model updates the states of the entities by executing learned action operators. Empirical results demonstrate that our proposed model can reason about the unstated causal effects of actions, allowing it to provide more accurate contextual information for understanding and generating procedural text, all while offering more interpretable internal representations than existing alternatives.

Summary
This paper presents Neural Process Networks, an architecture for capturing procedural knowledge stated in texts that makes use of a differentiable memory, a sentence and word attention mechanism, as well as learning action representations and their effect on entity representations. The architecture is tested for tracking entities in recipes, as well as generating the natural language description for the next step in a recipe. It is compared against a suit of baselines, such as GRUs, Recurrent Entity Networks, Seq2Seq and the Neural Checklist Model. While I liked the overall paper, I am worried about the generality of the model, the qualitative analysis, as well as a fair comparison to Recurrent Entity Networks and non-neural baselines.
Strengths
I believe the authors made a good effort in comparing against existing neural baselines (Recurrent Entity Networks, Neural Checklist Model) *for their task*. That said, it is unclear to me how generally applicable the method is and whether the comparison against Recurrent Entity Networks is fair (see Weaknesses).
I like the ablation study.
Weaknesses
While I find the Neural Process Networks architecture interesting and I acknowledge that it outperforms Recurrent Entity Networks for the presented tasks, after reading the paper it is not clear to me how generally applicable the architecture is. Some design choices seem rather tailored to the task at hand (manual collection of actions MTurk annotation in section 3.1) and I am wondering where else the authors see their method being applied given that the architecture relies on all entities and actions being known in advance. My understanding is that the architecture could be applied to bAbI and CBT (the two tasks used in the Recurrent Entity Networks paper). If that is the case, a fair comparison to Recurrent Entity Networks would have been to test against Recurrent Entity Networks on these tasks too. If they the architecture cannot be applied in these tasks, the authors should explain why.
I am not convinced by the qualitative analysis. Table 2 tells me that even for the best model the entity selection performance is rather unreliable (only 55.39% F1), yet all examples shown in Table 3 look really good, missing only the two entities oil (1) and sprinkles (3). This suggests that these examples were cherry-picked and I would like to see examples that are sampled randomly from the dev set. I have a similar concern regarding the generation task. First, it is not mentioned where the examples in Table 6 are taken from – is it the train, dev or test set? Second, the overall BLEU score seems quite low even for the best model, yet the examples in Table 6 look really good. In my opinion, a good qualitative analysis should also discuss failure cases. Since the BLEU score is so low here, you might also want to compare perplexity of the models.
The qualitative analysis in Table 5 is not convincing either. In Appendix A.1 it is mentioned that word embeddings are initialized from word2vec trained on the training set. My suspicion is that one would get the clustering in Table 4 already from those pretrained vectors, maybe even when pretrained on the Google news corpus. Hence, it is not clear what propagating gradients through the Neural Process Networks into the action embeddings adds, or put differently, why does it have to be a differentiable architecture when an NLP pipeline might be enough? This could easily be tested by another ablation where action embeddings are pretrained using word2vec and then fixed during training of the Neural Process Network. Moreover, in 3.3 it is mentioned that even the Action Selection is pretrained, which makes me wonder what is actually trained jointly in the architecture and what is not.
I think the difficulty of the task at hand needs to be discussed at some point, ideally early in the paper. Until examples on page 7 are shown, I did not have a sense for why a neural architecture is chosen. For example, in 2.3 it is mentioned that for "wash and cut" the two functions fwash and fcut need to be selected. For this example, this seems trivial as the functions have the same name (and you could even have a function per name!). As far as I understand, the point of the action selector is to only have a fixed number of learned actions and multiple words (cut, slice etc.) should select the same action fcut. Otherwise (if there is little language ambiguity) I would not see the need for a complex neural architecture. Related to that, a non-neural baseline for the entity selection task that in my opinion definitely needs to be added is extracting entities using a pretrained NER system and returning all of them as the selection.
p2 Footnote 1: So if I understand this correctly, this work builds upon a dataset of over 65k recipes from Kiddon et al. (2016), but only for 875 of those detailed annotations were created?
Minor Comments
p1: The statement "most natural language understanding algorithms do not have the capacity …" should be backed by reference.
p2: "context representation ht" – I would directly mention that this is a sentence encoding.
p3: 2.4: I have the impression what you are describing here is known in the literature as entity linking.
p3 Eq.3: Isn't c3*0 always a vector of zeros?
p4 Eq.6: W4 is an order-3 tensor, correct?
p4 Eq.8: What is YC and WC here and what are their dimensions? I am confused by the softmax, as my understanding (from reading the paragraph on the Action Selection Loss on p.5) was that the expression in the softmax here is a scalar (as it is done for every possible action), so this should be a sigmoid to allow for multiple actions to attain a probability of 1?
p5: "See Appendix for details" -> "see Appendix C for details"
p5 3.3: Could you elaborate on the heuristic for extracting verb mentions? Is only one verb mention per sentence extracted?
p5: "trained to minimize cross-entropy loss" -> "trained to minimize the cross-entropy loss"
p5 3.3: What is the global loss?
p6: "been read (§2.5." -> "been read (§2.5)."
p6: "We encode these vectors using a bidirectional GRU" – I think you composing a fixed-dimensional vector from the entity vectors? What's eI?
p7: For which statement is (Kim et al. 2016) the reference? Surely, they did not invent the Hadamard product.
p8: "Our model, in contrast" use" -> "Our model, in contrast, uses".
p8 Related Work: I think it is important to mention that existing architectures such as Memory Netwroks could, in principle, learn to track entities and devote part of their parameters to learn the effect of actions. What Neural Process Networks are providing is a strong inductive bias for tracking entities and learning the effect of actions that is useful for the task considered in this paper. As mentioned in the weaknesses, this might however come at the price of a less general model, which should be discussed.
# Update after the rebuttal
Thanks for the clarifications and updating the paper. I am increasing my score by two points and expect to see the ablations as well as the NER baseline mentioned in the rebuttal in the next revision of the paper. Furthermore, I encourage the authors to include the analysis of pretrained word2vec embeddings vs the embeddings learned by this architecture into the paper.


We develop a comprehensive description of the active inference framework, as proposed by Friston (2010), under a machine-learning compliant perspective. Stemming from a biological inspiration and the auto-encoding principles, a sketch of a cognitive architecture is proposed that should provide ways to implement estimation-oriented control policies. Computer simulations illustrate the effectiveness of the approach through a foveated inspection of the input data. The pros and cons of the control policy are analyzed in detail, showing interesting promises in terms of processing compression. Though optimizing future posterior entropy over the actions set is shown enough to attain locally optimal action selection, offline calculation using class-specific saliency maps is shown better for it saves processing costs through saccades pathways pre-processing, with a negligible effect on the recognition/compression rates.

This paper introduces a machine learning adaptation of the active inference framework proposed by Friston (2010), and applies it to the task of image classification on MNIST through a foveated inspection of images. It describes a cognitive architecture for the same, and provide analyses in terms of processing compression and "confirmation biases" in the model.
– Active perception, and more specifically recognition through saccades (or viewpoint selection) is an interesting biologically-inspired approach and seems like an intuitive and promising way to improve efficiency. The problem and its potential applications are well motivated.
– The perception-driven control formulation is well-detailed and simple to follow.
– The achieved compression rates are significant and impressive, though additional demonstration of performance on more challenging datasets would have been more compelling
Questions and comments:
– While an 85% compression rate is significant, 88% accuracy on MNIST seems poor. A plot demonstrating the tradeoff of
accuracy for compression (by varying Href or other parameters) would provide a more complete picture of performance. Knowing baseline performance (without active inference) would help put numbers in perspective by providing a performance bound due to modeling choices.
– What does the distribution of number of saccades required per recognition (for a given threshold) look like over the entire dataset, i.e. how many are dead-easy vs difficult?
– Steady state assumption: How can this be relaxed to further generalize to non-static scenes?
– Figure 3 is low resolution and difficult to read.
Post-rebuttal comments:
I have revised my score after considering comments from other reviewers and the revised paper. While the revised version contains more experimental details, the paper in its present form lacks comparisons to other gaze selection and saliency models which are required to put results in context. The paper also contains grammatical errors and is somewhat difficult to understand. Finally, while it proposes an interesting formulation of a well-studied problem, more comparisons and analysis are required to validate the approach.


Published as a conference paper at ICLR 2018 SIMULATED+UNSUPERVISED LEARNING WITH ADAPTIVE DATA GENERATION AND BIDIRECTIONAL MAPPINGS
Collecting a large dataset with high quality annotations is expensive and timeconsuming. Recently, Shrivastava et al. (2017) propose Simulated+Unsupervised (S+U) learning: It first learns a mapping from synthetic data to real data, translates a large amount of labeled synthetic data to the ones that resemble real data, and then trains a learning model on the translated data. Bousmalis et al. (2017b) propose a similar framework that jointly trains a translation mapping and a learning model. While these algorithms are shown to achieve the state-of-the-art performances on various tasks, it may have a room for improvement, as they do not fully leverage flexibility of data simulation process and consider only the forward (synthetic to real) mapping. Inspired by this limitation, we propose a new S+U learning algorithm, which fully leverage the flexibility of data simulators and bidirectional mappings between synthetic and real data. We show that our approach achieves the improved performance on the gaze estimation task, outperforming (Shrivastava et al., 2017).

Review, ICLR 2018, Simulated+Unsupervised Learning With Adaptive Data Generation and Bidirectional Mappings
Summary:
The paper presents several extensions to the method presented in SimGAN (Shirvastava et al. 2017).
First, it adds a procedure to make the distribution of parameters of the simulation closer to the one in real world images. A predictor is trained on simulated images created with a manually initialized distribution. This predictor is used to estimate pseudo labels for the unlabeled real-world data. The distribution of the estimated pseudo labels is used produce a new set of simulated images. This process is iterated.
Second, it adds the idea of cycle consistency (e.g., from CycleGAN) in order to counter mode collapse and label shift.
Third, since cycle consistency does not fully solve the label shift problem, a feature consistency loss is added.
Finally, in contrast to ll related methods, the final system used for inference is not a predictor trained on a mix of real and “fake” images from the real-world target domain. Instead the predictor is trained purely on synthetic data and it is fed real world examples by using the back/real-to-sim-generator (trained in the conjunction with the forward mapping cycle) to map the real inputs to “fake” synthetic ones.
The paper is well written. The novelty is incremental in most parts, but the overall system can be seen as novel.
In particular, I am not aware of any published work that uses of the (backwards) real-to-sim generator plus sim-only trained predictor for inference (although I personally know several people who had the same idea and have been working on it). I like this part because it perfectly makes sense not to let the generator hallucinate real-world effects on rather clean simulated data, but the other way around, remove all kinds of variations to produce a clean image from which the prediction should be easier.
The paper should include Bousmalis et al., “Unsupervised Pixel-Level Domain Adaptation With Generative Adversarial Networks”, CVPR 2017 in its discussion, since it is very closely related to Shirvastava et al. 2017.
With respect to the feature consistency loss the paper should also discuss related work defining losses over feature activations for very similar reasons, such as in image stylization (e.g. L. A. Gatys et al. “Image Style Transfer Using Convolutional Neural Networks” CVPR 2016, L. A. Gatys et al. “Controlling Perceptual Factors in Neural Style Transfer” CVPR 2017), or the recently presented “Photographic Image Synthesis with Cascaded Refinement Networks”, ICCV 2017.
Bousmalis et al., “Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Grasping”, arXiv:1709.07857, even uses the same technique in the context of training GANs.
Adaptive Data Generation:
I do not fully see the point in matching the distribution of parameters of the real world samples with the simulated data. For the few, easily interpretable parameters in the given task it should be relatively easy to specify reasonable ranges. If the simulation in some edge cases produces samples that are beyond the range of what occurs in the real world, that is maybe not very efficient, but I would be surprised if it ultimately hurt the performance of the predictor.
I do see the advantage when training the GAN though, since a good discriminator would learn to pick out those samples as generated. Again, though, I am not very sure whether that would hurt the performance of the overall system in practice.
Limiting the parameters to those values of the real world data also seems rather restricting. If the real world data does not cover certain ranges, not because those values are infeasible or infrequent, but just because it so happens that this range was not covered in the data acquisition, the simulation could be used to fill in those ranges.
Additionally, the whole procedure of training on sim data and then pseudo-labeling the real data with it is based on the assumption that a predictor trained on simulated data only already works quite well on real data. It might be possible in the case of the task at hand, but for more challenging domain adaptation problem it might not be feasible.
There is also no guarantee for the convergence of the cycle, which is also evident from the experiments (Table 1. After three iterations the angle error increases again. (The use of the Hellinger distance is unclear to me since it, as explained in the text, does not correspond with what is being optimized). In the experiments the cycle was stopped after two iterations. However, how would you know when to stop if you didn’t have ground truth labels for the real world data?
Comparisons:
The experiments should include a comparison to using the forward generator trained in this framework to train a predictor on “fake” real data and test it on real data (ie. a line “ours | RS | R | ?” in Table 2, and a more direct comparison to Shrivastava). This would be necessary to prove the benefit of using the back-generator + sim trained predictor.
Detailed comments:
* Figure 1 seems not to be referenced in the text.
* I don’t understand the choice for reduction of the sim parameters. Why was, e.g., the yaw and pitch parameters of the eyeball set equal to those of the camera? Also, I guess there is a typo in the last equality (pitch and yaw of the camera?).
* The Bibliography needs to be checked. Names of journals and conferences are inconsistent, long and short forms mixed, year several times, “Proceedings” multiple times, ...


This paper presents a novel variant of hierarchical hidden Markov models (HMMs), the multiscale hidden Markov model (MSHMM), and an associated spectral estimation and prediction scheme that is consistent, finds global optima, and is computationally efficient. Our MSHMM is a generative model of multiple HMMs evolving at different rates where the observation is a result of the additive emissions of the HMMs. While estimation is relatively straightforward, prediction for the MSHMM poses a unique challenge, which we address in this paper. Further, we show that spectral estimation of the MSHMM outperforms standard methods of predicting the asset covariance of stock prices, a widely addressed problem that is multiscale, non-stationary, and requires processing huge amounts of data.

The paper focuses on a very particular HMM structure which involves multiple, independent HMMs. Each HMM emits an unobserved output with an explicit duration period. This explicit duration modelling captures multiple scale of temporal resolution. The actual observations are a weighted linear combination of the emissions from each latent HMM. The structure allows for fast inference using a spectral approach.
I found the paper unclear and lacking in detail in several key aspects:
1. It is unclear to me from Algorithm 2 how the weight vectors w are estimated. This is not adequately explained in the section on estimation.
2. The authors make the assumption that each HMM injects noise into the unobserved output which then gets propagated into the overall observation. What are reasons for his choice of model over a simpler model where the output of each HMM is uncorrupted?
3. The simulation example does not really demonstrate the ability of the MSHMM to do anything other than recover structure from data simulated under an MSHMM. It would be more interesting to apply to data simulated under non-Markovian or other setups that would enable richer frequency structures to be included and the ability of MSHMM to capture these.
4. The real data experiments shows some improvements in predictive accuracy with fast inference. However, the authors do not give a sufficiently broad exploration of the representations learnt by the model which allows us to understand the regimes in which the model would be advantageous.
Overall, the paper presents an interesting approach but the work lacks maturity. Furthermore, simulation and real data examples to explore the properties and utility of the method are required.


We study the problem of explaining a rich class of behavioral properties of deep neural networks. Our influence-directed explanations approach this problem by peering inside the network to identify neurons with high influence on the property of interest using an axiomatically justified influence measure, and then providing an interpretation for the concepts these neurons represent. We evaluate our approach by training convolutional neural networks on Pubfig, ImageNet, and Diabetic Retinopathy datasets. Our evaluation demonstrates that influence-directed explanations (1) localize features used by the network, (2) isolate features distinguishing related instances, (3) help extract the essence of what the network learned about the class, and (4) assist in debugging misclassifications.

SUMMARY
========
This paper proposes to measure the "influence" of single neurons w.r.t. to a quantity of interest represented by another neuron, typically w.r.t. to an output neuron for a class of interest, by simply taking the gradient of the corresponding output neuron w.r.t to the considered neuron. This gradient is used as is, given a single input instance, or else, gradients are averaged over several input instances.
In the latter case the averaging is described by an ad-hoc distribution of interest P which is introduced in the definition of the influence measure, however in the present work only two types of averages are practically used: either the average is performed over all instances belonging to one class, or over all input instances.
In other words, standard gradient backpropagation values (or average of them) are used as a proxy to quantify the importance of neurons (these neurons being within hidden layers or at the input layer), and are intended to better explain the classification, or sometimes even misclassification, performed by the network.
The proposed importance measure is theoretically justified by stating a few properties (called axioms) an importance measure should generally verify, and then showing the proposed measure fullfills these requirements.
Empirically the proposed measure is used to inspect the classification of a few input instances, to extract "class-expert" neurons, and to find a preprocessing bug in one model. The only comparison to a related work method is done qualitatively on one image visualization, where the proposed method is compared to Integrated Gradients [Sundararajan et al. 2017].
WEAKNESSES
==========
The similarity and differences between the proposed method and related work is not made clear. For example, in the case of a single input instance, and when the quantity of interest is one output neuron corresponding to one class, the proposed measure is identical to the image-specific class saliency of [Simonyan et al. 2014].
The difference to Integrated Gradients [Sundararajan et al. 2017] at the end of Section 1.1 is also not clearly formulated: why is the constraint on distribution marginality weaker here ?
An important class of explanation methods, namely decomposition-based methods (e.g. LRP, Excitation Backprop, Deep Taylor Decomposition), are not mentioned. Recent work (Montavon et al., Digital Signal Processing, 2017), discusses the advantages of decomposition-based methods over gradient-based approaches. Thus, the authors should clearly state the advantages/disadvantes of the proposed gradient-based method over decomposition-based techniques.
Concerning the theoretical justification:
It is not clear how Axiom 2 ensures that the proposed measure only depends on points within the input data manifold. This is indeed an important issue, since otherwise the gradients in equation (1) might be averaged completely outside the data manifold and thus the influence measure be unrelated to the data and problem the neural network was trained on. Also the notation used in Axiom 5 is very confusing. Moreover it seems this axiom is even not used in the proof of Theorem 2.
Concerning the experiments:
The experimental setup, especially in Section 3.3.1, is not well defined: on which layer of the network is the mask applied? What is the "quantity of interest": shouldn't it be an output neuron value rather than h|i (as stated at the begin of the fourth paragraph of Section 3.3.1)?
The proposed method should to be quantitatively compared with other explanation techniques (e.g. by iteratively perturbing most relevant pixels and tracking the performance drop, see Samek et al., IEEE TNNLS, 2017).
The last example of explaining the bug is not very convincing, since the observation that class 2 distinctive features are very small in the image space, and thus might have been erased through gaussian blur, is not directly related to the influence measure and could have been made aso independently from it.
CONCLUSION
==========
Overall this work does not introduce any new importance measure for neurons, it merely formalizes the use of standard backpropagation gradients as influence measure.
Using gradients as importance measure was already done in previous work (e.g. [Simonyan et al. 2014]). Though taking the average of gradients over several input instances is new, this information might not be of great help for practical applications.
Recent work also showed that raw gradients are less informative than decomposition-based quantities to explain the classification decisions made by a neural network.


Published as a conference paper at ICLR 2018 SMOOTH LOSS FUNCTIONS FOR DEEP TOP-K CLASSIFICATION
The top-k error is a common measure of performance in machine learning and computer vision. In practice, top-k classification is typically performed with deep neural networks trained with the cross-entropy loss. Theoretical results indeed suggest that cross-entropy is an optimal learning objective for such a task in the limit of infinite data. In the context of limited and noisy data however, the use of a loss function that is specifically designed for top-k classification can bring significant improvements. Our empirical evidence suggests that the loss function must be smooth and have non-sparse gradients in order to work well with deep neural networks. Consequently, we introduce a family of smoothed loss functions that are suited to top-k optimization via deep learning. The widely used cross-entropy is a special case of our family. Evaluating our smooth loss functions is computationally challenging: a naïve algorithm would require O( n k ) operations, where n is the number of classes. Thanks to a connection to polynomial algebra and a divideand-conquer approach, we provide an algorithm with a time complexity of O(kn). Furthermore, we present a novel approximation to obtain fast and stable algorithms on GPUs with single floating point precision. We compare the performance of the cross-entropy loss and our margin-based losses in various regimes of noise and data size, for the predominant use case of k = 5. Our investigation reveals that our loss is more robust to noise and overfitting than cross-entropy.

The paper is clear and well written. The proposed approach seems to be of interest and to produce interesting results. As datasets in various domain get more and more precise, the problem of class confusing with very similar classes both present or absent of the training dataset is an important problem, and this paper is a promising contribution to handle those issues better.
The paper proposes to use a top-k loss such as what has been explored with SVMs in the past, but with deep models. As the loss is not smooth and has sparse gradients, the paper suggests to use a smoothed version where maximums are replaced by log-sum-exps.
I have two main concerns with the presentation.
A/ In addition to the main contribution, the paper devotes a significant amount of space to explaining how to compute the smoothed loss. This can be done by evaluating elementary symmetric polynomials at well-chosen values.
The paper argues that classical methods for such evaluations (e.g., using the usual recurrence relation or more advanced methods that compensate for numerical errors) are not enough when using single precision floating point arithmetic. The paper also advances that GPU parallelization must be used to be able to efficiently train the network.
Those claims are not substantiated, however, and the method proposed by the paper seems to add substantial complexity without really proving that it is useful.
The paper proposes a divide-and-conquer approach, where a small amount of parallelization can be achieved within the computation of a single elementary symmetric polynomial value. I am not sure why this is of interest - can't the loss evaluation already be parallelized trivially over examples in a training/testing minibatch? I believe the paper could justify this approach better by providing a bit more insights as to why it is required. For instance:
- What accuracies and train/test times do you get using standard methods for the evaluation of elementary symmetric polynomials?
- How do those compare with CE and L_{5, 1} with the proposed method?
- Are numerical instabilities making this completely unfeasible? This would be especially interesting to understand if this explodes in practice, or if evaluations are just a slightly inaccurate without much accuracy loss.
B/ No mention is made of the object detection problem, although multiple of the motivating examples in Figure 1 consider cases that would fall naturally into the object detection framework. Although top-k classification considers in principle an easier problem (no localization), a discussion, as well as a comparison of top-k classification vs., e.g., discarding localization information out of object detection methods, could be interesting.
Additional comments:
- Figure 2b: this visualization is confusing. This is presented in the same figure and paragraph as the CIFAR results, but instead uses a single synthetic data point in dimension 5, and k=1. This is not convincing. An actual experiment using full dataset or minibatch gradients on CIFAR and the same k value would be more interesting.


Published as a conference paper at ICLR 2018 TRUNCATED HORIZON POLICY SEARCH: COMBINING REINFORCEMENT LEARNING & IMITATION LEARNING
In this paper, we propose to combine imitation and reinforcement learning via the idea of reward shaping using an oracle. We study the effectiveness of the nearoptimal cost-to-go oracle on the planning horizon and demonstrate that the costto-go oracle shortens the learner's planning horizon as function of its accuracy: a globally optimal oracle can shorten the planning horizon to one, leading to a onestep greedy Markov Decision Process which is much easier to optimize, while an oracle that is far away from the optimality requires planning over a longer horizon to achieve near-optimal performance. Hence our new insight bridges the gap and interpolates between imitation learning and reinforcement learning. Motivated by the above mentioned insights, we propose Truncated HORizon Policy Search (THOR), a method that focuses on searching for policies that maximize the total reshaped reward over a finite planning horizon when the oracle is sub-optimal. We experimentally demonstrate that a gradient-based implementation of THOR can achieve superior performance compared to RL baselines and IL baselines even when the oracle is sub-optimal.

=== SUMMARY ===
The paper considers a combination of Reinforcement Learning (RL) and Imitation Learning (IL), in the infinite horizon discounted MDP setting.
The IL part is in the form of an oracle that returns a value function V^e, which is an approximation of the optimal value function. The paper defines a new cost (or reward) function based on V^e, through shaping (Eq. 1). It is known that shaping does not change the optimal policy.
A key aspect of this paper is to consider a truncated horizon problem (say horizon k) with the reshaped cost function, instead of an infinite horizon MDP.
For this truncated problem, one can write the (dis)advantage function as a k-step sum of reward plus the value returned by the oracle at the k-th step (cf. Eq. 5).
Theorem 3.3 shows that the value of the optimal policy of the truncated MDP w.r.t. the original MDP is only O(gamma^k eps) worse than the optimal policy of the original problem (gamma is the discount factor and eps is the error between V^e and V*).
This suggests two things:
1) Having an oracle that is accurate (small eps) leads to good performance. If oracle is the same as the optimal value function, we do not need to plan more than a single step ahead.
2) By planning for k steps ahead, one can decrease the error in the oracle geometrically fast. In the limit of k —> inf, the error in the oracle does not matter.
Based on this insight, the paper suggests an actor-critic-like algorithm called THOR (Truncated HORizon policy search) that minimizes the total cost over a truncated horizon with a modified cost function.
Through a series of experiments on several benchmark problems (inverted pendulum, swimmer, etc.), the paper shows the effect of planning horizon k.
=== EVALUATION & COMMENTS ===
I like the main idea of this paper. The paper is also well-written. But one of the main ideas of this paper (truncating the planning horizon and replacing it with approximation of the optimal value function) is not new and has been studied before, but has not been properly cited and discussed.
There are a few papers that discuss truncated planning. Most closely is the following paper:
Farahmand, Nikovski, Igarashi, and Konaka, “Truncated Approximate Dynamic Programming With Task-Dependent Terminal Value,” AAAI, 2016.
The motivation of AAAI 2016 paper is different from this work. The goal there is to speedup the computation of finite, but large, horizon problem with a truncated horizon planning. The setting there is not the combination of RL and IL, but multi-task RL. An approximation of optimal value function for each task is learned off-line and then used as the terminal cost.
The important point is that the learned function there plays the same role as the value provided by the oracle V^e in this work. They both are used to shorten the planning horizon. That paper theoretically shows the effect of various error terms, including terms related to the approximation in the planning process (this paper does not do that).
Nonetheless, the resulting algorithms are quite different. The result of this work is an actor-critic type of algorithm. AAAI 2016 paper is an approximate dynamic programming type of algorithm.
There are some other papers that have ideas similar to this work in relation to truncating the horizon. For example, the multi-step lookahead policies and the use of approximate value function as the terminal cost in the following paper:
Bertsekas, “Dynamic Programming and Suboptimal Control: A Survey from ADP to MPC,” European Journal of Control, 2005.
The use of learned value function to truncate the rollout trajectory in a classification-based approximate policy iteration method has been studied by
Gabillon, Lazaric, Ghavamzadeh, and Scherrer, “Classification-based Policy Iteration with a Critic,” ICML, 2011.
Or in the context of Monte Carlo Tree Search planning, the following paper is relevant:
Silver et al., “Mastering the game of Go with deep neural networks and tree search,” Nature, 2016.
Their “value network” has a similar role to V^e. It provides an estimate of the states at the truncated horizon to shorten the planning depth.
Note that even though these aforementioned papers are not about IL, this paper’s stringent requirement of having access to V^e essentially make it similar to those papers.
In short, a significant part of this work’s novelty has been explored before. Even though not being completely novel is totally acceptable, it is important that the paper better position itself compared to the prior art.
Aside this main issue, there are some other comments:
- Theorem 3.1 is not stated clearly and may suggest more than what is actually shown in the proof. The problem is that it is not clear about the fact the choice of eps is not arbitrary.
The proof works only for eps that is larger than 0.5. With the construction of the proof, if eps is smaller than 0.5, there would not be any error, i.e., J(\hat{pi}^*) = J(pi^*).
The theorem basically states that if the error is very large (half of the range of value function), the agent does not not perform well. Is this an interesting case?
- In addition to the papers I mentioned earlier, there are some results suggesting that shorter horizons might be beneficial and/or sufficient under certain conditions. A related work is a theorem in the PhD dissertation of Ng:
Andrew Ng, Shaping and Policy Search in Reinforcement Learning, PhD Dissertation, 2003.
(Theorem 5 in Appendix 3.B: Learning with a smaller horizon).
It is shown that if the error between Phi (equivalent to V^e here) and V* is small, one may choose a discount factor gamma’ that is smaller than gamma of the original MDP, and still have some guarantees. As the discount factor has an interpretation of the effective planning horizon, this result is relevant. The result, however, is not directly comparable to this work as the planning horizon appears implicitly in the form of 1/(1-gamma’) instead of k, but I believe it is worth to mention and possibly compare.
- The IL setting in this work is that an oracle provides V^e, which is the same as (Ross & Bagnell, 2014). I believe this setting is relatively restrictive as in many problems we only have access to (state, action) pairs, or sequence thereof, and not the associated value function. For example, if a human is showing how a robot or a car should move, we do not easily have access to V^e (unless the reward function is known and we estimate the value with rollouts; which requires us having a long trajectory). This is not a deal breaker, and I would not consider this as a weakness of the work, but the paper should be more clear and upfront about this.
- The use of differential operator nabla instead of gradient of a function (a vector field) in Equations (10), (14), (15) is non-standard.
- Figures are difficult to read, as the colors corresponding to confidence regions of different curves are all mixed up. Maybe it is better to use standard error instead of standard deviation.
===
After Rebuttal: Thank you for your answer. The revised paper has been improved. I increase my score accordingly.


We introduce a new approach to estimate continuous actions using actor-critic algorithms for reinforcement learning problems. Policy gradient methods usually predict one continuous action estimate or parameters of a presumed distribution (most commonly Gaussian) for any given state which might not be optimal as it may not capture the complete description of the target distribution. Our approach instead predicts M actions with the policy network (actor) and then uniformly sample one action during training as well as testing at each state. This allows the agent to learn a simple stochastic policy that has an easy to compute expected return. In all experiments, this facilitates better exploration of the state space during training and converges to a better policy.

This paper describes an approach to stochastic control using RL that extends DDPG with a stochastic policy.  A standard DDPG setup is extended such that the actor now produces M actions at each timestep.  Only one of the M actions will be executed in the environment using a uniform sampling.  The sampled action is the only that will receive a gradient update from the critic network. The authors demonstrate that such a stochastic policy performs better on average in a series of benchmark control tasks.
I find the general idea of the work compelling, but the particular approach is rather poor.  The fact that we are choosing the number of modes in the uniform distribution is a bit underwhelming (a more compelling architecture could have proposed a policy conditioned on gaussian noise for example, thus having better coverage of the distribution). I found the proposed apprach to be under-analyzed and the stochastic aspects of the policy are undervalued.   The main claim being argued in the paper is that the proposed stochastic policy has better final performance on average than a deterministic policy, but the only practical difference seems to be a slightly more structured approach to exploration.
However, very little attention is paid to trying different exploration methods with the deterministic policy (incidentally, Ornstein-Uhlenbeck process noise is not something I'm familiar with, a citation to the use of this noise for exploration as well as a more explicit explanation would have been appreciated).  One interpretation is that each of the M sub-policies follows a different mode of the Q-value distribution over the action space.  But is this indeed the case?  There is a brief analysis of this with cartpole, but a more complete look at how actions are clustered in the action space would make this paper much more compelling.  Even in higher-dimensional action spaces, you could look at a t-SNE projection or cluster analysis to try and see how many modes the agent is reasoning over.  Additionally, the baseline agent should have used additional exploration methods as these can quickly change the performance of the agent.
I also think that better learning is not the only redeeming aspect of a stochastic policy.  In the face of a non-stationary environment, a stochastic policy will likely be much more robust.  Additionally, it will have much better performance against adversarial environments.  Given the remaining available space in the paper it would have been interesting to provide more insight into the proposed methods gains in these areas.


For inference operations in deep neural networks on end devices, it is desirable to deploy a single pre-trained neural network model, which can dynamically scale across a computation range without comprising accuracy. To achieve this goal, Incomplete Dot Product (IDP) has been proposed to use only a subset of terms in dot products during forward propagation. However, there are some limitations, including noticeable performance degradation in operating regions with low computational costs, and essential performance limitations since IDP uses hand-crafted profile coefficients. In this paper, we extend IDP by proposing new training algorithms involving a single profile, which may be trainable or pre-determined, to significantly improve the overall performance, especially in operating regions with low computational costs. Specifically, we propose the Task-wise Early Stopping and Loss Aggregation (TESLA) algorithm, which is showed in our 3-layer multilayer perceptron on MNIST that outperforms the original IDP by 32% when only 10% of dot products terms are used and achieves 94.7% accuracy on average. By introducing trainable profile coefficients, TESLA further improves the accuracy to 95.5% without specifying coefficients in advance. Besides, TESLA is applied to the VGG-16 model, which achieves 80% accuracy using only 20% of dot product terms on CIFAR-10 and also keeps 60% accuracy using only 30% of dot product terms on CIFAR-100, but the original IDP performs like a random guess in these two datasets at such low computation costs. Finally, we visualize the learned representations at different dot product percentages by class activation map and show that, by applying TESLA, the learned representations can adapt over a wide range of operation regions.

An approach to adjust inference speed, power consumption or latency by using incomplete dot products McDanel et al. (2017) is investigated.
The approach is based on `profile coefficients’ which are learned for every channel in a convolution layer, or for every column in the fully connected layer. Based on the magnitude of this profile coefficient, which determines the importance of this `filter,’ individual components in a neural net are switched on or off. McDanel et al. (2017) propose to train such an approach in a stage-by-stage manner.
Different from a recently proposed method by McDanel et al. (2017), the authors of this submission argue that the stage-by-stage training doesn’t fully utilize the deep net performance. To address this issue a `loss aggregation’ is proposed which jointly optimizes a deep net when multiple fractions of incomplete products are used.
The method is evaluated on the MNIST and CIFAR-10 datasets and shown to outperform work on incomplete dot products by McDanel et al. (2017) by 32% in the low resource regime.
Summary:
——
In summary, I think the paper proposes an interesting approach but more work is necessary to demonstrate the effectiveness of the discussed method. The results are preliminary and should be extended to CIFAR-100 and ImageNet to be convincing. In addition, the writing should be improved as it is often ambiguous. See below for details.
Review:
—————
1. Experiments are only provided on very small datasets. According to my opinion, this isn’t sufficient to illustrate the effectiveness of the proposed approach. As a reader I wouldn’t want to see results on CIFAR-100 and ImageNet using multiple network architectures, e.g., AlexNet and VGG16.
2. Usage of the incomplete dot product for the fully connected layer and the convolutional layer seems inconsistent. More specifically, while the profile coefficient is applied for every input element in Eq. (1), it’s applied based on output channels in Eq. (2). This seems inconsistent and a comment like `These two approaches, however, are equivalent with negligible difference induced by the first hidden layer’ is more confusing than clarifying.
3. The writing should be improved significantly and statements should be made more precise, e.g., `From now on, x% DP, where \leq x \geq 100, means the x% of terms used in dot products’. While sentences like those can be deciphered, they aren’t that appealing.
4. The loss functions in Eq. (3) should be made more precise. It remains unclear whether the profile coefficients and the weights are trained jointly, separately, incrementally etc.
5. Algorithm 1 and Algorithm 2 call functions that aren’t described/defined.
6. Baseline numbers for training on datasets without incomplete dot products should be provided.


Published as a conference paper at ICLR 2018 PREDICTING FLOOR LEVEL FOR 911 CALLS WITH NEURAL NETWORKS AND SMARTPHONE SENSOR DATA
In cities with tall buildings, emergency responders need an accurate floor level location to find 911 callers quickly. We introduce a system to estimate a victim's floor level via their mobile device's sensor data in a two-step process. First, we train a neural network to determine when a smartphone enters or exits a building via GPS signal changes. Second, we use a barometer equipped smartphone to measure the change in barometric pressure from the entrance of the building to the victim's indoor location. Unlike impractical previous approaches, our system is the first that does not require the use of beacons, prior knowledge of the building infrastructure, or knowledge of user behavior. We demonstrate real-world feasibility through 63 experiments across five different tall buildings throughout New York City where our system predicted the correct floor level with 100% accuracy.

Update: Based on the discussions and the revisions, I have improved my rating. However I still feel like the novelty is somewhat limited, hence the recommendation.
======================
The paper introduces a system to estimate a floor-level via their mobile device's sensor data using an LSTM to determine when a smartphone enters or exits a building, then using the change in barometric pressure from the entrance of the building to indoor location. Overall the methodology is a fairly simple application of existing methods to a problem, and  there remain some methodological issues (see below).
General Comments
- The claim that the bmp280 device is in most smartphones today doesn’t seem to be backed up by the “comScore” reference (a simple ranking of manufacturers).  Please provide the original source for this information.
- Almost all exciting results based on RNNs are achieved with LSTMs, so calling an RNN with LSTM hidden units a new name IOLSTM seems rather strange - this is simply an LSTM.
- There exist models for modelling multiple levels of abstraction, such as the contextual LSTM of [1]. This would be much more satisfying that the two level approach taken here, would likely perform better, would replace the need for the clustering method, and would solve issues such as the user being on the roof.  The only caveat is that it may require an encoding of the building (through a one-hot encoding) to ensure that the relationship between the floor height and barometric pressure is learnt. For unseen buildings a background class could be used, the estimators as used before, or aggregation of the other buildings by turning the whole vector on.
- It’s not clear if a bias of 1 was added to the forget gate of the LSTM or not. This has been shown to improve results [2].
- Overall the whole pipeline feels very ad-hoc, with many hand-tuned parameters.  Notwithstanding the network architecture, here I’m referring to the window for the barometric pressure, the Jaccard distance threshold, the binary mask lengths, and the time window for selecting p0.
- Are there plans to release the data and/or the code for the experiments? Currently the results would be impossible to reproduce.
- The typo of accuracy given by the authors is somewhat worrying, given that the result is repeated several times in the paper.
Typographical Issues
- Page 1:  ”floor-level accuracy” back ticks
- Page  4:   Figure  4.1→Figure  1;  Nawarathne  et  al  Nawarathne  et  al.→Nawarathne et al.
- Page 6:  ”carpet to carpet” back ticks
- Table 2:  What does -4+ mean?
- References.  The references should have capitalisation where appropriate.For example,  Iodetector→IODetector,  wi-fi→Wi-Fi,  apple→Apple, iphone→iPhone, i→I etc.
[1]  Shalini Ghosh, Oriol Vinyals, Brian Strope, Scott Roy, Tom Dean, and LarryHeck. Contextual LSTM (CLSTM) models for large scale NLP tasks. arXivpreprint arXiv:1602.06291, 2016.
[2]  Rafal Jozefowicz, Wojciech Zaremba, and Ilya Sutskever.  An empirical exploration of recurrent network architectures.  InProceedings of the 32nd International Conference on Machine Learning (ICML-15), pages 2342–2350,2015


Modern machine learning algorithms are often susceptible to adversarial examples -maliciously crafted inputs that are undetectable by humans but that fool the algorithm into producing undesirable behavior. In this work, we show that adversarial examples exist in natural language classification: we formalize the notion of an adversarial example in this setting and describe algorithms that construct such examples. Adversarial perturbations can be crafted for a wide range of tasks -including spam filtering, fake news detection, and sentiment analysis -and affect different models -convolutional and recurrent neural networks as well as linear classifiers to a lesser degree. Constructing an adversarial example involves replacing 10-30% of words in a sentence with synonyms that don't change its meaning. Up to 90% of input examples admit adversarial perturbations; furthermore, these perturbations retain a degree of transferability across models. Our findings demonstrate the existence of vulnerabilities in machine learning systems and hint at limitations in our understanding of classification algorithms.

The paper shows that neural networks are sensitive to adversarial perturbation for a set of NLP text classifications. They propose constructing (model-dependent) adversarial examples by optimizing a function J (that doesn't seem defined in the paper) subject to a constraint c(x, x') < \gamma (i.e. that the original input and adversarial input should be similar)
c is composed of two constraints:
1. || v - v' ||_2 < \gamma_1, where v and v' are bag of embeddings for each input
2. |log P(x') - log P(x)| < \gamma_2 where P is a language model
The authors then show that for 3 classification problems "Trec07p", "Yelp", and "News" and 4 models (Naive Bayes, LSTM, word CNNs, deep-char-CNNs) that the models that perform considerably worse on adversarial examples than on the test set.  Furthermore to test the validity of their adversarial examples, the authors show the following:
1. Humans achieve somewhat similar accuracy on the original adverarial examples (8 points higher on one dataset and 8 points lower on the other two)
2. Humans rate the writing quality of both the original and adversarial examples to be similar
3. The adversarial examples only somewhat transfer across models
My main questions/complaints/suggestions for the paper are:
-Novelty/Methodology. The paper has mediocre novelty given other similar papers recently.
On question I have is about whether the generated examples are actually close to the original examples. The authors do show some examples that do look good, but do not provide any systematic study (e.g. via human annotation)
 This is a key challenge in NLP (as opposed to vision where the inputs are continuous so it is easy to perturb them and be reasonably sure that the image hasn't changed much). In NLP however, the words are discrete, and the authors measure the difference between an original example and the adversary only in continuous space which may not actually be a good measure of how different they are.
They do have some constraint that the fraction of changed words cannot differ by more than delta, but delta = 0.5 in the experiments, which is really large! (i.e. 50% of the words could be different according to Algorithm 1)
-Writing: the function J is never mathematically defined, neither is the function c (except that it is known to be composed of the semantic/syntactic similarity constraints).
The authors talk about "syntactic" similarity but then propose a language model constraint. I think is a better word is "fluency" constraint.
The results in Table 3 and Table 6 seem different, shouldn't the diagonal of Table 6 line up with the results in Table 3?
-Experimental methodology (more of a question since authors are unclear): The authors write that "all adversarial examples are generated and evaluated on the test set".
There are many hyperparameters in the proposed authors' approach, are these also tuned on the test set? That is unfair to the base classifier. The adversarial model should be tuned on the validation set, and then the same model should be used to generate test set examples. (The authors can even show the validation adversarial accuracy to show how/if it deviates from the test accuracy)
-Lack of related work in NLP (see the anonymous comment for some examples). Even the related work in NLP that is cited e.g. Jia and Liang 2017 is obfuscated in the last page. The authors' introduction only refers to related works in vision/speech and ignores related NLP work.
Furthermore, adversarial perturbation is related to domain transfer  (since both involve shifts between the training and test distribution) and it is well known for instance that models that are trained on Wall Street Journal perform poorly on other domains.  See SJ Pan and Q Yang, A Survey on transfer learning, 2010, for some example references.


Published as a conference paper at ICLR 2018 LEARNING TO MULTI-TASK BY ACTIVE SAMPLING
One of the long-standing challenges in Artificial Intelligence for learning goaldirected behavior is to build a single agent which can solve multiple tasks. Recent progress in multi-task learning for goal-directed sequential problems has been in the form of distillation based learning wherein a student network learns from multiple task-specific expert networks by mimicking the task-specific policies of the expert networks. While such approaches offer a promising solution to the multi-task learning problem, they require supervision from large expert networks which require extensive data and computation time for training. In this work, we propose an efficient multi-task learning framework which solves multiple goaldirected tasks in an on-line setup without the need for expert supervision. Our work uses active learning principles to achieve multi-task learning by sampling the harder tasks more than the easier ones. We propose three distinct models under our active sampling framework. An adaptive method with extremely competitive multi-tasking performance. A UCB-based meta-learner which casts the problem of picking the next task to train on as a multi-armed bandit problem. A meta-learning method that casts the next-task picking problem as a full Reinforcement Learning problem and uses actor critic methods for optimizing the multi-tasking performance directly. We demonstrate results in the Atari 2600 domain on seven multi-tasking instances: three 6-task instances, one 8-task instance, two 12-task instances and one 21-task instance.

The authors show empirically that formulating multitask RL itself as an active learning and ultimately as an RL problem can be very fruitful.  They design and explore several approaches  to the active learning (or active sampling) problem, from a basic
change to the distribution to UCB to feature-based neural-network based RL. The domain is video games.   All proposed approaches beat the uniform sampling baselines and the more sophisticated approaches do better in the scenarios with more tasks (one multitask  problem had 21 tasks).
Pros:
- very promising results with an interesting active learning approach to multitask RL
- a number of approaches developed for the basic idea
- a variety of experiments, on challenging multiple task problems (up to 21 tasks/games)
- paper is overall well written/clear
Cons:
- Comparison only to a very basic baseline (i.e. uniform sampling)
Couldn't comparisons be made, in some way, to other multitask work?
Additional  comments:
- The assumption of the availability of a target score goes against
the motivation that one need not learn individual networks ..  authors
say instead one can use 'published' scores, but that only assumes
someone else has done the work (and furthermore, published it!).
The authors do have a section on eliminating the need by doubling an
estimate for each task) which makes this work more acceptable (shown
for 6 tasks or MT1, compared to baseline uniform sampling).
Clearly there is more to be done here for a future direction (could be
mentioned in future work section).
- The averaging metrics (geometric, harmonic vs arithmetic, whether
  or not to clip max score achieved) are somewhat interesting, but in
  the main paper, I think they are only used in section 6 (seems like
  a waste of space). Consider moving some of the results, on showing
  drawbacks of arithmetic mean with no clipping (table 5 in appendix E), from the appendix to
  the main paper.
- The can be several benefits to multitask learning, in particular
  time and/or space savings in learning new tasks via learning more
  general features. Sections 7.2 and 7.3 on specificity/generality of
  features were interesting.
--> Can the authors show that a trained network (via their multitask
    approached) learns significantly faster on a brand new game
    (that's similar to games already trained on), compared to learning from
    scratch?
--> How does the performance improve/degrade (or the variance), on the
    same set of tasks, if the different multitask instances (MT_i)
    formed a supersets hierarchy, ie if MT_2 contained all the
    tasks/games in MT_1, could training on MT_2 help average
    performance on the games in MT_1 ? Could go either way since the network
   has to allocate resources to learn other games too.  But is there a pattern?
- 'Figure 7.2' in section 7.2 refers to Figure 5.
- Can you motivate/discuss better why not providing the identity of a
  game as an input is an advantage? Why not explore both
  possibilities? what are the pros/cons? (section 3)


Published as a conference paper at ICLR 2018 AUTOMATICALLY INFERRING DATA QUALITY FOR SPATIOTEMPORAL FORECASTING
Spatiotemporal forecasting has become an increasingly important prediction task in machine learning and statistics due to its vast applications, such as climate modeling, traffic prediction, video caching predictions, and so on. While numerous studies have been conducted, most existing works assume that the data from different sources or across different locations are equally reliable. Due to cost, accessibility, or other factors, it is inevitable that the data quality could vary, which introduces significant biases into the model and leads to unreliable prediction results. The problem could be exacerbated in black-box prediction models, such as deep neural networks. In this paper, we propose a novel solution that can automatically infer data quality levels of different sources through local variations of spatiotemporal signals without explicit labels. Furthermore, we integrate the estimate of data quality level with graph convolutional networks to exploit their efficient structures. We evaluate our proposed method on forecasting temperatures in Los Angeles.

The paper is an application of neural nets to data quality assessment. The authors introduce a new definition of data quality that relies on the notion of local variation defined in (Zhou and Schölkopf, 2004), and they extend it to multiple heterogenous data sources. The data quality function is learned using a GCN as defined in (Kipf and Welling, 2016).

1) How many neighbors are used in the experiments? Is this fixed or is defined purely by the Gaussian kernel weights as mentioned in 4.2? Setting all weights less than 0.9 to zero seems quite abrupt. Could you provide a reason for this? How many neighbors fit in this range?
2) How many data points? What is the temporal resolution of your data (every day/hour/minute/etc.)? What is the value of N, T?
3) The bandwidth of the Gaussian kernel (\gamma) is quite different (0.2 and 0.6) for the two datasets from Weather Underground (WU) and Weather Bug (WB) (sect. 4.2). The kernel is computed on the features (e.g., latitude, longitude, vegetation fraction, etc.). Location, longitude, distance from coast, etc. are the same no matter the data source (WU or WB). Maybe the way they compute other features (e.g., vegetation fraction) vary slightly, but I would guess the \gamma should be (roughly) the same.
4) Are the features (e.g., latitude, longitude, vegetation fraction, etc.) normalized in the kernel? If they are given equal weight (which is in itself questionable) they should be normalized, otherwise some of them will always dominate the distances. If they were indeed normalized, that should be made clear in the paper.
5) Why do you choose the summer months? How does the framework perform on other months and other meteorological signals except temperature? The application is very nice and complex, but I find that the experiments are a little bit too limited.
6) The notations w and W are used for different things, and that is slightly confusing. Usually one is used as a matrix notation of the other.
7) I would tend to associate data quality with how noisy observations are at a certain node, and not heterogeneity. It would be good to add some discussion on noise in the paper. How do you define an anomaly in 5.2? Is it a spatial or temporal anomaly? Not sure I understand the difference between an anomaly and a bridge node.
8) “A bridge node highly likely has a lower data quality level due to the heterogeneity”. I would think a bridge node is very relevant, and I don't necessarily see it as having a lower data quality. This approach seems to give more weight to very similar data, while discarding the transitions, which in a meteorological setting, could be the most relevant ?!
9) Update the references, some papers have updated information (e.g., Bruna et al. - ICLR 2014, Kipf and Welling – ICLR 2017, etc.).
Quality – The experiments need more work and editing as mentioned in the comments.
Clarity – The framework is fairly clearly presented, however the English needs significant improvement.
Originality – The paper is a nice application of machine learning and neural nets to data quality assessment, and the forecasting application is relevant and challenging. However the paper mostly provides a framework that relies on existing work.
Significance – While the paper could be relevant to data quality applications by introducing advanced machine learning techniques, it has limited reach outside the field. Maybe publish it in a data quality conference/journal.


Neural networks have recently had a lot of success for many tasks. However, neural network architectures that perform well are still typically designed manually by experts in a cumbersome trial-and-error process. We propose a new method to automatically search for well-performing CNN architectures based on a simple hill climbing procedure whose operators apply network morphisms, followed by short optimization runs by cosine annealing. Surprisingly, this simple method yields competitive results, despite only requiring resources in the same order of magnitude as training a single network. E.g., on CIFAR-10, our method designs and trains networks with an error rate below 6% in only 12 hours on a single GPU; training for one day reduces this error further, to almost 5%.

This paper proposes a variant of neural architecture search.  It uses established work on network morphisms as a basis for defining a search space.  Experiments search for effective CNN architectures for the CIFAR image classification task.
Positives:
(1) The approach is straightforward to implement and trains networks in a reasonable amount of time.
(2) An advantage over prior work, this approach integrates architectural evolution with the training procedure.  Networks are incrementally grown; child networks are initialized with learned parameters from their parents.  This eliminates the need to restart training when making an architectural change, and drastically speeds the search.
Negatives:
(1) The state-of-the-art CNN architectures are not mysterious or difficult to find, despite the paper's characterization of them being so.  Indeed, ResNet and DenseNet designs are both guided by extremely simple principles: stack a series of convolutional layers, pool occasionally, and use some form of skip-connection throughout.  The need for architectural search is unclear.
(2) The proposed search space is boring.  As described in Section 4, the possibly evolutionary changes are limited to deepening the network, widening the network, and adding a skip connection.  But these are precisely the design aspects that have been well-explored by human trial and error and for which good rules of thumb are already available.
(3) As a consequence of (1) and (2), the result is essentially rigged.  Since only depth, width, and skip connections are considered, the end network must end up looking like a ResNet or DenseNet, but with some connections pruned.  There is no way to discover a network outside of the principled design space articulated in point (1) above.  Indeed, the discovered network diagrams (Figures 4 and 5) fall in this space.
(4) Performance is worse than the best hand-designed baselines.  One would hope that, even if the search space is limited, the discovered networks might be more efficient or higher performing in comparison to the human designs which fall within that same space.  However, the results in Tables 3 and 4 show this not to be the case.  The best human designs outperform the evolved networks.  Moreover, the evolved networks are woefully inefficient in terms of parameter count.
Together, these negatives imply the proposed approach is not yet at the point of being useful in practice.  I think further work is required (perhaps expanding the search space) to resolve the current limitations of automated architecture search.
Misc:
Tables 3 and 4 would be easier to parse if resources were simply reported in terms of total GPU hours.


FUSIONNET: FUSING VIA FULLY-AWARE ATTENTION WITH APPLICATION TO MACHINE COMPREHENSION
This paper introduces a new neural structure called FusionNet, which extends existing attention approaches from three perspectives. First, it puts forward a novel concept of "history of word" to characterize attention information from the lowest word-level embedding up to the highest semantic-level representation. Second, it identifies an attention scoring function that better utilizes the "history of word" concept. Third, it proposes a fully-aware multi-level attention mechanism to capture the complete information in one text (such as a question) and exploit it in its counterpart (such as context or passage) layer by layer. We apply FusionNet to the Stanford Question Answering Dataset (SQuAD) and it achieves the first position for both single and ensemble model on the official SQuAD leaderboard at the time of writing (Oct. 4th, 2017). Meanwhile, we verify the generalization of FusionNet with two adversarial SQuAD datasets and it sets up the new state-of-the-art on both datasets: on AddSent, FusionNet increases the best F1 metric from 46.6% to 51.4%; on AddOneSent, FusionNet boosts the best F1 metric from 56.0% to 60.7%.

(Score before author revision: 4)
(Score after author revision: 7)
I think the authors have taken both the feedback of reviewers as well as anonymous commenters thoroughly into account, running several ablations as well as reporting nice results on an entirely new dataset (MultiNLI) where they show how their multi level fusion mechanism improves a baseline significantly. I think this is nice since it shows how their mechanism helps on two different tasks (question answering and natural language inference).
Therefore I would now support accepting this paper.
------------(Original review below) -----------------------
The authors present an enhancement to the attention mechanism called "multi-level fusion" that they then incorporate into a reading comprehension system. It basically takes into account a richer context of the word at different levels in the neural net to compute various attention scores.
i.e. the authors form a vector "HoW" (called history of the word), that is defined as a concatenation of several vectors:
HoW_i = [g_i, c_i, h_i^l, h_i^h]
where g_i = glove embeddings, c_i = COVE embeddings (McCann et al. 2017), and h_i^l and h_i^h are different LSTM states for that word.
The attention score is then a function of these concatenated vectors i.e. \alpha_{ij} = \exp(S(HoW_i^C, HoW_j^Q))
Results on SQuAD show a small gain in accuracy (75.7->76.0 Exact Match). The gains on the adversarial set are larger but that is because some of the higher performing, more recent baselines don't seem to have adversarial numbers.
The authors also compare various attention functions (Table 5) showing a particularone (Symmetric + ReLU) works the best.
Comments:
-I feel overall the contribution is not very novel.  The general neural architecture that the authors propose in Section 3 is generally quite similar to the large number of neural architectures developed for this dataset (e.g. some combination of attention between question/context and LSTMs over question/context). The only novelty is these "HoW" inputs to the extra attention mechanism that takes a richer word representation into account.
-I feel the model is seems overly complicated for the small gain (i.e. 75.7->76.0 Exact Match), especially on a relatively exhausted dataset (SQuAD) that is known to have lots of pecularities (see anonymous comment below). It is possible the gains just come from having more parameters.
-The authors (on page 6) claim that that by running attention multiple times with different parameters but different inputs (i.e. \alpha_{ij}^l, \alpha_{ij}^h, \alpha_{ij}^u) it will learn to attend to "different regions for different level". However, there is nothing enforcing this and the gains just probably come from having more parameters/complexity.


Although word analogy problems have become a standard tool for evaluating word vectors, little is known about why word vectors are so good at solving these problems. In this paper, I attempt to further our understanding of the subject, by developing a simple, but highly accurate generative approach to solve the word analogy problem for the case when all terms involved in the problem are nouns. My approach solves the word analogy problem using a small fraction of the data that is typically used to train word vectors. My results demonstrate the ambiguities associated with learning the relationship between a word pair, and the role of the training dataset in determining the relationship which gets most highlighted. Furthermore, my results show that the ability of a model to accurately solve the word analogy problem may not be indicative of a model's ability to learn the relationship between a word pair the way a human does.

This paper proposes a new method for solving the analogy task, which can potentially provide some insight as to why word2vec recovers word analogies.
In my view, there are three main issues with this paper: (1) the assumptions it makes about our understanding of the analogy phenomenon; (2) the authors' understanding of the proposed method, what it models, and its relation to prior art; (3) the very selective subset of analogies that the author used for evaluation.
ASSUMPTIONS
The author assumes that there the community does not understand why word embedding methods such as word2vec recover analogies. I believe that, in fact, we do have a good understanding of this phenomena. Levy & Goldberg [1] showed that optimizing x for
argmax(cos(x, A - B + C))
is equivalent to optimizing x for
argmax(cos(x, A) - cos(x, B) + cos(x, C))
which means that one can interpret this objective as searching for a word x that is similar to A, similar to C, and dis-similar to B. Linzen [2] cemented this explanation by removing the negative term (-cos(x, B)) and showing that for a wide variety of analogies, the method still works. Drozd et al [3] and Rogers et al [4] also argue that the original datasets used by Mikolov et al were too easy because they focused on encyclopedic facts, and expand these datasets to other non-encyclopedic relations, which are significantly more difficult to solve using simple vector arithmetic.
In other words, we know why word2vec is able to solve analogies via vector arithmetic: because many analogies (like those in Mikolov et al's original dataset) are "gameable", and can be solved by finding a term that is similar to A and similar to C at the same time. For example, if A="woman" and C="king", then x="queen" fits the bill.
METHOD
From what I can understand, the proposed method models the 3-way co-occurrence between A, B, and a context noun (let's call it R). Leveraging the distribution of (X, R, Y) for solving problems in lexical semantics has been studied quite a bit in the past, e.g. Latent Relational Analysis [5] and even Hearst patterns. I think the current description overlooks this major deviation from word2vec and other distributional methods, which only model the 2-way co-occurrence (X, R). This is a much more profound difference than just filtering non-nouns. I think the proposed method should be redescribed in these terms, and compared to other work that modeled 3-way co-occurrences.
EVALUATION DATA
The author evaluates their method on a subset of the original analogy task, which is very limited. I would like to see an evaluation on (A) the original two datasets of Mikolov et al (without non-nouns), and (B) the larger datasets provided by Drozd et al [3] and Rogers et al [4].
In addition, since I think the analogy phenomenon is well understood, I would like to see some demonstration that this method has added value beyond the analogy benchmark.
MISCELLANEOUS COMMENTS
* The author does not state the important fact that when searching for the closest x to A - B + C, the search omits A, B, and C. It is often the case that the result is A or C without this omission.
* The paper is partially de-anonymized (e.g. links and acknowledgements).
* One of the problems with modeling 3-way co-occurrences (as opposed to 2-way co-occurrences) is that they are much sparser. I think this is a more precise explanation for why the currency relation is particularly hard to capture with this method.
[1] http://www.aclweb.org/anthology/W14-1618
[2] http://anthology.aclweb.org/W16-2503
[3] http://aclweb.org/anthology/C/C16/C16-1332.pdf
[4] http://www.aclweb.org/anthology/S17-1017
[5] https://arxiv.org/pdf/cs/0508053.pdf


Nodes residing in different parts of a graph can have similar structural roles within their local network topology. The identification of such roles provides key insight into the organization of networks and can also be used to inform machine learning on graphs. However, learning structural representations of nodes is a challenging unsupervised-learning task, which typically involves manually specifying and tailoring topological features for each node. Here we develop GRAPHWAVE, a method that represents each node's local network neighborhood via a low-dimensional embedding by leveraging spectral graph wavelet diffusion patterns. We prove that nodes with similar local network neighborhoods will have similar GRAPHWAVE embeddings even though these nodes may reside in very different parts of the network. Our method scales linearly with the number of edges and does not require any hand-tailoring of topological features. We evaluate performance on both synthetic and real-world datasets, obtaining improvements of up to 71% over state-of-the-art baselines.

The term "structural equivalence" is used incorrectly in the paper.  From sociology, two nodes with the same position are in an equivalence relation.  An equivalence, Q, is any relation that satisfies these three conditions:
  - Transitivity: (a,b), (b,c) ∈ Q ⇒ (a,c) ∈Q
  - Symmetry: (a, b) ∈ Q if and only if (b, a) ∈Q
  - Reflexivity: (a, a) ∈Q
There are three deterministic equivalences: structural, automorphic, and regular.
From Lorrain & White (1971), two nodes u and v are structurally equivalent if they have the same relationships to all other nodes.  Exact structural equivalence is rare in real-world networks.
From Borgatti, et al. (1992) and Sparrow (1993), two nodes u and v are automorphically equivalent if all the nodes can be relabeled to form an isomorphic graph with the labels of u and v interchanged.
From Everett & Borgatti (1992), two nodes u and v are regularly equivalent if they are equally related to equivalent others.
Parts of this statement are false: "A notable example of such approaches is RolX (Henderson et al., 2012), which aims to recover a soft-clustering of nodes into a predetermined number of K distinct roles using recursive feature extraction (Henderson et al., 2011)."  RolX (as described in KDD 2012 paper) uses MDL to automatically determine the number of roles.
As indicated above, this statement is also false: "We note that RolX requires the number of desired structural classes as input, ...".
The paper does not discuss how the free parameter d (which represents the number of evenly spaced sample points)  is chosen.
This statement is misleading: "In particular, a small perturbation of the graph yields small perturbations of the eigenvalues."  What is considered a small perturbation?  One can delete an edge (seemingly a small perturbation) and change the eigenvalues of the Laplacian dramatically --  e.g., deleting an edge that increases the number of connected components.
The barbell graph experiment seemed contrived.  Why would one except such a graph to have 8 classes?  Why not 3?  One for cliques, one for the chain, and one for connectors of the clique to the chain.
In Section 4.2, how many roles were selected for RolX?
The paper states: "Furthermore, nodes from different graphs can be embedded into the same space and their structural roles can be compared across different graphs."  Experiments were not conducted to see how the competing approaches such as RolX compare with GraphWave on transfer learning tasks.
Gilpin et al (KDD 2013) extended RolX to incorporate sparsity and diversity constraints on the role space and showed that their approach is superior to RolX on measuring distances.  This is applicable to experiments in Figure 4.
I strongly recommend running experiments that test the predictive power of the roles found by GraphWave.


Published as a conference paper at ICLR 2018 LEARNING LATENT PERMUTATIONS WITH GUMBEL- SINKHORN NETWORKS
Permutations and matchings are core building blocks in a variety of latent variable models, as they allow us to align, canonicalize, and sort data. Learning in such models is difficult, however, because exact marginalization over these combinatorial objects is intractable. In response, this paper introduces a collection of new methods for end-to-end learning in such models that approximate discrete maximum-weight matching using the continuous Sinkhorn operator. Sinkhorn operator is attractive because it functions as a simple, easy-to-implement analog of the softmax operator. With this, we can define the Gumbel-Sinkhorn method, an extension of the Gumbel-Softmax method (Jang et al., 2016;Maddison et al., 2016) to distributions over latent matchings. We demonstrate the effectiveness of our method by outperforming competitive baselines on a range of qualitatively different tasks: sorting numbers, solving jigsaw puzzles, and identifying neural signals in worms.

Quality: The paper is built on solid theoretical grounds and supplemented by experimental demonstrations. Specifically, the justification for using the Sinkhorn operator is given by theorem 1 with proof given in the appendix. Because the theoretical limit is unachievable, the authors propose to truncate the Sinkhorn operator at level $L$. The effect of approximation for the truncation level $L$ as well as the effect of temperature $\tau$ are demonstrated nicely through figures 1 and 2(a). The paper also presents a nice probabilistic approach to permutation learning, where the doubly stochastic matrix arises from Gumbel matching distribution.
Clarity: The paper has a good flow, starting out with the theoretical foundation, description of how to construct the network, followed by the probabilistic formulation. However, I found some of the notation used to be a bit confusing.
1. The notation $l$ appears in Section 2 to denote the number of iterations of Sinkhorn operator. In Section 3, the notation $l$ appears as $g_l$, where in this case, it refers to the layers in the neural network. This led me to believe that there is one Sinkhorn operator for each layer of neural network. But after reading the paper a few times, it seemed to me that the Sinkhorn operator is used only at the end, just before the final output step (the part where it says the truncation level was set to $L=20$ for all of the experiments confirmed this). If I'm correct in my understanding, perhaps different notation need to be used for the layers in the NN and the Sinkhorn operator. Additionally, it would have been nice to see a figure of the entire network architecture, at least for one of the applications considered in the paper.
2. The distinction between $g$ and $g_l$ was also a bit unclear. Because the input to $M$ (and $S$) is a square matrix, the function $g$ seems to be carrying out the task of preparing the final output of the neural network into the input formate accepted by the Sinkhorn operator. However, $g$ is stated as "the output of the computations involving $g_l$". I found this statement to be a bit unclear and did not really describe what $g$ does; of course my understanding may be incorrect so a clarification on this statement would be helpful.
Originality: I think there is enough novelty to warrant publication.  The paper does build on a set of previous works, in particular Sinkhorn operator, which achieves continuous relaxation for permutation valued variables. However, the paper proposes how this operator can be used with standard neural network architectures for learning permutation valued latent variable. The probabilistic approach also seems novel. The applications are interesting, in particular, it is always nice to see a machine learning method applied to a unique application; in this case from computational neuroscience.
Other comments:
1. What are the differences between this paper and the paper by Adams and Zemel (2011)? Adams and Zemel also seems to propose Sinkhorn operator for neural network. Although they focus only on the document ranking problem, it would be good to hear the authors' view on what differentiates their work from Adams and Zemel.
2. As pointed out in the paper, there is a concurrent work: DeepPermNet. Few comments regarding the difference between their work and this work would also be helpful as well.
Significance: The Sinkhorn network proposed in the paper is useful as demonstrated in the experiments. The methodology appears to be straight forward  to implement using the existing software libraries, which should help increase its usability.
The significance of the paper can greatly improve if the methodology is applied to other popular machine learning applications such as document ranking, image matching, DNA sequence alignment, and etc. I wonder how difficult it is to extend this methodology to bipartite matching problem with uneven number of objects in each partition, which is the case for document ranking. And for problems such as image matching (e.g., matching landmark points), where each point is associated with a feature (e.g., SIFT), how would one formulate such problem in this setting?