The task of Reading Comprehension with Multiple Choice Questions, requires a human (or machine) to read a given {passage, question} pair and select one of the n given options. The current state of the art model for this task first computes a query-aware representation for the passage and then selects the option which has the maximum similarity with this representation. However, when humans perform this task they do not just focus on option selection but use a combination of elimination and selection. Specifically, a human would first try to eliminate the most irrelevant option and then read the document again in the light of this new information (and perhaps ignore portions corresponding to the eliminated option). This process could be repeated multiple times till the reader is finally ready to select the correct option. We propose ElimiNet, a neural network based model which tries to mimic this process. Specifically, it has gates which decide whether an option can be eliminated given the {document, question} pair and if so it tries to make the document representation orthogonal to this eliminatedd option (akin to ignoring portions of the document corresponding to the eliminated option). The model makes multiple rounds of partial elimination to refine the document representation and finally uses a selection module to pick the best option. We evaluate our model on the recently released large scale RACE dataset and show that it outperforms the current state of the art model on 7 out of the 13 question types in this dataset. Further we show that taking an ensemble of our elimination-selection based method with a selection based method gives us an improvement of 7% (relative) over the best reported performance on this dataset.

In this paper, a model is built for reading comprehension with multiple choices. The model consists of three modules: encoder, interaction module and elimination module. The major contributions are two folds: firstly, proposing the interesting option elimination problem for multi-step reading comprehension;  and secondly, proposing the elimination module where a eliminate gate is used to select different orthogonal factors from the document representations. Intuitively, one answer option can be viewed as eliminated if the document representation vector has its factor along the option vector ignored.
The elimination module is interesting, but the usefulness of “elimination” is not well justified for two reasons. First, the improvement of the proposed model over the previous state of the art is limited. Second, the model is built upon GAR until the elimination module, then according to Table 1 it seems to indicate that the elimination module does not help significantly (0.4% improvement).
In order to show the usefulness of the elimination module, the model should be exactly built on the GAR with an additional elimination module (i.e. after removing the elimination module, the performance should be similar to GAR but not something significantly worse with a 42.58% accuracy). Then we can explicitly compare the performance between GAR and the GAR w/ elimination module to tell how much the new module helps.
Other issues:
1) Is there any difference to directly use $x$ and $h^z$ instead of $x^e$ and $x^r$ to compute $\tilde{x}_i$? Even though the authors find the orthogonal vectors, they’re gated summed together very soon. It would be better to show how much “elimination” and “subtraction” effect the final performance, besides the effect of subtraction gate.
2) A figure showing the model architecture and the corresponding QA process will better help the readers understand the proposed model.
3) $c_i$ in page 5 is not defined. What’s the performance of only using $s_i$ for answer selection or replacing $x^L$ with $s_i$ in score function?
4) It would be better to have the experiments trained with different $n$ to show how multi-hop effects the final performance, besides the case study in Figure 3.
Minor issues:
1) In Eqn. (4), it would be better to use a vector as the input of softmax.
2) It would be easier for discussion if the authors could assign numbers to every equation.


Within-class variation in a high-dimensional dataset can be modeled as being on a low-dimensional manifold due to the constraints of the physical processes producing that variation (e.g., translation, illumination, etc.). We desire a method for learning a representation of the manifolds induced by identity-preserving transformations that can be used to increase robustness, reduce the training burden, and encourage interpretability in machine learning tasks. In particular, what is needed is a representation of the transformation manifold that can robustly capture the shape of the manifold from the input data, generate new points on the manifold, and extend transformations outside of the training domain without significantly increasing the error. Previous work has proposed algorithms to efficiently learn analytic operators (called transport operators) that define the process of transporting one data point on a manifold to another. The main contribution of this paper is to define two transfer learning methods that use this generative manifold representation to learn natural transformations and incorporate them into new data. The first method uses this representation in a novel randomized approach to transfer learning that employs the learned generative model to map out unseen regions of the data space. These results are shown through demonstrations of transfer learning in a data augmentation task for few-shot image classification. The second method use of transport operators for injecting specific transformations into new data examples which allows for realistic image animation and informed data augmentation. These results are shown on stylized constructions using the classic swiss roll data structure and in demonstrations of transfer learning in a data augmentation task for few-shot image classification. We also propose the use of transport operators for injecting transformations into new data examples which allows for realistic image animation.

Overview:
The paper aim to model non-linear, intrinsically low-dimensional structure, in data by estimating "transport operators" that predict how points move along the manifold. This is an old idea, and the stated contribution of the paper is:
"The main contribution of this paper is to show that the manifold representation learned in the transport operators is valuable both as a probabilistic model to improve general machine learning tasks as well as for performing transfer learning in classification tasks."
The paper provide nice illustrative experiments arguing why transport operators may be a useful modeling tool, but does not go beyond illustrative experiments.
While I follow the intuitions behind transport operators I am doubtful if they will generalize beyond very simple manifold structures (see detailed comments below).
Quality:
The paper is well-written and fairly easy to follow. In particular, I appreciate that the authors make no attempt to overclaim contributions. From a methodology point-of-view, the paper has limited novelty (transport operators, and learning thereof has been studied elsewhere), but there are some technical insights (likelihood model, use in data augmentation). Since the provided experiments are mostly illustrations, I would argue that the significance of the paper is limited. I'd say that to really convince a broader audience that this old idea is worth revisiting, the work must go beyond illustrations and apply to a real data problem.
Detailed Comments and Questions:
*) Equation 1 of the paper describe the key dynamics of the applied transport operators. Basically, the paper assume that the underlying data manifold is locally governed by a linear differential equation. This is a very suitable assumption, e.g., for the swiss roll data set, but it is unclear to this reader why it is a suitable assumption beyond such toy data. I would very much appreciate a detailed discussion of when this is a suitable modeling choice, and when it is not. My intuition is that this is mostly a suitable model when the data manifold appears due to simple transformations (e.g. rotations) of data. This is also exactly the type of data considered in the paper.
*) In Eq. 3, should it be "expm" instead of "exp" ?
*) The first two paragraphs of Sec. 2 are background material, whereas paragraph 3 and beyond describe material that is key to the paper. I would recommend introducing a \subsection (or something like it) to make this more clear.
*) The idea of working with transformations of data rather than the actual data is the corner-stone of Ulf Grenander's renowned "Pattern Theory". A citation to this seminal work would be appropriate.
*) In the first paragraph of the introduction links are drawn to the neuroscience literature; it would be appropriate to cite a suitable publication.
Pros(+) & Cons(-):
+ Well-written.
+ Good illustrative experiments.
- Real-life experiments are lacking.
- Limited methodology contribution.
- The assumed dynamics might be too simplistic (at least a discussing of this is missing).
For the AC:
The submitted paper acknowledges several grants (including grant numbers), which can directly be tied to the authors identity. This may be a violation of the double blind review policy. I did not use this information to determine the authors identity, though, so this review is still double blind.
Post-rebuttal comments:
The paper has improved with the incorporated revisions, but my main concerns remain. I find the Swiss Roll / rotated-USPS examples to be too contrived as the dynamics are exactly tailored to the linear ODE assumption. These are examples where the model assumptions are perfect. What is unclear is how the model behaves when the linear ODE assumption is not-quite-correct-but-also-not-totally-incorrect, i.e. how the model behaves in real life. I didn't get that from the newly added experiment. So, I'll keep my rating as is.


Entailment vectors are a principled way to encode in a vector what information is known and what is unknown. They are designed to model relations where one vector should include all the information in another vector, called entailment. This paper investigates the unsupervised learning of entailment vectors for the semantics of words. Using simple entailment-based models of the semantics of words in text (distributional semantics), we induce entailment-vector word embeddings which outperform the best previous results for predicting entailment between words, in unsupervised and semi-supervised experiments on hyponymy.

This work proposes to learn word vectors that are intended to specifically model the lexical entailment relationship. This is achieved in an unsupervised manner from unstructured data, through an approach heavily influenced by recent work by Henderson and Popa, which "reinterprets word2vec" by modeling distributions over discrete latent "pseudo-phrase" vectors. That is, instead of using two vectors per word, as in word2vec, a latent representation is introduced that models the joint properties of the target and context words. While Henderson and Popa represent the latent vector as the evidence for the target and context, or the likelihood, this work suggests to represent it based on the posterior distribution instead. The resultant representations are evaluated on Weeds et al.'s (2014) version of BLESS, as well as the full BLESS dataset, where they do better than the original.
The paper is confusingly written, fails to mention a lot of related work, has a weak evaluation where it doesn't compare to related systems, and I feel that it would benefit from "toning down". Hence, I do not recommend it for acceptance. In more detail:
1. The idea behind Henderson and Popa's model, as well as the suggested modification, should be easy to explain, but I really had to struggle to make sense of it. This work relies very heavily on that paper, and would be better off if it was more standalone. I think part of the confusion stems from using y for the latent representation but not specifying whether it is a word or latent representation in Equation 1 - that only becomes obvious later. The exposition clearly needs more work, and more precise technical writing.
2. There is a lot of related work around word embeddings that is not mentioned, both on word2vec-style representation learning (e.g. it would be useful to relate this more to word2vec and what it learns, as in Omer Levy's work on "interpreting" word2vec, rather than reinterpreting) and word embeddings on hypernymy detection and lexical entailment (see e.g. Stephen Roller's thesis for references).
3. There has been a lot of work on the Weeds BLESS dataset that is not mentioned, or compared against, including unsupervised approaches (e.g. Levy's work, Santus's work, Kiela's work, Roller's work, etc.), that perform better than the numbers in Table 1. There are many other datasets that measure lexical entailment, none of which are evaluated on (apart from the original BLESS set, which is mentioned in passing). It would make sense to show that the method works on more than one dataset, and to do a thorough comparison against other work; especially given that:
4. The tone of the work appears to imply that word2vec was wrong and needs to be reinterpreted: the work leads to "unprecedented results" (not true), claims to be a completely novel method for inducing word representations (together with LSA, BOW and Word2Vec, third paragraph; not true), and suggests it has found "the best way to extract information about the semantics of a word from this model" (7th paragraph; not true). This, together with the "reinterpretation of word2vec" and the proposed "new distributional semantic models" almost makes it hard for me to take the work seriously.


We present a new unsupervised method for learning general-purpose sentence embeddings. Unlike existing methods which rely on local contexts, such as words inside the sentence or immediately neighboring sentences, our method selects, for each target sentence, influential sentences in the entire document based on a document structure. We identify a dependency structure of sentences using metadata or text styles. Furthermore, we propose a novel out-of-vocabulary word handling technique to model many domain-specific terms, which were mostly discarded by existing sentence embedding methods. We validate our model on several tasks showing 30% precision improvement in coreference resolution in a technical domain, and 7.5% accuracy increase in paraphrase detection compared to baselines.

This paper extends the idea of forming an unsupervised representation of sentences used in the SkipThought approach by using a broader set of evidence for forming the representation of a sentence. Rather than simply encoding the preceding sentence and then generating the next sentence, the model suggests that a whole bunch of related "sentences" could be encoded, including document title, section title, footnotes, hyperlinked sentences. This is a valid good idea and indeed improves results. The other main new and potentially useful idea is a new idea for handling OOVs in this context where they are represented by positional placeholder variables. This also seems helpful. The paper is able to show markedly better results on paraphrase detection that skipthought and some interesting and perhaps good results on domain-specific coreference resolution.
On the negative side, the model of the paper isn't very excitingly different. It's a fairly straightforward extension of the earlier SkipThought model to a situation where you have multiple generators of related text. There isn't a clear evaluation that shows the utility of the added OOV Handler, since the results with and without that handling aren't comparable. The OOV Handler is also related to positional encoding ideas that have been used in NMT but aren't reference. And the coreference experiment isn't that clearly described nor necessarily that meaningful. Finally, the finding of dependencies between sentences for the multiple generators is done in a rule-based fashion, which is okay and works, but not super neural and exciting.
Other comments:
 - p.3. Another related sentence you could possibly use is first sentence of paragraph related to all other sentences? (Works if people write paragraphs with a "topic sentence" at the beginning.
 - p.5. Notation seemed a bit non-standard. I thought most people use \sigma for a sigmoid (makes sense, right?), whereas you use it for a softmax and use calligraphic S for a sigmoid....
 - p.5. Section 5 suggests the standard way to do OOVs is to average all word vectors. That's one well-know way, but hardly the only way. A trained UNK encoding and use of things like character-level encoders is also quite common.
 - p.6. The basic idea of the OOV encoder seems a good one. In domain specific contexts, you want to be able to refer to and re-use words that appear in related sentences, since they are likely to appear again and you want to be able to generate them. A weakness of this section however is that it makes no reference to related work whatsoever. It seems like there's quite a bit of related work. The idea of using a positional encoding so that you can generate rare words by position has previously been used in NMT, e.g. Luong et al. (Google brain) (ACL 2015). More generally, a now quite common way to handle this problem is to use "pointing" or "copying", which appears in a number of papers. (e.g., Vinyals et al. 2015) and might also have been used here and might be expected to work too.
 - p.7. Why such an old Wikipedia dump? Most people use a more recent one!
 - p.7. The paraphrase results seem good and prove the idea works. It's a shame they don't let you see the usefulness of the OOV model.
 - p.8. For various reasons, the coreference results seem less useful than they could have been, but they do show some value for the technique in the area of domain-specific coreference.


LINEARLY CONSTRAINED WEIGHTS: RESOLVING THE VANISHING GRADIENT PROBLEM BY REDUCING AN- GLE BIAS
In this paper, we first identify angle bias, a simple but remarkable phenomenon that causes the vanishing gradient problem in a multilayer perceptron (MLP) with sigmoid activation functions. We then propose linearly constrained weights (LCW) to reduce the angle bias in a neural network, so as to train the network under the constraints that the sum of the elements of each weight vector is zero. A reparameterization technique is presented to efficiently train a model with LCW by embedding the constraints on weight vectors into the structure of the network. Interestingly, batch normalization (Ioffe & Szegedy, 2015) can be viewed as a mechanism to correct angle bias. Preliminary experiments show that LCW helps train a 100-layered MLP more efficiently than does batch normalization.

This paper studies the impact of angle bias on learning deep neural networks, where angle bias is defined to be the expected value of the inner product of a random vectors (e.g., an activation vector) and a given vector (e.g., a weight vector).  The angle bias is non-zero as long as the random vector is non-zero in expectation and the given vector is non-zero.  This suggests that the some of the units in a deep neural network have large values (either positive or negative) regardless of the input, which in turn suggests vanishing gradient.  The proposed solution to angle bias is to place a linear constraint such that the sum of the weight becomes zero.  Although this does not rule out angle bias in general, it does so for the very special case where the expected value of the random vector is a vector consisting of a common value.  Nevertheless, numerical experiments suggest that the proposed approach can effectively reduce angle bias and improves the accuracy for training data in the CIFAR-10 task.  Test accuracy is not improved, however.
Overall, this paper introduces an interesting phenomenon that is worth studying to gain insights into how to train deep neural networks, but the results are rather preliminary both on theory and experiments.
On the theoretical side, the linearly constrained weights are only shown to work for a very special case.  There can be many other approaches to mitigate the impact of angle bias.  For example, how about scaling each variable in a way that the mean becomes zero, instead of scaling it into [-1,+1] as is done in the experiments?  When the mean of input is zero, there is no angle bias in the first layer.  Also, what about if we include the bias term so that b + w a is the preactivation value?
On the experimental side, it has been shown that linearly constrained weights can mitigate the impact of angle bias on vanishing gradient and can reduce the training error, but the test error is unfortunately increased for the particular task with the particular dataset in the experiments.  It would be desirable to identify specific tasks and datasets for which the proposed approach outperforms baselines.  It is intuitively expected that the proposed approach has some merit in some domains, but it is unclear exactly when and where it is.
Minor comments:
In Section 2.2, is Layer 1 the input layer or the next?


Published as a conference paper at ICLR 2018 UNSUPERVISED MACHINE TRANSLATION USING MONOLINGUAL CORPORA ONLY
Machine translation has recently achieved impressive performance thanks to recent advances in deep learning and the availability of large-scale parallel corpora. There have been numerous attempts to extend these successes to low-resource language pairs, yet requiring tens of thousands of parallel sentences. In this work, we take this research direction to the extreme and investigate whether it is possible to learn to translate even without any parallel data. We propose a model that takes sentences from monolingual corpora in two different languages and maps them into the same latent space. By learning to reconstruct in both languages from this shared feature space, the model effectively learns to translate without using any labeled data. We demonstrate our model on two widely used datasets and two language pairs, reporting BLEU scores of 32.8 and 15.1 on the Multi30k and WMT English-French datasets, without using even a single parallel sentence at training time.

The authors present an approach for unsupervised MT which uses a weighted loss function containing 3 components: (i) self reconstruction (ii) cross reconstruction and (iii) adversarial loss. The results are interesting (but perhaps less interesting than what is hinted in the abstract).
1) In the abstract the authors mention that they achieve a BLEU score of 32.8 but omit the fact that this is only on Multi30K dataset and not on the more standard WMT datasets. At first glance, most people from the field would assume that this is on the WMT dataset. I request the authors to explicitly mention this in the abstract itself (there is clearly space and I don't see why this should be omitted)
2) In section 2.3, the authors talk about the Noise Model which is inspired by the standard Denoising Autoencoder setup.  While I understand the robustness argument in the case of AEs I am not convinced that the same applies to languages. Such random permutations can often completely alter the meaning of the sentence. The ablation test seems to suggest that this process helps. I read another paper which suggests that this noise does not help (which intuitively makes sense). I would like the authors to comment on this (of course, I am not asking you to compare with  the other paper but I am just saying that I have read contradicting observations - one which seems intuitive and the other does not).
3) How were the 3 lambdas in Equation 3 selected ? What ranges did you consider. The three loss terms seem to have very different ranges. How did you account for that?
4) Clarification: In section 2.5 what exactly do you mean by "as long as the two monolingual corpora exhibit strong structure in feature space." How do you quantify this ?
5) In section 4.1, can you please mention the exact number of sentences that you sampled from WMT'14. You mention that selected sentences from 15M random pairs but how many did you select ? The caption of one of the figure mentions that there were 10M sentences. Just want to confirm this.
6) The improvements are much better on the Multi30k dataset. I guess this is because this dataset has smaller sentences with smaller vocabulary. Can you provide a table comparing the average number of sentences and vocabulary size of the two datasets (Multi30k and WMT).
7) The ablation results are provided only for the Multi30k dataset. Can you provide similar results for the WMT dataset. Perhaps this would help in answering my query in point (2) above.
8) Can you also check the performance of a PBSMT system trained on 100K parallel sentences? Although NMT outperforms PBSMT when the data size is large, PBSMT might still be better suited for low resource settings.
9) There are some missing citations (already pointed by others in the forum) . Please add those.
+++++++++++++++++++++++
I have noted the clarifications posted by the authors. I still have concerns about a couple of things.  For example, I am still not convinced about the justification given for word order. I understand that empirically it works better but I don't get the intuition. Similarly, I don't get the argument about "strong structure in feature space". This is just a conjecture and it is very hard to measure it. I would request the authors to not emphasize on it or give a different more grounded intuition.
I do acknowledge the efforts put in by the authors to address some of my comments and for that I would like to change my rating a bit.


Long Short-Term Memory (LSTM) is one of the most widely used recurrent structures in sequence modeling. Its goal is to use gates to control the information flow (e.g., whether to skip some information/transformation or not) in the recurrent computations, although its practical implementation based on soft gates only partially achieves this goal and is easy to overfit. In this paper, we propose a new way for LSTM training, which pushes the values of the gates towards 0 or 1. By doing so, we can (1) better control the information flow: the gates are mostly open or closed, instead of in a middle state; and (2) avoid overfitting to certain extent: the gates operate at their flat regions, which is shown to correspond to better generalization ability. However, learning towards discrete values of the gates is generally difficult. To tackle this challenge, we leverage the recently developed GumbelSoftmax trick from the field of variational methods, and make the model trainable with standard backpropagation. Experimental results on language modeling and machine translation show that (1) the values of the gates generated by our method are more reasonable and intuitively interpretable, and (2) our proposed method generalizes better and achieves better accuracy on test sets in all tasks. Moreover, the learnt models are not sensitive to low-precision approximation and low-rank approximation of the gate parameters due to the flat loss surface.

This paper propose a new "gate" function for LSTM to enable the values of the gates towards 0 or 1. The motivation behind is a  flat region of the loss surface is likely to generalize well. It shows the experimental results are comparable or better than vanilla LSTM and much more robust to low-precision approximation and low-rank approximation.
In section 3.2, the paper claimed using a smaller temperature cannot guarantee the outputs to be close to the boundary. Is there any experimental evidence to show it's not working? It also claimed pushing output gate to 0/1 will drop the performance. It actually quite interesting because there are bunch of paper claimed output gate is not important for language modeling, e.g. https://openreview.net/pdf?id=HJOQ7MgAW .
In the sensitive analysis, what if apply rounding / low-rank for all the parameters?
How was this approach compare to binarynet https://arxiv.org/abs/1602.02830 ? Applying the same idea, but only for forget gate/ input gate. Also, can we apply this idea to the binarynet?
Overall, I think it's an interesting paper but I feel it should compare with some simple baseline to binarized the gate function.
Updates: Thanks a lot for all the clarification. It do improve the paper quality but I'm still thinking it's higher than "6" but lower than "7". To me, improve ppl from "52.8" to "52.1" isn't very significant. For WMT, it improve on DE->EN but not for EN->DE (although it improve both for the author's own baseline). So I'm not fully convinced this approach could improve the generalization. But I feel this work can have many other applications such as "binarynet".


Deep learning yields great results across many fields, from speech recognition, image classification, to translation. But for each problem, getting a deep model to work well involves research into the architecture and a long period of tuning. We present a single model that yields good results on a number of problems spanning multiple domains. In particular, this single model is trained concurrently on ImageNet, multiple translation tasks, image captioning (COCO dataset), a speech recognition corpus, and an English parsing task. Our model architecture incorporates building blocks from multiple domains. It contains convolutional layers, an attention mechanism, and sparsely-gated layers. Each of these computational blocks is crucial for a subset of the tasks we train on. Interestingly, even if a block is not crucial for a task, we observe that adding it never hurts performance and in most cases improves it on all tasks. We also show that tasks with less data benefit largely from joint training with other tasks, while performance on large tasks degrades only slightly if at all.

The paper presents a multi-task, multi-domain model based on deep neural networks. The proposed model is able to take inputs from various domains (image, text, speech) and solves multiple tasks, such as image captioning, machine translation or speech recognition. The proposed model is composed of several features learning blocks (one for each input type) and of an encoder and an auto-regressive decoder, which are domain-agnostic. The model is evaluated on 8 different tasks and is compared with a model trained separately on each task, showing improvements on each task.
The paper is well written and easy to follow.
The contributions of the paper are novel and significant. The approach of having one model able to perform well on completely different tasks and type of input is very interesting and inspiring. The experiments clearly show the viability of the approach and give interesting insights. This is surely an important step towards more general deep learning models.
Comments:
* In the introduction where the 8 databases are presented, the tasks should also be explained clearly, as several domains are involved and the reader might not be familiar with the task linked to each database. Moreover, some databases could be used for different tasks, such as WSJ or ImageNet.
* The training procedure of the model is not explained in the paper. What is the cost function and what is the strategy to train on multiple tasks ? The paper should at least outline the strategy.
* The experiments are sufficient to demonstrate the viability of the approach, but the experimental setup is not clear. Specifically, there is an issue about the speech recognition part of the experiment. It is not clear what the task exactly is: continuous speech recognition, isolated word recognition ? The metrics used in Table 1 are also not clear, they should be explained in the text. Also, if the task is continuous speech recognition, the WER (word error rate) metric should be used. Information about the detailed setup is also lacking, specifically which test and development sets are used (the WSJ corpus has several sets).
* Using raw waveforms as audio modality is very interesting, but this approach is not standard for speech recognition, some references should be provided, such as:
P. Golik, Z. Tuske, R. Schluter, H. Ney, Convolutional Neural Networks for Acoustic Modeling of Raw Time Signal in LVCSR, in: Proceedings of the Annual Conference of the International Speech Communication Association (INTERSPEECH), 2015, pp. 26–30.
D. Palaz, M. Magimai Doss and R. Collobert, (2015, April). Convolutional neural networks-based continuous speech recognition using raw speech signal. In Acoustics, Speech and Signal Processing (ICASSP), 2015 IEEE International Conference on (pp. 4295-4299). IEEE.
T. N. Sainath, R. J. Weiss, A. Senior, K. W. Wilson, and O. Vinyals. Learning the Speech Front-end With Raw Waveform CLDNNs. Proceedings of the Annual Conference of the International Speech Communication Association (INTERSPEECH), 2015.
Revised Review:
The main idea of the paper is very interesting and the work presented is impressive. However, I tend to agree with Reviewer2, as a more comprehensive analysis should be presented to show that the network is not simply multiplexing tasks. The experiments are interesting, except for the WSJ speech task, which is almost meaningless. Indeed, it is not clear what the network has learned given the metrics presented, as the WER on WSJ should be around 5% for speech recognition.
I thus suggest to either drop the speech experiment, or the modify the network to do continuous speech recognition. A simpler speech task such as Keyword Spotting could also be investigated.


In this paper, we propose a generic and simple strategy for utilizing stochastic gradient information in optimization. The technique essentially contains two consecutive steps in each iteration: 1) computing and normalizing each block (layer) of the mini-batch stochastic gradient; 2) selecting appropriate step size to update the decision variable (parameter) towards the negative of the block-normalized gradient. We conduct extensive empirical studies on various non-convex neural network optimization problems, including multi layer perceptron, convolution neural networks and recurrent neural networks. The results indicate the blocknormalized gradient can help accelerate the training of neural networks. In particular, we observe that the normalized gradient methods having constant step size with occasionally decay, such as SGD with momentum, have better performance in the deep convolution neural networks, while those with adaptive step sizes, such as Adam, perform better in recurrent neural networks. Besides, we also observe this line of methods can lead to solutions with better generalization properties, which is confirmed by the performance improvement over strong baselines.

This paper proposes a family of first-order stochastic optimization schemes based on (1)  normalizing (batches of) stochastic gradient descents and (2) choosing from a step size updating scheme. The authors argue that iterative first-order optimization algorithms can be interpreted as a choice of an update direction and a step size, so they suggest that one should always normalize the gradient when computing the direction and then choose a step size using the normalized gradient.
The presentation in the paper is clear, and the exposition is easy to follow. The authors also do a good job of presenting related work and putting their ideas in the proper context. The authors also test their proposed method on many datasets, which is appreciated.
However, I didn't find the main idea of the paper to be particularly compelling. The proposed technique is reasonable on its own, but the empirical results do not come with any measure of statistical significance. The authors also do not analyze the sensitivity of the different optimization algorithms to hyperparameter choice, opting to only use the default. Moreover, some algorithms were used as benchmarks on some datasets but not others. For a primarily empirical paper, every state-of-the-art algorithm should be used as a point of comparison on every dataset considered. These factors altogether render the experiments uninformative in comparing the proposed suite of algorithms to state-of-the-art methods. The theoretical result in the convex setting is also not data-dependent, despite the fact that it is the normalized gradient version of AdaGrad, which does come with a data-dependent convergence guarantee.
Given the suite of optimization algorithms in the literature and in use today, any new optimization framework should either demonstrate improved (or at least matching) guarantees in some common (e.g. convex) settings or definitively outperform state-of-the-art methods on problems that are of widespread interest. Unfortunately, this paper does neither.
Because of these points, I do not feel the quality, originality, and significance of the work to be high enough to merit acceptance.
Some specific comments:
p. 2: "adaptive feature-dependent step size has attracted lots of attention". When you apply feature-dependent step sizes, you are effectively changing the direction of the gradient, so your meta learning formulation, as posed, doesn't make as much sense.
p. 2: "we hope the resulting methods can benefit from both techniques". What reason do you have to hope for this? Why should they be complimentary? Existing optimization techniques are based on careful design and coupling of gradients or surrogate gradients, with specific learning rate schedules. Arbitrarily mixing the two doesn't seem to be theoretically well-motivated.
p. 2: "numerical results shows that normalized gradient always helps to improve the performance of the original methods when the network structure is deep". It would be great to provide some intuition for this.
p. 2: "we also provide a convergence proof under this framework when the problem is convex and the stepsize is adaptive". The result that you prove guarantees a \theta(\sqrt{T}) convergence rate. On the other hand, the AdaGrad algorithm guarantees a data-dependent bound that is O(\sqrt{T}) but can also be much smaller. This suggests that there is no theoretical motivation to use NGD with an adaptive step size over AdaGrad.
p. 2-3: "NGD can find a \eps-optimal solution....when the objective function is quasi-convex. ....extended NGD for upper semi-continuous quasi-convex objective functions...". This seems like a typo. How are results that go from quasi-convex to upper semi-continuous quasi-convex an extension?
p. 3: There should be a reference for RMSProp.
p. 3: "where each block of parameters x^i can be viewed as parameters associated to the ith layer in the network". Why is layer parametrization (and later on normalization) a good way idea? There should be either a reference or an explanation.
p. 4: "x=(x_1, x_2, \ldots, x_B)". Should these subscripts be superscripts?
p. 4: "For all the algorithms, we use their default settings." This seems insufficient for an empirical paper, since most problems often involve some amount of hyperparameter tuning. How sensitive is each method to the choice of hyperparameters? What about the impact of initialization?
p. 4-8: None of the experimental results have error bars or any measure of statistical significance.
p. 5: "NG... is a variant of the NG_{UNIT} method". This method is never motivated.
p. 5-6: Why are SGD and Adam used for MNIST but not on CIFAR?
p. 5: "we chose the best heyper-paerameter from the 56 layer residual network." Apart from the typos, are these parameters chosen from the training set or the test set?
p. 6: Why isn't Adam tested on ImageNet?

POST AUTHOR RESPONSE: After reading the author response and taking into account the fact that the authors have spent the time to add more experiments and clarify their theoretical result, I have decided to upgrade my score from a 3 to a 4. However, I still do not feel that the paper is up to the standards of the conference.


Published as a conference paper at ICLR 2018 BREAKING THE SOFTMAX BOTTLENECK: A HIGH-RANK RNN LANGUAGE MODEL
We formulate language modeling as a matrix factorization problem, and show that the expressiveness of Softmax-based models (including the majority of neural language models) is limited by a Softmax bottleneck. Given that natural language is highly context-dependent, this further implies that in practice Softmax with distributed word embeddings does not have enough capacity to model natural language. We propose a simple and effective method to address this issue, and improve the state-of-the-art perplexities on Penn Treebank and WikiText-2 to 47.69 and 40.68 respectively. The proposed method also excels on the large-scale 1B Word dataset, outperforming the baseline by over 5.6 points in perplexity.

The authors argue in this paper that due to the limited rank of the  context-to-vocabulary logit matrix in the currently used version of the softmax output layer, it is not able to capture the full complexity of language. As a result, they propose to use a mixture of softmax output layers instead where the mixing probabilities are context-dependent, which allows to obtain a full rank logit matrix in complexity linear in the number of mixture components (here 15). This leads to improvements in the word-level perplexities of the PTB and wikitext2 data sets, and Switchboard BLEU scores.
The question of the expressiveness of the softmax layer, as well as its suitability for word-level prediction, is indeed an important one which has received too little attention. This makes a lot of the questions asked in this paper extremely relevant to the field. However, it is unclear that the rank of the logit matrix is the right quantity to consider. For example, it is easy to describe a rank D NxM matrix where up to 2^D lines have max values at different indices. Further, the first two "observations" in Section 2.2 would be more accurately described as "intuitions" of the authors. As they write themselves "there is no evidence showing that semantic meanings are fully linearly correlated." Why then try to link "meanings" to basis vectors for the rows of A?
To be clear, the proposed model is undoubtedly more expressive than a regular softmax, and although it does come at a substantial computational cost (a back-of-the envelope calculation tells us that computing 15 components of 280d MoS takes the same number of operations as one with dimension 1084 = sqrt (280*280*15)), it apparently manages not to drastically increase overfitting, which is significant.
Unfortunately, this is only tested on relatively small data sets, up to 2M tokens and a vocabulary of size 30K for language modeling. They do constitute a good starting place to test a model, but given the importance of regularization on those specific tasks, it is difficult to predict how the MoS would behave if more training data were available, and if one could e.g. simply try a 1084 dimension embedding for the softmax without having to worry about overfitting.
Another important missing experiment would consist in varying the number of mixture components (this could very well be done on WikiText2). This could help validate the hypothesis: how does the estimated rank vary with the number of components? How about the performance and pairwise KL divergence?
This paper offers a promising direction for language modeling research, but would require more justification, or at least a more developed experimental section.
Pros:
- Important starting question
- Thought-provoking approach
- Experimental gains on small data sets
Cons:
- The link between the intuition and reality of the gains is not obvious
- Experiments limited to small data sets, some obvious questions remain


We present Value Propagation (VProp), a parameter-efficient differentiable planning module built on Value Iteration which can successfully be trained in a reinforcement learning fashion to solve unseen tasks, has the capability to generalize to larger map sizes, and can learn to navigate in dynamic environments. We evaluate on configurations of MazeBase grid-worlds, with randomly generated environments of several different sizes. Furthermore, we show that the module enables to learn to plan when the environment also includes stochastic elements, providing a cost-efficient learning system to build low-level size-invariant planners for a variety of interactive navigation problems.

ORIGINALITY & SIGNIFICANCE
The authors build upon value iteration networks: the idea that the value function can be computed efficiently from rewards and transitions using a dedicated convolutional network. The authors point out that the original "value iteration network” (Tamar 2016) did not handle non-stationary dynamics models or variable size problems well and propose a new formulation to extend the model to this case which they call a value propagation network.  It seems useful and practical to compute value iteration explicitly as this will propagate values for us without having to learn the propagated form through extensive gradient update steps. Extending to the scenario of non-stationary dynamics is important to make the idea applicable to common problems. The work is therefore original and significant.
The algorithm is evaluated on the original obstacle grids from Tamar 2016 and larger grids generated to test scalability. The authors Prop and MVProp are able to solve the grids with much higher reliability at the end of training and converge much faster.  The M in MVProp in particular seems to be very useful in scaling up to the large grids. The authors also show that the algorithm handles non-stationary dynamics in an avalanche task where obstacles can fall over time.
QUALITY
The symbol d_{rew} is never defined — what does “new” stand for? It appears to be the number of latent convolutional filters or channels generated by the state embedding network.
Section 2.2 Sentence 2: The final layer representing the encoding is given as ( R^{d_rew  x d_x x d_y }.
Based on the description  in the first paragraph of section 2, it sounds like d_rew might be the number of channels or filters in the last convolutional layer.
In equation 1, it wasn’t obvious to me that the expression max_a q_{ij}^{k-1} q^{k} corresponds to an actual operation?
The h( \Phi( x ), v^{k-1} ) sort of makes sense …  value is only calculated with respect to only the observation of the maze obstacles but the policy \pi is calculated with respect to the joint  observation and agent state.
The expression
   h_{aid} ( \phi(0), v )   =   <  Wa,   [ \phi(o) ; v ]   >   +   b
makes sense and reminds me of the Value Iteration network work where we take the previous value function, combine it with the reward function and use convolution to compute the expectation (the weights Wa encode the effect of transitions). I gather the tensor Wa = R^{|A| x (d_{rew} x d_x x d_y } both converts the feature embedding \phi{o} to rewards and represents the transition / propagation of reward across states due to transitions and discounts at the same time?
I didn’t understand the r^in, r&out representation in section 4.1. These are given by the domain?
I did get the overall idea of efficiently creating a local value function in the neighborhood of the current state and passing this to the policy so that it can make a local decision.
A bit more detail defining terms, explaining their intuitive role and how the output of one module feeds into the next would be helpful.
POST REVISION COMMENTS:
- I didn't reread the whole thing -  just used the diff tool.
- It looks like the typos in the equations got fixed
- The new phrase "enables to learn to plan" seems pretty awkward


MULTI-TASK LEARNING FOR DOCUMENT RANKING AND QUERY SUGGESTION
We propose a multi-task learning framework to jointly learn document ranking and query suggestion for web search. It consists of two major components, a document ranker and a query recommender. Document ranker combines current query and session information and compares the combined representation with document representation to rank the documents. Query recommender tracks users' query reformulation sequence considering all previous in-session queries using a sequence to sequence approach. As both tasks are driven by the users' underlying search intent, we perform joint learning of these two components through session recurrence, which encodes search context and intent. Extensive comparisons against state-of-the-art document ranking and query suggestion algorithms are performed on the public AOL search log, and the promising results endorse the effectiveness of the joint learning framework.

The work is interesting and novel. The novelty lies not in the methods used (existing methods are used), but in the way these methods are combined to solve two problems (that so far have been treated separately in IR) simultaneously. The fitness of the proposed architecture and methodological choices to the task at hand is sufficiently argued.
The experimental evaluation is not the strongest, in terms of datasets and evaluation measures. While I understand why the AOL dataset was used, the document ranking experiments should also include runs on any of the conventional TREC datasets of documents, queries and actual (not simulated) relevance assessments. Simulating document relevance from clicks is a good enough approximation, but why not also use datasets with real human relevance assessments, especially since so many of them exist and are so easy to access?
When evaluating ranking, MAP and NDCG are indeed two popular measures. But the choice of NDCG@1,3,10 seems a bit adhoc. Why not NDCG@5? Furthermore, as the aim seems to be to assess early precision, why not also report MRR?
The paper reports that the M-NSRF query suggestion method outperforms all baselines. This is not true. Table 2 shoes that M-NSRF is best for BLEU-1/2, but not for BLEU-3/4.
Three final points:
- Out of the contributions enumerated at the end of Section 1, only the novel model and the code & data release are contributions. The rigorous comparison to soa and its detailed analysis are the necessary evaluation parts of any empirical paper.
- The conclusion states that this work provides useful intuitions about the advantages of multi-task learning involving deep neural networks for IR tasks. What are these? Where were they discussed? They should be outlined here, or referred to somehow.
- Although the writing is coherent, there are a couple of recurrent English language mistakes (e.g. missing articles). The paper should be proofread and corrected.


Published as a conference paper at ICLR 2018 SKIP CONNECTIONS ELIMINATE SINGULARITIES
Skip connections made the training of very deep networks possible and have become an indispensable component in a variety of neural architectures. A completely satisfactory explanation for their success remains elusive. Here, we present a novel explanation for the benefits of skip connections in training very deep networks. The difficulty of training deep networks is partly due to the singularities caused by the non-identifiability of the model. Several such singularities have been identified in previous works: (i) overlap singularities caused by the permutation symmetry of nodes in a given layer, (ii) elimination singularities corresponding to the elimination, i.e. consistent deactivation, of nodes, (iii) singularities generated by the linear dependence of the nodes. These singularities cause degenerate manifolds in the loss landscape that slow down learning. We argue that skip connections eliminate these singularities by breaking the permutation symmetry of nodes, by reducing the possibility of node elimination and by making the nodes less linearly dependent. Moreover, for typical initializations, skip connections move the network away from the "ghosts" of these singularities and sculpt the landscape around them to alleviate the learning slow-down. These hypotheses are supported by evidence from simplified models, as well as from experiments with deep networks trained on real-world datasets.

This paper proposes to explain the benefits of skip connections in terms of eliminating the singularities of the loss function. The discussion is largely based on a sequence of experiments, some of which are interesting and insightful. The discussion here can be useful for other researchers.
My main concern is that the result here is purely empirical, with no concrete theoretical justification. What the experiments reveal is an empirical correlation between the Eigval index and training accuracy, which can be caused by lots of reasons (and cofounders), and does not necessarily establish a causal relation. Therefore, i found many of the discussion to be questionable. I would love to see more solid theoretical discussion to justify the hypothesis proposed in this paper.

Do you have a sense how accurate is the estimation of the tail probabilities of the eigenvalues? Because the whole paper is based on the approximation of the eigval indexes, it is critical to exam the estimation is accurate enough to draw the conclusions in the paper.
All the conclusions are based on one or two datasets. Could you consider testing the result on more different datasets to verify if the results are generalizable?


Recurrent neural networks (RNNs) have achieved state-of-the-art performance on many diverse tasks, from machine translation to surgical activity recognition, yet training RNNs to capture long-term dependencies remains difficult. To date, the vast majority of successful RNN architectures alleviate this problem using nearlyadditive connections between states, as introduced by long short-term memory (LSTM). We take an orthogonal approach and introduce MIST RNNs, a NARX RNN architecture that allows direct connections from the very distant past. We show that MIST RNNs 1) exhibit superior vanishing-gradient properties in comparison to LSTM and previously-proposed NARX RNNs; 2) are far more efficient than previously-proposed NARX RNN architectures, requiring even fewer computations than LSTM; and 3) improve performance substantially over LSTM and Clockwork RNNs on tasks requiring very long-term dependencies.

The presented MIST architecture certainly has got its merits, but in my opinion is not very novel, given the fact that NARX RNNs have been described 20 years ago, and Clockwork RNNs (which, as the authors point out in section 2, have a similar structure) have also been in use for several years. Still, the presented results are good, with standard LSTMs being substantially outperformed in three out of five standard RNN/LSTM benchmark tasks. The analysis in section 3 is decent (see however the minor comments below), but does not offer revolutionary new insights - it's perhaps more like a corollary of previous work (Pascanu et al., 2013).
Regarding the concrete results, I would have wished for a more detailed analysis of the more surprising results, in particular, for the copy task (section 5.2): Is it really true that Clockwork RNNs fail because they make it "difficult to learn long-term behavior that must be detected at high frequency" [section 2]? How relevant are the results in figure 2 (yes, the gradient properties are very different, but is this an issue for accuracy)? In the sequential pMNIST classification, what about increasing the LSTM number of hidden units? If this brings the error rate further down, one could ask why exactly the LSTM captures long-term structure so differently with different number of units?
In summary, for me this paper is solid, and although the architecture is not that new, it is worth bringing it again into the focus of attention.
Minor comments:
- In several places, the formulas are rather strange and/or occasionally incorrect. In particular,
* on the right-hand sind of the inline formula in section 3.1, the symbol v is missing completely, which cannot be right;
* in formula 16, the primes seem to be misplaced, and the symbols t', t''', etc. should be defined;
* the \theta_l in the beginning of section 3.3 (formula 13) is completely superfluous.
- The position of the tables and figures is rather weird, making the paper less readable than necessary. The authors should consider moving floating parts around (one could also move figure three to the bottom of a suitable page, for example).
- It is a matter of taste, but since all experimental results except the ones on the copy task are tabulated, one could think of adding a table with the results now contained in figure 3.
Relation to prior work: the authors are aware of most relevant work.
On p2 they write: "Many other approaches have also been proposed to capture long-term dependencies." There is one that seems close to what the authors do:
J. Schmidhuber. Learning complex, extended sequences using the principle of history compression. Neural Computation, 4(2):234-242, 1992
It is related to clockwork RNNs, about which the authors write:
"A recent architecture that is similar in spirit to our work is that of Clockwork RNNs (Koutnik et al., 2014), which split weights and hidden units into partitions, each with a distinct period. When it’s not a partition’s time to tick, its hidden units are passed through unchanged, thus in some ways mimicking the behavior of NARX RNNs. However Clockwork RNNs differ in two key ways. First, Clockwork RNNs sever high-frequency-to-low-frequency paths, thus making it difficult to learn long-term behavior that must be detected at high frequency (for example, learning to depend on quick motions from the past for activity recognition). Second, Clockwork RNNs require hidden units to be partitioned a priori, which in practice is difficult to do in any meaningful way. NARX RNNs suffer from neither of these drawbacks."
The neural history compressor, however, adapts to the frequency of unexpected events, by ticking only when there is an unpredictable event, thus overcoming some of the issues above. Perhaps this trick could further improve the system of the authors, as well as the Clockwork RNNs, at least for certain tasks?
General recommendation: Accept, provided the comments are taken into account.


Published as a conference paper at ICLR 2018 PIXELNN: EXAMPLE-BASED IMAGE SYNTHESIS
We present a simple nearest-neighbor (NN) approach that synthesizes highfrequency photorealistic images from an "incomplete" signal such as a lowresolution image, a surface normal map, or edges. Current state-of-the-art deep generative models designed for such conditional image synthesis lack two important things: (1) they are unable to generate a large set of diverse outputs, due to the mode collapse problem. (2) they are not interpretable, making it difficult to control the synthesized output. We demonstrate that NN approaches potentially address such limitations, but suffer in accuracy on small datasets. We design a simple pipeline that combines the best of both worlds: the first stage uses a convolutional neural network (CNN) to map the input to a (overly-smoothed) image, and the second stage uses a pixel-wise nearest neighbor method to map the smoothed output to multiple high-quality, high-frequency outputs in a controllable manner. Importantly, pixel-wise matching allows our method to compose novel high-frequency content by cutting-and-pasting pixels from different training exemplars. We demonstrate our approach for various input modalities, and for various domains ranging from human faces, pets, shoes, and handbags.

This paper presents a pixel-matching based approach to synthesizing RGB images from input edge or normal maps. The approach is compared to Isola et al’s conditional adversarial networks, and unlike the conditional GAN, is able to produce a diverse set of outputs.
Overall, the paper describes a computer visions system based on synthesizing images, and not necessarily a new theoretical framework to compete with GANs. With the current focus of the paper being the proposed system, it is interesting to the computer vision community. However, if one views the paper in a different light, namely showing some “blind-spots” of current conditional GAN approaches like lack of diversity, then it can be of much more interest to the broader ICLR community.
Pros:
Overall the paper is well-written
Makes a strong case that random noise injection inside conditional GANs does not produce enough diversity
Shows a number of qualitative and quantitative results
Concerns about the paper:
1.) It is not clear how well the proposed approach works with CNN architectures other than PixelNet
2.) Since the paper used “the pre-trained PixelNet to extract surface normal and edge maps” for ground-truth generation, it is not clear whether the approach will work as well when the input is a ground-truth semantic segmentation map.
3.) Since the paper describes a computer-vision image synthesis system and not a new theoretical result, I believe reporting the actual run-time of the system will make the paper stronger. Can PixelNN run in real-time? How does the timing compare to Isola et al’s Conditional GAN?
Minor comments:
1.) The paper mentions making predictions from “incomplete” input several times, but in all experiments, the input is an edge map, normal map, or low-resolution image. When reading the manuscript the first time, I was expecting experiments on images that have regions that are visible and regions that are masked out. However, I am not sure if the confusion is solely mine, or shared with other readers.
2.) Equation 1 contains the norm operator twice, and the first norm has no subscript, while the second one has an l_2 subscript. I would expect the notation style to be consistent within a single equation (i.e., use ||w||_2^2, ||w||^2, or ||w||_{l_2}^2)
3.) Table 1 has two sub-tables: left and right. The sub-tables have the AP column in different places.
4.) “Dense pixel-level correspondences” are discussed but not evaluated.


In this paper, we propose a nonlinear unsupervised metric learning framework to boost of the performance of clustering algorithms. Under our framework, nonlinear distance metric learning and manifold embedding are integrated and conducted simultaneously to increase the natural separations among data samples. The metric learning component is implemented through feature space transformations, regulated by a nonlinear deformable model called Coherent Point Drifting (CPD). Driven by CPD, data points can get to a higher level of linear separability, which is subsequently picked up by the manifold embedding component to generate well-separable sample projections for clustering. Experimental results on synthetic and benchmark datasets show the effectiveness of our proposed approach over the state-of-the-art solutions in unsupervised metric learning.

This paper presents a scheme for unsupervised metric learning using coherent point drifting (CPD)-- the core idea is to learn a parametric model of CPD that shifts the input points such that the shifted points lead to better clustering in a K-Means setup. Following the work of Myronenko & Song, 2010, this paper uses a linear parametric model for the drift (in CPD) after mapping the input points to a kernel feature space using an RBF kernel. The CPD model is directly used within the KMeans objective -- the drift parameter matrix and the KMeans cluster assignment matrix are jointly learned using block-coordinate descent (BCD). The paper uses some interesting properties of the CPD model to derive an efficient optimization solver for the BCD subproblems. Experiments are provided on UCI datasets and demonstrate some promise.
Pros:
1) The idea of using CPD for unsupervised metric learning is quite interesting
2) The exploration into the convexity of the CPD parameter learning -- although straightforward -- is also perhaps interesting.
3) The experiments show some promise.
Cons:
1) Lacking motivation/Intuition
The main motivation for the approach, as far as I understand, is to learn cluster boundaries for non-linear data -- where K-Means fails. However, it is unclear to me why would one need to use K-Means for non-linear data, why not use kernelized kmeans? The proposed CPD model also is essentially learning a linear transformation of the kernelized feature space. So in contrast to kernelized kmeans, what is the advantage of the proposed framework? I see there is an improvement in performance compared to kernelized kmeans, however, intuitively I do not see how that improvement comes from? Perhaps providing some specific examples/scenarios or graphic illustrations will help appreciate the method.
2) Novelty/Significance
I think the novelty of this paper is perhaps marginal. The main idea is to directly use CPD from a prior work in a KMeans setup. There are a few parameters to be estimated in the joint learning objective, for which a block-coordinate descent strategy is proposed. The derivations are perhaps straightforward. As noted above, it is not clear what is the significance of this combination or how does it improve performance. As far as CPD goes, it looks to me that the performance depends heavily on the choice of the Gaussian RBF bandwidth parameter, and it is not clear to me how such a parameter can be selected in a unsupervised setting, when class labels are not available for cross-validation. The paper does not provide any intuitions on this front.
3) Technical details.
There are a few important details that I do not quite follow in the paper.
a) The CPD is originally designed for the point matching problem, and its parametric form (\Psi) is derived using a different a Tikhonov regularized regression model as described just above (1). The current paper directly uses this parametric form in a KMeans setup and solve the resultant problem jointly for the CPD parameter and the clustering assignment. However, it is not clear to me how the paper could use the optimal parametric form for Tikhonov regression as the optimum for the clustering problem. Ideally, I would think when formulating the joint optimization for the clustering problem, the optimal functional v(x) should also be learned/derived for the clustering problem, or some proof should be provided showing the functionals are the same. Without this, I am not convinced that the proposed formulation indeed learns the optimum drifts and the clusters jointly.
b)  The subproblem on Y (the assignment matrix) looks like a standard SVD objective. It is not clear why would it be necessary to resort to Ky Fan's theorem for its optimal solution.
c) The paper talks about manifold embedding in the abstract and in Sec. 2.2. However, it appears to be a straightforward dimensionality reduction (PCA) of data. If not, what is the precise manifold that is described here?
d) Eq. 9, the definition of Y_c is incorrect and unclear. p is defined as a vector of ones, earlier.
e) Although the assignment matrix Y has orthogonal columns, it is a binary matrix. If it is approximated by an orthonormal frame, how do you reduce it to a binary matrix? Does taking the largest values in each column suffice -- it does not look like so. However, in the paper, Y is relaxed to an orthonormal frame, which is estimated using PCA, the data points are then projected onto this low-dimensional subspace, and then k-means applied to get the Y matrix. The provided math does not support any of these steps. Thus, the technical exposition is imprecise and the solutions appear rather heuristic.
f) The kernelized variant of the proposed scheme, described in Sec. 2.4 is missing important details. How precisely is the kernelization done? How is CPD extended to that setup and what would be the Gaussian kernel G in that case, and what does \Psi signify?
g) Figure 2, it seems that kernel kmeans and the proposed CPD-UML show similar cluster boundaries for low-kernel widths. Why are the high kernel widths beneficial?
4) Experiments
There is some improvement of the proposed method -- however overall, the improvements are marginal. The discussion is missing any analysis of the results. Why it works at times, how well it improves on kernelized kmeans, and why? What is the advantage over other competitive schemes, etc.
In summary, while there is a minor novelty in connecting two separate ideas (CPD and UML) into a joint UML setup, the paper lacks sufficient motivations for proposing this setup (in contrast to say kernelized kmeans), the technical details are unconvincing, and the experiments lack sufficient details or analysis. Thus, I do not think this paper is ready to be accepted in its current form.


Workshop track -ICLR 2018 DEEP LEARNING MUTATION PREDICTION ENABLES EARLY STAGE LUNG CANCER DETECTION IN LIQUID BIOPSY
Somatic cancer mutation detection at ultra-low variant allele frequencies (VAFs) is an unmet challenge that is intractable with current state-of-the-art mutation calling methods. Specifically, the limit of VAF detection is closely related to the depth of coverage due to the requirement of multiple supporting reads in extant methods, precluding the detection of mutations at VAFs that are orders of magnitude lower than the depth of coverage. Nevertheless, the ability to detect cancer-associated mutations in ultra low VAFs is a fundamental requirement for low-tumor burden cancer diagnostics applications such as early detection, monitoring, and therapy nomination using liquid biopsy methods (cell-free DNA). Here we defined a spatial representation of sequencing information adapted for convolutional architecture that enables variant detection in a manner independent of the depth of sequencing. This method enables the detection of cancer mutations even in VAFs as low as 10 4 , more than two orders of magnitude below the current state-of-theart. We validated our method on both simulated plasma and on clinical cfDNA plasma samples from cancer patients and non-cancer controls. This method introduces a new domain within bioinformatics and personalized medicine -somatic whole genome mutation calling for liquid biopsy.

Summary:
In this paper the authors offer a new algorithm to detect cancer mutations from sequencing cell free DNA (cfDNA). The idea is that in the sample being sequenced there would also be circulating tumor DNA (ctDNA) so such mutations could be captured in the sequencing reads. The issue is that the ctDNA are expected to be found with low abundance in such samples, and therefore are likely to be hit by few or even single reads. This makes the task of differentiating between sequencing errors and true variants due to ctDNA hard. The authors suggest to overcome this problem by training an algorithm that will identify the sequence context that characterize sequencing errors from true mutations. To this, they add channels based on low base quality, low mapping quality. The algorithm for learning the context of sequencing reads compared to true mutations is based on a multi layered CNN, with 2/3bp long filters to capture di and trinucleotide frequencies, and a fully connected layer to a softmax function at the top. The data is based on mutations in 4 patients with lung cancer for which they have a sample both directly from the tumor and from a healthy region. One more sample is used for testing and an additional cancer control which is not lung cancer is also used to evaluate performance.
Pros:
The paper tackles what seems to be both an important and challenging problem. We also liked the thoughtful construction of the network and way the reference, the read, the CIGAR and the base quality were all combined as multi channels to make the network learn the discriminative features of from the context. Using matched samples of tumor and normal from the patients is also a nice idea to mimic cfDNA data.
Cons:
While we liked both the challenge posed and the idea to solve it we found several major issues with the work.
First, the writing is far from clear. There are typos and errors all over at an unacceptable level. Many terms are not defined or defined after being introduced (e.g. CIGAR, MF, BQMQ). A more reasonable CS style of organization is to first introduce the methods/model and then the results, but somehow the authors flipped it and started with results first, lacking many definitions and experimental setup to make sense of those.  Yet Sec. 2 “Results” p. 3 is not really results but part of the methods. The “pipeline” is never well defined, only implicitly in p.7 top, and then it is hard to relate the various figures/tables to bottom line results (having the labels wrong does not help that).
The filters by themselves seem trivial and as such do not offer much novelty. Moreover, the authors filter the “normal” samples using those (p.7 top), which makes the entire exercise a possible circular argument.
If the entire point is to classify mutations versus errors it would make sense to combine their read based calls from multiple reads per mutations (if more than a single read for that mutation is available) - but the authors do not discuss/try that.
The entire dataset is based on 4 patients. It is not clear what is the source of the other cancer control case. The authors claim the reduced performance show they are learning lung cancer-specific context. What evidence do they have for that? Can they show a context they learned and make sense of it? How does this relate to the original papers they cite to motivate this direction (Alexandrov 2013)? Since we know nothing about all these samples it may very well be that that are learning technical artifacts related to their specific batch of 4 patients. As such, this may have very little relevance for the actual problem of cfDNA.
Finally, performance itself did not seem to improve significantly compared to previous methods/simple filters, and the novelty in terms of ML and insights about learning representations seemed limited.
Albeit the above caveats, we iterate the paper offers a nice construction for an important problem. We believe the method and paper could potentially be improved and make a good fit for a future bioinformatics focused meeting such as ISMB/RECOMB.


Convolution neural networks typically consist of many convolutional layers followed by several fully-connected layers. While convolutional layers map between high-order activation tensors, the fully-connected layers operate on flattened activation vectors. Despite its success, this approach has notable drawbacks. Flattening discards the multi-dimensional structure of the activations, and the fullyconnected layers require a large number of parameters. We present two new techniques to address these problems. First, we introduce tensor contraction layers which can replace the ordinary fully-connected layers in a neural network. Second, we introduce tensor regression layers, which express the output of a neural network as a low-rank multi-linear mapping from a high-order activation tensor to the softmax layer. Both the contraction and regression weights are learned end-to-end by backpropagation. By imposing low rank on both, we use significantly fewer parameters. Experiments on the ImageNet dataset show that applied to the popular VGG and ResNet architectures, our methods significantly reduce the number of parameters in the fully connected layers (about 65% space savings) while negligibly impacting accuracy.

This paper incorporates tensor decomposition and tensor regression into CNN by replacing its flattening operations and fully-connected layers with a new tensor regression layer.
Pros:
The low-rank representation of tensors is able to reduce the model complexity in the original CNN without sacrificing much prediction accuracy. This is promising as it enables the implementation of complex deep learning algorithms on mobile devices due to its huge space saving performance.  Overall, this paper is easy to follow.
Cons:
Q1: Can the authors discuss the computational time of the proposed tensor regression layers and compare it to that of the baseline CNN? The tensor regression layer is computationally more expensive than the flattening operations in original CNN. Usually, it also involves expensive model selection procedure to choose the tuning parameters (N+1 ranks and a L2 norm sparsity parameter). In the experiments, the authors simply tried a few ranks without serious tuning.
Q2: The authors reported the space saving in Table 1 but not in Table 2. Since spacing saving is a major contribution of the proposed method, can authors add the space saving percentage in Table 2?
Q3: There are a few typos in the current paper. I would suggest the authors to take a careful proofreading. For example,
(1) In the “Related work“ paragraph on page 2, “Lebedev et al. (2014) proposes…” should be “Lebedev et al. (2014) propose…”. Many other references have the same issue.
(2) In Figure 1, the letter $X$ should be $\tilde{\cal X}$.
(3) In expression (5) on page 3, the core tensor is denoted by $\tilde{\cal G}$. Is this the same as $\tilde{\cal X}^{‘}$ in Figure 1?
(4) In expression (5) on page 3, the core tensor $\tilde{\cal G}$ is of dimension $(D_0, R_1, \ldots, R_N)$. However, in expression (8) on page 5, $\tilde{\cal G}$ is of dimension $(R_0, R_1, \ldots, R_N, R_{N+1})$.
(5) Use \cite{} and \citep{} correctly. For example, in the “Related work“ paragraph on page 2,
“Several prior papers address the power of tensor regression to preserve natural multi-modal structure and learn compact predictive models Guo et al. (2012); Rabusseau & Kadri (2016); Zhou et al. (2013); Yu & Liu (2016).”
should be
“Several prior papers address the power of tensor regression to preserve natural multi-modal structure and learn compact predictive models (Guo et al., 2012; Rabusseau & Kadri, 2016; Zhou et al., 2013; Yu & Liu, 2016).”


We propose Significance-Offset Convolutional Neural Network, a deep convolutional network architecture for regression of multivariate asynchronous time series. The model is inspired by standard autoregressive (AR) models and gating mechanisms used in recurrent neural networks. It involves an AR-like weighting system, where the final predictor is obtained as a weighted sum of adjusted regressors, while the weights are data-dependent functions learnt through a convolutional network. The architecture was designed for applications on asynchronous time series and is evaluated on such datasets: a hedge fund proprietary dataset of over 2 million quotes for a credit derivative index, an artificially generated noisy autoregressive series and household electricity consumption dataset. The proposed architecture achieves promising results as compared to convolutional and recurrent neural networks. The code for the numerical experiments and the architecture implementation will be shared online to make the research reproducible.

The author proposed:
1. A data augmentation technique for asynchronous time series data.
2. A convolutional 'Significance' weighting neural network that assigns normalised weights to the outputs of a fully-connected autoregressive 'Offset' neural network, such that the output is a weighted average of the 'Offset' neural net.
3. An 'auxiliary' loss function.
The experiments showed that:
1. The proposed method beat VAR/CNN/ResNet/LSTM 2 synthetic asynchronous data sets, 1 real electricity meter data set and 1 real financial bid/ask data set. It's not immediately clear how hyper-parameters for the benchmark models were chosen.
2. The author observed from the experiments that the depth of the offset network has negligible effect, and concluded that the 'Significance' network has crucial impact. (I don't see how this conclusion can be made.)
3. The proposed auxiliary loss is not useful.
4. The proposed architecture is more robust to noise in the synthetic data set compared to the benchmarks, and together with LSTM, are least prone to overfitting.
Pros
- Proposed a useful way of augmenting asynchronous multivariate time series for fitting autoregressive models
- The convolutional Significance/weighting networks appears to reduce test errors (not entirely clear)
Cons
- The novelties aren't very well-justified. The 'Significance' network was described as critical to the performance, but there is no experimental result to show the sensitivity of the model's performance with respect to the architecture of the 'Significance' network. At the very least, I'd like to see what happens if the weighting was forced to be uniform while keeping the 'Offset' network and loss unchanged.
- It's entirely unclear how the train and test data was split. This may be quite important in the case of the financial data set.
- It's also unclear if model training was done on a rolling basis, which is common for time series forecasting.
- The auxiliary loss function does not appear to be very helpful, but was described as a key component in the paper.
Quality: The quality of the paper was okay. More details of the experiments should be included in the main text to help interpret the significance of the experimental results. The experiment also did not really probe the significance of the 'Significance' network even though it's claimed to be important.
Clarity: Above average.
Originality: Mediocre. Nothing really shines. Weighted average-type architecture has been proposed many times in neural networks (e.g., attention mechanisms).
Significance: Low. It's unclear how useful the architecture really is.


This paper concerns open-world classification, where the classifier not only needs to classify test examples into seen classes that have appeared in training but also reject examples from unseen or novel classes that have not appeared in training. Specifically, this paper focuses on discovering the hidden unseen classes of the rejected examples. Clearly, without prior knowledge this is difficult. However, we do have the data from the seen training classes, which can tell us what kind of similarity/difference is expected for examples from the same class or from different classes. It is reasonable to assume that this knowledge can be transferred to the rejected examples and used to discover the hidden unseen classes in them. This paper aims to solve this problem. It first proposes a joint open classification model with a sub-model for classifying whether a pair of examples belongs to the same or different classes. This sub-model can serve as a distance function for clustering to discover the hidden classes of the rejected examples. Experimental results show that the proposed model is highly promising.

The main goal of this paper is to cluster images from classes unseen during training.
This is an interesting extension of the open-world paradigm, where at test time, the classifier has to identify images beloning to the C seen classes during training, but also identify (reject) images which were previously unseen. These rejected images could be clustered to identify the number of unseen classes; either for revealing the underlying structure of the unseen classes, or to reduce annotation costs.
In order to do so, an extensive framework is proposed, consisting of 3 ConvNet architectures, followed by a hierarchical clustering approach. The 3 ConvNets all have a different goal:
1. an Open Classification Network (per class sigmoid, trained 1vsRest, with thresholds for rejection)
2. Pairwise Classification Network, (binary sigmoid, trained on pairs of images of same/different classes)
3. Auto encoder network
These network are jointly trained, and the joint-loss is simply the addition of a cross-entropy loss (from OCN), the binary cross-entropy loss (from PCN) and a pixel wise loss (from AE).
Remarks:
- it is unclear if the ConvNet weights of the first layers are shared).
- it is unclear how joint training might help, given that the objectives do not influence each other
- Eq 1:
  *label "y_i" has two different semantics (L_ocn it is the class label, while in L_pcn it is the label of an image pair being from the same class or not)
  * s_j is undefined
  * relation between the p(y_i = 1) (in PCN) and g(x_p,x_q) in Eq 2 could be made more explicit, PCN depends on two images, according to Eq 1, it seems just a sum over single images.
- It is unclear why the Auto Encoder network is added, and what its function is.
- It is unclear wether OCN requires/uses unseen class examples during training.
- Last paragraph of 3.1 "The 1-vs-rest ... rejected", I don't see why you need 1vsRest classifiers for this, a multi-class (softmax) output can also be thresholded to reject an test image from the known classes and to assign it to the unknown class.
Experimental evaluation
The experimental evaluation uses 2 datasets, MNIST and EMNIST, both are very specific for character recognition. It is a pity that not also more general image classification has been considered (CIFAR100, ImageNet, Places365, etc), that would provide insights to the more general behaviour of the proposed ideas.
My major concern is that the clustering task is not extensively explored. Just a single setting (with a single random sampling of seen/unseen classes) has been evaluated. This is -in part- due to the nature of the chosen datasets, in a 10 class dataset it is difficult to show the influence of the number of unseen classes. So, I'd really urge the authors to extend this evaluation. Will the method discover more classes when 100 unknown classes are used? What kind of clusters are discovered? Are the types of classes in the seen/unseen classes important, I'd expect at least multiple runs of the current experiments on (E)MNIST.
Further, I miss some baselines and ablation study. Questions which I'd like to seen answered: how good is the OCN representation when used for clustering compared to the PCN representation? What is the benefit of joint-training? How important is the AE in the loss?
Remaining remarks
- Just a very simple / non-standard ConvNet architecture is trained. Will a ResNet(32) show similar performance?
- In Eq 4, |C_i || y_j| seems a strange notation for union.
Conclusion
This paper brings in an interesting idea, is it possible to cluster the unseen classes in an open-world classification scenario?  A solution using a pairwise convnet followed by hierarchical clustering is proposed. This is a plausible solution, yet in total I miss an exploration of the solution.
Both in terms of general visual classification (only MNIST is used, while it would be nice to see results on CIFAR and/or ImageNet as in Bendale&Boult 2016), as in exploration of different scenarios (different number of unseen classes, different samplings) and ablation of the method (independent training, using OCN for hierarchical clustering, influence of Auto Encoder). Therefore, I rate this paper as a (weak) reject: it is just not (yet) good enough for acceptance.


We introduce FigureQA, a visual reasoning corpus of over one million questionanswer pairs grounded in over 100, 000 images. The images are synthetic, scientific-style figures from five classes: line plots, dot-line plots, vertical and horizontal bar graphs, and pie charts. We formulate our reasoning task by generating questions from 15 templates; questions concern various relationships between plot elements and examine characteristics like the maximum, the minimum, area-under-the-curve, smoothness, and intersection. To resolve, such questions often require reference to multiple plot elements and synthesis of information distributed spatially throughout a figure. To facilitate the training of machine learning systems, the corpus also includes side data that can be used to formulate auxiliary objectives. In particular, we provide the numerical data used to generate each figure as well as bounding-box annotations for all plot elements. We study the proposed visual reasoning task by training several models, including the recently proposed Relation Network as strong baseline. Preliminary results indicate that the task poses a significant machine learning challenge. We envision FigureQA as a first step towards developing models that can intuitively recognize patterns from visual representations of data.

Summary:
The paper introduces a new visual reasoning dataset called Figure-QA which consists of 140K figure images and 1.55M QA pairs. The images are generated synthetically by plotting perturbed sampled data using a visualization tool. The questions are also generated synthetically using 15 templates. Performance of baseline models and humans show that it is a challenging task and more advanced models are required to solve this task.
Strengths:
— FigureQA can help in developing models that can extract useful information from visual representations of data.
— Since performance on CLEVR dataset is already close to 100%, more challenging visual reasoning datasets would encourage the community to develop more advanced reasoning models. One of such datasets can be FigureQA.
— The paper is well written and easy to follow.
Weaknesses:
— Since the dataset is created synthetically, it is not clear if it is actually visual reasoning which is needed to solve this task, or the models can exploit biases (not necessarily language biases) to perform well on this dataset. In short, how do we know if the models trained on this dataset are actually learning something useful? One way to ensure this would be to show that models trained on this dataset can perform well on some other task. The first thing to try to show the usefulness of FigureQA is to show that the models trained on FigureQA dataset perform well on a real (figure, QA) dataset.
— The only advantages mentioned in the paper of using a synthetic dataset for this task are having greater control over task’s complexity and enabling auxiliary supervision signals, but none of them are shown in this paper, so it’s not clear if they are needed or useful.
— The paper should discuss what type of abilities are required in the models to perform well on this task, and how these abilities are currently not studied in the research community. Or in short, what new challenges are being introduced by FigureQA and how should researchers go about solving them on a high level?
— With what goal were these 15 types of questions chosen? Are these the most useful questions analysts want to extract out of plots? I am especially concerned about finding the roughest/smoothest and low/high median. Even humans are relatively bad at these tasks. Why do we expect models to do well on them?
— Why only binary questions? It is probably more difficult for analysts to ask a binary question than to ask non-binary ones such as “What is the highest in this plot?”
— Why these 5 types of plots? Can the authors justify that these 5 types of plots are the most frequent ones dealt by analysts?
— Are the model accuracies in Table 3 on the same subset as humans or on the complete test set? Can the authors please report both separately?
Overall:
The proposed dataset seems reasonable but neither the dataset seems properly motivated (something where analysts actually struggle and models can help) nor it is clear if it will actually be useful for the research community (models performing well on this dataset will need to focus on specific abilities which have not been studied in the research community).


Adversarial feature learning (AFL) is one of the promising ways for explicitly constrains neural networks to learn desired representations; for example, AFL could help to learn anonymized representations so as to avoid privacy issues. AFL learn such a representations by training the networks to deceive the adversary that predict the sensitive information from the network, and therefore, the success of the AFL heavily relies on the choice of the adversary. This paper proposes a novel design of the adversary, multiple adversaries over random subspaces (MARS) that instantiate the concept of the volunerableness. The proposed method is motivated by an assumption that deceiving an adversary could fail to give meaningful information if the adversary is easily fooled, and adversary rely on single classifier suffer from this issues. In contrast, the proposed method is designed to be less vulnerable, by utilizing the ensemble of independent classifiers where each classifier tries to predict sensitive variables from a different subset of the representations. The empirical validations on three user-anonymization tasks show that our proposed method achieves state-of-the-art performances in all three datasets without significantly harming the utility of data. This is significant because it gives new implications about designing the adversary, which is important to improve the performance of AFL.

The below review addresses the first revision of the paper. The revised version does address my concerns. The fact that the paper does not come with substantial theoretical contributions/justification still stands out.
---
The authors present a variant of the adversarial feature learning (AFL) approach by Edwards & Storkey. AFL aims to find a data representation that allows to construct a predictive model for target variable Y, and at the same time prevents to build a predictor for sensitive variable S. The key idea is to solve a minimax problem where the log-likelihood of a model predicting Y is maximized, and the log-likelihood of an adversarial model predicting S is minimized. The authors suggest the use of multiple adversarial models, which can be interpreted as using an ensemble model instead of a single model.
The way the log-likelihoods of the multiple adversarial models are aggregated does not yield a probability distribution as stated in Eq. 2. While there is no requirement to have a distribution here - a simple loss term is sufficient - the scale of this term differs compared to calibrated log-likelihoods coming from a single adversary. Hence, lambda in Eq. 3 may need to be chosen differently depending on the adversarial model. Without tuning lambda for each method, the empirical experiments seem unfair. This may also explain why, for example, the baseline method with one adversary effectively fails for Opp-L. A better comparison would be to plot the performance of the predictor of S against the performance of Y for varying lambdas. The area under this curve allows much better to compare the various methods.
There are little theoretical contributions. Basically, instead of a single adversarial model - e.g., a single-layer NN or a multi-layer NN - the authors propose to train multiple adversarial models on different views of the data. An alternative interpretation is to use an ensemble learner where each learner is trained on a different (overlapping) feature set. Though, there is no theoretical justification why ensemble learning is expected to better trade-off model capacity and robustness against an adversary. Tuning the architecture of the single multi-layer NN adversary might be as good?
In short, in the current experiments, the trade-off of the predictive performance and the effectiveness of obtaining anonymized representations effectively differs between the compared methods. This renders the comparison unfair. Given that there is also no theoretical argument why an ensemble approach is expected to perform better, I recommend to reject the paper.