Convolutional neural networks (CNNs) were inspired by human vision and, in some settings, achieve a performance comparable to human object recognition. This has lead to the speculation that both systems use similar mechanisms to perform recognition. In this study, we conducted a series of simulations that indicate that there is a fundamental difference between human vision and CNNs: while object recognition in humans relies on analysing shape, CNNs do not have such a shape-bias. We teased apart the type of features selected by the model by modifying the CIFAR-10 dataset so that, in addition to containing objects with shape, the images concurrently contained non-shape features, such as a noise-like mask. When trained on these modified set of images, the model did not show any bias towards selecting shapes as features. Instead it relied on whichever feature allowed it to perform the best prediction -even when this feature was a noise-like mask or a single predictive pixel amongst 50176 pixels. We also found that regularisation methods, such as batch normalisation or Dropout, did not change this behaviour and neither did past or concurrent experience with images from other datasets.

The paper seeks to establish via a series of well-designed experiments that CNNs trained for image classification differ in a fundamental way from human vision – they don’t encode shape-bias like human vision. Towards this goal, the authors modified the training data with ‘shortcut’ features to be functions of the category label using single diagnostic pixels and their placements, noise masks (salt and pepper, additive) and their parameters and demonstrate that image categorization CNNs learn whatever statistical features are there in the data most relevant to the learning task.
Investigation of the properties of neural architectures like CNNs and using the understanding thus developed to create better neural architectures, learning algorithms and training paradigms are good directions for the community and from that perspective, the direction explored in the paper is of great relevance and interest to the community.
The paper presents careful experimentation to establish that image categorization CNNs learn the statistical features most relevant to the learning task. And, it seems to satisfactorily demonstrate this. It shows that such features could be single pixels, noise masks and even parameters of stochastic distributions which randomly produce these features, as long as the parameters are predictive of the image category. The experiments are well designed and they demonstrate this point quite well. They also demonstrate the well-known problem of catastrophic forgetting.
Nonetheless, there are significant drawbacks in the presented work:
1.	The experiments don't seem to effectively demonstrate the main claim of the paper that categorization CNNs do not have inductive shape bias (encode shape information). (Let’s make this claim more concrete: categorization CNNs when trained via supervised learning with paired training data of {(image, category_label)} do not have inductive shape bias.)
The best way to demonstrate this would have been to subject a trained image-categorization CNN to test data with object shapes in a way that the appearance information couldn’t be used to predict the object label. The paper doesn’t do this. None of the experiments logically imply that with an unaltered training regime, a trained network would not be predictive of the category label if shapes corresponding to that category are presented.
2.	Due to the surprising results (especially the intensity of observed effects), we tried to reproduce some results from the paper in our lab and faced difficulties in doing so:
a.	We tried to replicate Figure 4(a) 'nopix' and 'same' cases on a standard setting (VGG-12-BN on CIFAR-10). The results deviated significantly (33%-72% margin) on ‘nopix’ case from the results reported in the paper on a much stronger setting (1/3072 pixels vs 1/50176 as in the paper). Please let me know any crucial settings (see below) that we might have missed.
Details: We used the vgg-cifar10 repository by chengyangfu. The only additions was fixing the pixel values while sending in the data. The code is anonymized and hosted here: https://file.io/qiziAK. The pixel values in CIFAR-10 using the pytorch dataloader are between [-0.45, 0.45] theoretically, typically much smaller. We set the (0,0) RGB pixels categorically spacing it uniformly from [-0.25, 0.25), [-0.025, 0.025), [-0.0025, 0.0025) as a simple experiment. The third case did not suffer any decrease in the nopix case or any increase in the pix at all. The first case showed significant deviations from the claimed results with the no-pix resulting in ~43% accuracy which is 33% off vis-à-vis the results in the paper. The ‘same’ setting didn’t achieve 100% either though it got close - achieving 98.4%.
Summary: The paper presents an important line of investigation to understand the properties of CNNs. However, it fails to effectively demonstrate its main claim. Further, we had difficulties in reproducing the results. As it stands, the submission is not of publishable quality.
I encourage the authors to do more careful experimentation to demonstrate their main claim and perhaps work on strategies to encourage CNNs to learn more meaningful features, including ‘shape’-features and submit to a future conference.
Revision: Updated my rating to acknowledge that the reproducibility issue is addressed.


There are more than 7,000 languages in the world, but due to the lack of training sets, only a small number of them have speech recognition systems. Multilingual speech recognition provides a solution if at least some audio training data is available. Often, however, phoneme inventories differ between the training languages and the target language, making this approach infeasible. In this work, we address the problem of building an acoustic model for languages with zero audio resources. Our model is able to recognize unseen phonemes in the target language, if only a small text corpus is available. We adopt the idea of zero-shot learning, and decompose phonemes into corresponding phonetic attributes such as vowel and consonant. Instead of predicting phonemes directly, we first predict distributions over phonetic attributes, and then compute phoneme distributions with a customized acoustic model. We extensively evaluate our English-trained model on 20 unseen languages, and find that on average, it achieves 9.9% better phone error rate over a traditional CTC based acoustic model trained on English.

Overview:
This paper proposed an approach for zero-shot phoneme recognition, where it is possible to recognise phonemes in a target language which has never been seen before. Rather than just training a phoneme recogniser directly on background data and then applying it to unseen data, phonetic features are first predicted, allowing phonemes not in the source language set to be predicted.
Main strengths:
The paper's main strength lies in that this is a very unexplored area that could assist in the development of speech technology where it is currently not possible. The proposed model (Section 2) has also not been considered in prior work.
Main weaknesses:
The paper's main weakness is in some of its claims and that it misses some very relevant literature. Detailed comments together with a minimal list of references are given below (but I would encourage the authors to also read a bit more broadly). But in short I do not think it is that easy to claim that this is the first paper to do zero-shot learning on speech; many of the zero-resource studies where unlabelled audio is used could be seen as doing some for of zero-shot matching. Specifically [5] is able to predict unseen phoneme targets.  Multilingual bottleneck features can be applied to languages that have never been seen before [2], and the output of phoneme recognisers trained on one language have long been applied to get output on another unseen language. The first one-shot learning speech paper [4] (to my knowledge) is also not mentioned at all. The approach in the paper also still relies on some text data from the target language; if this then can be described as "zero-shot" learning, then I think many of these previous studies c also make this claim.
Overall feedback:
There is definitely value in this work, but it should be much better situated within the broader literature. Below I give some editorial suggestions and also outline some suggestions for further experiments.
Detailed comments, suggestions and questions:
- Abstract: It would be useful to have some details of the "baseline model" here already, especially since it is such a new task.
- Introduction: "... but they can hardly predict phones or words directly due to their unsupervised nature." This is a strong statement that maybe requires more justification. On the one hand, the statement is true, and the high word error rates in e.g. [3] can be cited. On the other hand, it has been shown that at the phone-distinction level, these models perform quite well and sometimes outperform supervised models [1]. Since this paper also considers phone error rate as a metric, I think care should be taken with such statements.
- Introduction: "While zero-shot learning has attracted a lot of attention in *the* computer vision community, this setup has hardly been studied in speech recognition research especially in acoustic modeling." Definitely look at some of the studies mentioned below, and also [4] specifically.
- "However, we note that our model can be combined with a well-resourced language model to recognize words." How would this be done, since I think this is actually quite a challenging task.
- Section 2: "... useful the original ESZSL architecture ..." -> "... useful in the original ESZSL architecture ..."
- Section 2.2: I assume the small text corpus is at the phone level (and not characters directly)? This should be clarified, and it could raise the question of whether this approach is truly "zero-shot".
- Section 3.2: "We used EESEN framework ..." -> "We used the EESEN framework ..."
- Section 4: You could look at the recent work in [2], which uses multilingual bottleneck features trained on 10 languages and applied to multiple unseen languages. It would be interesting to also train your approach on multiple languages instead of only English.
Missing references:
1. M. Heck, S. Sakti, and S. Nakamura, "Feature Optimized DPGMM Clustering for Unsupervised Subword Modeling: A Contribution to Zerospeech 2017," in Proc. ASRU, 2017.
2. E. Hermann and S. J. Goldwater, "Multilingual bottleneck features for subword modeling in zero-resource languages," in Proc. Interspeech, 2018.
3. H. Kamper, K. Livescu, and S. Goldwater, An embedded segmental k-means model for unsupervised segmentation and clustering of speech," in Proc. ASRU, 2017.
4. B. M. Lake, C.-Y. Lee, J. R. Glass, and J. B. Tenenbaum, "One-shot learning of generative speech concepts," in Proc. CogSci, 2014.
5. O. Scharenborg, F. Ciannella, S. Palaskar, A. Black, F. Metze, L. Ondel, and M. Hasegawa-Johnson, "Building an ASR system for a low-resource language through the adaptation of a high-resource language asr system: Preliminary results,"in Proc. ICNLSSP, 2017.
Edit: Based on the rebuttal I've changed my rating from 4 to 5.


DON'T SETTLE FOR AVERAGE, GO FOR THE MAX: FUZZY SETS AND MAX-POOLED WORD VECTORS
Recent literature suggests that averaged word vectors followed by simple postprocessing outperform many deep learning methods on semantic textual similarity tasks. Furthermore, when averaged word vectors are trained supervised on large corpora of paraphrases, they achieve state-of-the-art results on standard STS benchmarks. Inspired by these insights, we push the limits of word embeddings even further. We propose a novel fuzzy bag-of-words (FBoW) representation for text that contains all the words in the vocabulary simultaneously but with different degrees of membership, which are derived from similarities between word vectors. We show that max-pooled word vectors are only a special case of fuzzy BoW and should be compared via fuzzy Jaccard index rather than cosine similarity. Finally, we propose DynaMax, a completely unsupervised and non-parametric similarity measure that dynamically extracts and max-pools good features depending on the sentence pair. This method is both efficient and easy to implement, yet outperforms current baselines on STS tasks by a large margin and is even competitive with supervised word vectors trained to directly optimise cosine similarity.

This submission presents a simple model for sentence representation based on max-pooling of word vectors. The model is motivated by fuzzy-set theory, providing both a funded pooling scheme and a similarity score between documents. The proposed approach is evaluated on sentence similarity tasks (STS) and achieves very strong performance, comparable to state-of-the-art, computationally demanding methods.
Pros:
+ The problem tackled by this paper is interesting and well motivated. Fast, efficient and non-parametric sentence similarity has tons of important applications (search, indexing, corpus mining).
+ The proposed solution is elegant and very simple to implement.
+ When compared to standard sentence representation models, the proposed approach has very good performance, while being very efficient. It only requires a matrix vector product and a dimension-wise max.
+ The paper is very well written and flows nicely.
+ Empirical results show significant differences between different word vectors. The simplicity of this approach makes it a good test bed for research on word vectors.
Cons:
- Nothing much, really.
- Eq. (3) is awkward, as it is a sequence of equalities, which has to be avoided. Moreover, if U is the identity, I don't think that the reader really need this Eq...
I have several questions and remarks that, if answered would make the quality of the presentation better:
* In infersent, the authors reported the performance of a randomly-initialized and max-pooled bi-lstm with fasttext vectors as the input lookup. This can be seen as an extreme case of the presented formalism, where the linear operator U is replaced by a complicated non linear function that is implemented by the random LSTM. Drawing that link, and maybe including this baseline in the results would be good.
* Related to this previous question, several choices for U are discussed in the paper. However, only two are compared in the experiments. It would be interesting to have an experimental comparison of:
- taking U = W
- taking U = I
- taking U as the principal directions of W
- taking U as a random matrix, and comparing performance for different output dimensions.
Overall, this paper is a very strong baseline paper. The presented model is elegant and efficient. I rate it as an 8 and await other reviews and the author's response.


EXECUTION-GUIDED NEURAL PROGRAM SYNTHESIS
Neural program synthesis from input-output examples has attracted an increasing interest from both the machine learning and the programming language community. Most existing neural program synthesis approaches employ an encoder-decoder architecture, which uses an encoder to compute the embedding of the given inputoutput examples, as well as a decoder to generate the program from the embedding following a given syntax. Although such approaches achieve a reasonable performance on simple tasks such as FlashFill, on more complex tasks such as Karel, the state-of-the-art approach can only achieve an accuracy of around 77%. We observe that the main drawback of existing approaches is that the semantic information is greatly under-utilized. In this work, we propose two simple yet principled techniques to better leverage the semantic information, which are execution-guided synthesis and synthesizer ensemble. These techniques are general enough to be combined with any existing encoder-decoder-style neural program synthesizer. Applying our techniques to the Karel dataset, we can boost the accuracy from around 77% to more than 90%.

This paper proposes guiding program synthesis with information from partial/incomplete program execution. The idea is that by executing partial programs, synthesizers can obtain the information of the state the (partial) program ended in and can, therefore, condition the next step on that (intermediate) state. The paper also mentions ensembling synthesizers to achieve a higher score, and by doing that it outperforms the current state-of-the-art on the Karel dataset program synthesis task.
In general, I like the idea of guiding synthesis with intermediate executions, and the evaluation in the paper shows this does make sense, and it outperforms the SOTA. The idea is original and the evaluation shows it is significant (enough). However, I have two major concerns with the paper, its presented contribution, and the clarity.
First, I cannot accept ensembling as a contribution to this paper. There is nothing novel about the ensemble proposed, and ensembling, as a standard method that pushes models that extra few percentage points, is present in a lot of other research. I have nothing against achieving SOTA results with it, while at the same time showing that the best performing model outperforms previous SOTA, which this paper orderly does. However, I cannot accept non-novel ensembling as a contribution of the paper.
Second, the clarity of the paper should be substantially improved:
- my main issue is that it is not clear how the Exec algorithm (see next point too) is trained. From what I understand Exec is trained on supervised data via MLE. What is the supervised data here?  Given the generality claims and the formulation in Algorithm 1/2, and possible ways one could use the execution information, as well as the fact that the model should be end-to-end trainable via MLE, it seems to me that the model is trained on prefixes (defined by Algorithm 1/2) of programs. Whether this is correct or not, please provide full details on how one can train Exec without using RL.
- By looking at Table 3, it seems that the generalization boost coming from Exec (I’m ignoring ensembling) is higher enough, and that’s great. However, it’s obvious that the exact match gain by Exec is minute, implying that the proposed algorithm albeit great on the generalization metric, does not improve the exact match at all. Do you have any idea why is that? Is that because Exec is trained via MLE and the Exec algorithm doesn’t add anything new to the training procedure?
- how do algorithm 1 and 2 exactly relate? I guess there is a meaning of ellipses in Lines 1 and 13, however, that is not mentioned anywhere. Is the mixture of algorithm 1 and 2 (and a non-presented algorithm for while loops) the Exec algorithm? How exactly are these algorithms joined, i.e what is the final algorithm?
- while on one side, I find some formalizations (problem definitions, definition 1, semantic rules in table 2) nicely done, I do not see their necessity nor big gains from them. In my opinion, the understanding of the rest of the paper does not depend on them, and they are well-described in the text.
- the paper says that the algorithm “helps boost the performance of different existing training algorithms”, however, it does so only on the Bunel et al model (and the MLE baseline in it), and albeit there’s mention of the generality, it has not been shown on anything other than those two models and the Karel dataset.
- do lines 6-7 in Algorithm 2 recurse? Does the model support arbitrarily nested loops/if statements?
- The claim that the shortest principle is most effective is supported by 2 data points, without any information on the variance of the prediction/dependence on the seed. Did you observe this for #models > 10 too? Up to what number?
- In table 3, is Exec on MLE? Could you please, for completeness, present the results of Exec + RL + ensemble in the table too?
- summarization, point 3 - what are the different modules mentioned here? Exec/RL/ensemble?
Minor issues, remarks, typos:
- table 1 position is very unfortunate
- figure 1 is not self-explanatory - it takes quite a lot of space to explain the network architecture, yet it fails to deliver meaning to parts of it (e.g. what is h_t^x, why is it max-pooled, what is g_t, etc)
- abstract & introduction - “Reducing error rate around 60%” absolute percentage points seem like a better evaluation measure (that the paper does use). Why is the error rate reduction necessary here?
- figure 2 - why is the marker in one of the corners, and not in the cell itself?
- Algorithm 1, step 4, is this here just as initialization, so S is non-empty to start with?
- Table 2 rule names are unclear (e.g. S-Seq-Bot ?)
- Table 3 mentions what Exec indicates twice


Current knowledge distillation methods require full training data to distill knowledge from a large "teacher" network to a compact "student" network by matching certain statistics between "teacher" and "student" such as softmax outputs and feature responses. This is not only time-consuming but also inconsistent with human cognition in which children can learn knowledge from adults with few examples. This paper proposes a novel and simple method for knowledge distillation from few samples. Taking the assumption that both "teacher" and "student" have the same feature map sizes at each corresponding block, we add a 1 × 1 conv-layer at the end of each block in the student-net, and align the block-level outputs between "teacher" and "student" by estimating the parameters of the added layer with limited samples. We prove that the added layer can be absorbed/merged into the previous conv-layer to formulate a new conv-layer with the same size of parameters and computation cost as the previous one. Experiments verify that the proposed method is very efficient and effective to distill knowledge from teacher-net to student-net constructing in different ways on various datasets.

Model distillation can be tricky and in my own experience can take a lot of samples (albeit unlabeled, so cheaper and more readily available), as well as time to train. This simple trick seems to be doing quite well at training students quickly with few samples. However, it departs from most student-teacher training that find its primary purpose by actually outperforming students trained from scratch (on the full dataset without time constraints). This trick does not outperform this baseline, so its emphasis is entirely on quick and cheap. However, it's unclear to me how often that is actually necessary and I don't think the paper makes a compelling case in this regard. I am borderline on this work and could probably be swayed either way.
Strengths:
- It's a very simple and fast technique. As I will cover in a later bullet point (under weaknesses), the paper does not make it clear why this type of model distillation is that useful (since it doesn't improve the student model over full fine-tuning, unlike most student-teacher work). However, the reason why I do see some potential for this paper is because there might be a use case in quickly being able to adapt a pretrained network. It is very common to start from a pretrained model and then attach a new loss and fine-tune. Under this paradigm, it is harder to make architectural adjustments, since you are starting from a finite set of pretrained models made available by other folks (or accept the cost of re-training one yourself). However, it is unclear how careful one needs to treat the pretrained model if more fine-tuning is going to occur. If for instance you could just remove layers, drop some channels, glue it all together, and then that model would still be reasonable as a pretrained model since the fine-tuning stage could tidy everything up, then this method would not be useful in this situation.
- The fact that least squares solvers can be used at each stage, without the need for a final end-to-end fine-tune is interesting.
- It is good that the paper demonstrates improvements coupled with three separate compression techniques (Li et al., Liu et al., Guo et al.).
- The paper is technically thorough.
- It's good that the method is evaluated on different styles of networks (VGG, ResNet, DenseNet).
Weaknesses:
- Limited application because it only makes the distillation faster and cheaper. The primary goal of student-teacher training in literature is to outperform a student trained from scratch by the wisdom of the teacher. It ties into this notion that networks are grossly over-parameterized, but perhaps that is where the training magic comes from. Student-teacher training acknowledges this and tries to find a way to benefit from the over-parameterized training and still end up with a small model. I think the same motivation is used for work in  low-rank decomposition and many other network compression methods. However, in Table 1 the "full fine-tune" model is actually the clear winner and presented almost as an upper bound here, so the only benefit this paper presents is quick and cheap model distillation, not better models. Because of this, I think this paper needs to spend more time making a case for why this is so important.
- Since this technique doesn't outperform full fine-tuning, the goal of this work is much more focused on pure model compression. This could put emphasis on reducing model size, RAM usage reduction, or FLOPS reduction. The paper focuses on the last one, which is an important one as it correlates fairly well with power (the biggest constraint in most on-device scenarios). However, it would be great if the paper gave a broader comparison with compression technique that may have slightly different focus, such as low-rank decomposition. Size and memory usage could be included as columns in tables like 1, along with a few of these methods.
- Does it work for aggressive compression? The paper presents mostly modest reductions (30-50%). I thin even if accuracy takes a hit, it could still work to various degrees. From what I can see, the biggest reduction is in Table 4, but FSKD is used throughout this table, so there is no comparison for aggressive compression with other techniques.
- The method requires appropriate blocks to line up. If you completely re-design a network, it is not as straightforward as regular student-teacher training. Even the zero-student method requires the same number of channels at certain block ends and it is unclear from the experiments how robust this is. Actually, a bit more analysis into the zero student would be great. For instance, it's very interesting how you randomly initialize (let's say 3x3) kernels, and then the final kernels are actually just linear combinations of these - so, will they look random or will they look fairly good? What if this was done at the initial layer where we can visualize the filters, will they look smooth or not?
Other comments:
- A comparison with "Deep Mutual Learning" might be relevant (Zhang et al.). I think there are also some papers on gradually adjusting neural network architectures (by adding/dropping layers/channels) that are not addressed but seem relevant. I didn't read this recently, but perhaps "Gradual DropIn of Layers to Train Very Deep Neural Networks" could be relevant. There is at least one more like this that I've seen that I can't seem to find now.
- It could be more clear in the tables exactly what cited method is. For instance, in Table 1, does "Fine-tuning" (without FitNet/FSKD) correspond to the work of Li et al. (2016)? I think this should be made more clear, for instance by including a citation in the table for the correct row. Right now, at a glance, it would seem that these results are only comparing against prior work when it compares to FitNet, but as I read further, I understood that's not the case.
- The paper could use a visual aid for explaining pruning/slimming/decoupling.
Minor comments:
- page 4, "due to that too much hyper-parameters"
- page 4, "each of the M term" -> "terms"
- page 6, methods like FitNet provides" -> "provide"


While deep learning has led to remarkable results on a number of challenging problems, researchers have discovered a vulnerability of neural networks in adversarial settings, where small but carefully chosen perturbations to the input can make the models produce extremely inaccurate outputs. This makes these models particularly unsuitable for safety-critical application domains (e.g. self-driving cars) where robustness is extremely important. Recent work has shown that augmenting training with adversarially generated data provides some degree of robustness against test-time attacks. In this paper we investigate how this approach scales as we increase the computational budget given to the defender. We show that increasing the number of parameters in adversarially-trained models increases their robustness, and in particular that ensembling smaller models while adversarially training the entire ensemble as a single model is a more efficient way of spending said budget than simply using a larger single model. Crucially, we show that it is the adversarial training of the ensemble, rather than the ensembling of adversarially trained models, which provides robustness.

Summary. The paper considers the robustness of neural nets against adversarial attacks. More precisely, the authors experimentally investigate the robustness of ensembles of neural nets. They empirically show that adversarially trained ensembles of 2 neural nets are more robust than ensembles of 2 adversarially trained neural nets.
Pros.
* Robustness of neural nets is a challenging problem of interest for ICLR
* The paper is easy to read
* Experimental results compare different algorithms for 2 neural nets
Cons.
* The study is experimental
* It is limited to gradient-based attacks
* It is limited to ensembles of size 2
* The Ensemble2Adv is a single NN model and not an ensemble model.
Evaluation.
The problem is significant and the use of ensemble methods for robustness against adversarial attacks is a promising line of research. The experimental study in this paper opens new lines of research in this direction. But, in my opinion, the paper is not ready for publication at ICLR. Detailed comments follow but the study is limited to k=2; the main finding is limited to the comparison between bagging two adversarially trained neural nets (SeparateEnsemble2Adv) and learning adversarially the average of two neural nets (Ensemble2adv). In my opinion, Ensemble2adv is a single model of double size and not an ensemble model thus somehow contradicting the main claim of the paper.
Detailed comments.
* Introduction, end of §2, it is said that non-gradient based attacks are still effective. But in the sequel you only consider gradient-based attacks and never discussed this question.
* Introduction, contributions, it should be made clear at the beginning of the paper that you will consider ensembles of size 2 and only gradient-based attacks.
* Section 2. The momentum-based attack should be cited and could be considered. "Boosting adversarial attacks with momentum, Dong et al, CVPR18"
* Section 3, §2, the discussion on ensemble methods is unprecise. Ensemble methods have different objectives. For instance, Bagging-like methods  aim at reducing the generalization error while others as Boosting aim at augmenting the capacity of individual models.
* Section 3. Here is my main concern on this paper. The classical method would be bagging of neural nets with different initializations. The neural nets could be adversarially trained. This would lead to the so-called SeparateEnsemble2Adv. Here, the authors consider another method. Their method can be viewed as k(=2) copies of the same neural network with different initializations and an additional layer computing the average of the k(=2) outputs. Then adversarially learn the obtained model which leads to the so-called Ensemble2Adv algorithm. This algorithm is not an ensemble method as such. In my opinion for k=2, it is equivalent to doubling the size of a neural net, adding averaging of the outputs, and adversarially training the obtained neural net.
* Note recent advances in ensemble NNs with papers such as Averaging weights leads to wider optima ..., Izmailov et al, UAI18; Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs, Garipov et al, arXiv:1802.10026
* Section 4.1. Here comes the limitation k=2. The case k=4 is considered in table 1 but is not discussed elsewhere in the paper.
* Section 4.1. I am not convinced by DoubleAdv. It is one way of doubling the size of a neural net but I am not convinced that this is the more efficient. As said before, in my opinion, Ensemble2Adv is another way for doubling the size. And many more should exist.
* Section 5. In my opinion, the main comparisons should concern SeparateEnsemble2Adv and Ensemble2Adv. Also other methods doubling the size should be considered.
* Section 5. For k greater than 2, SeparateEnsemblekAdv should be the better method because the adversarial learning phase could be easily parallelized.
* I am not convinced by the discussion in Section 6.
* Typos. and -> an l-13, p5; IFGSM5, l-19 p6; then l-6 p7; to due l-6 p9
* Biblio. Please give complete references


LEARNING IMPROVED DYNAMICS MODEL IN RE- INFORCEMENT LEARNING BY INCORPORATING THE LONG TERM FUTURE
In model-based reinforcement learning, the agent interleaves between model learning and planning. These two components are inextricably intertwined. If the model is not able to provide sensible long-term prediction, the executed planner would exploit model flaws, which can yield catastrophic failures. This paper focuses on building a model that reasons about the long-term future and demonstrates how to use this for efficient planning and exploration. To this end, we build a latent-variable autoregressive model by leveraging recent ideas in variational inference. We argue that forcing latent variables to carry future information through an auxiliary task substantially improves long-term predictions. Moreover, by planning in the latent space, the planner's solution is ensured to be within regions where the model is valid. An exploration strategy can be devised by searching for unlikely trajectories under the model. Our method achieves higher reward faster compared to baselines on a variety of tasks and environments in both the imitation learning and model-based reinforcement learning settings.

After the rebuttal and the authors providing newer experimental results, I've increased my score. They have addressed both the issue with the phrasing of the auxiliary loss, which I'm very happy they did as well as provided more solid experimental results, which in my opinion make the paper strong enough for publication.
#####
The paper proposes a variational framework for learning a Model of both the environment and the actor's policy in Reinforcement Learning. Specifically, the model is a deterministic RNN which at every step takes as input also a new stochastic latent variable z_t. Compared to more standard approaches, the prior over z_t is not standard normal but depends on the previously hidden state. The inference model combines information from the forward generative hidden state and a backward RNN that looks only at future observations. Finally, an auxiliary loss is added to the model that tries to predict the future states of the backward RNN using the latent variable z_t.  The idea of the paper is quite well presented and concise.
The paper tests the proposed framework on several RL benchmarks. Using it for imitation learning outperforms two baseline models: behaviour cloning and behaviour cloning trained with an auxiliary loss of predicting the next observation. Although the results are good, it would have been much better if there was also a comparison against a Generative model (identical to the one proposed) without the auxiliary loss added? The authors claim that the results of the experiment suggest that the auxiliary loss is indeed helping, where I find the evidence unconvincing given that there is no comparison against this obvious baseline. Extra comparison against the method from [1] or GAIL would make the results even stronger, but it is understandable that one can not compare against everything, hence I do not see this as a major issue.
The authors also compare on long-horizon video prediction. Although their method outperforms the method proposed in Ha & Schmidhuber, this by no means suggests that the method is really that superior. I would argue that in terms of future video prediction that [3] provides significantly better results than the World Models, nevertheless, at least one more baseline would have supported the authors claims much better.
On the Model-Based planning, the authors outperform SeCTAR model on the BabyAI tasks and the Wheeled locomotion. This result is indeed interesting and shows that the method is viable for planning. However, given that similar result has been shown in [1] regarding the planning framework it is unclear how novel the result is.
In conclusion, the paper presents a generative model for training a model-based approach with an auxiliary loss. The results look promising, however, stronger baselines and better ablation of how do different components actually contribute would make the paper significantly stronger than it is at the moment. Below are a few further comments on some specific parts of the paper.
A few comments regarding relevant literature:
Both in the introduction and during the main text the authors have not cited [1] which I think is a very closely related method. In this work similarly, a generative model of future segments is learned using a variational framework. In addition, the MPC procedure that the authors present in this paper is not novel, but has already been proposed and tried in [1] - optimizing over the latent variables rather than the actions directly, and there have been named Latent Action Priors.
The data gathering process is also not a new idea and using the error in a dynamics model for exploration is a well-known method, usually referred to as curiosity, for instance see [2] and some of the cited papers as Pathak et. al., Stadie et. al. - these all should be at least cited in section 3.2.2 as well not only in the background section regarding different topics.
On the auxiliary loss:
The authors claim that they train the auxiliary loss using Variational Inference, yet they drop the KL term, which is "kinda" an important feature of VI. Auxiliary losses are well understood that often help in RL, hence there is no need to over-conceptualize the idea of adding the extra term log p(b|z) as a VI and then doing something else. It would be much more clear and concise just to introduce it as an extra term and motivate it without referring to the VI framework, which the authors do not use for it (they still use it for the main generative model). The only way that this would have been acceptable if the experiment section contained experiments with the full VI objective as equation (6) suggest and without the sharing of the variational priors and posteriors and compared them against what they have done in the current version of the manuscript.
A minor mistake seems to be that equation (5) and (7) have double counted log p(z_t|h_t-1) since they are written as an explicit term as well as they appear in the KL(q(z_t|..)|p(z_t|h_t-1)).
[1] Prediction and Control with Temporal Segment Models [Nikhil Mishra, Pieter Abbeel, Igor Mordatch, 2017]
[2] Large-Scale Study of Curiosity-Driven Learning [Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, Alexei A. Efros, 2018]
[3] Action-Conditional Video Prediction using Deep Networks in Atari Games [Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard Lewis, Satinder Singh, 2015]


textTOvec: DEEP CONTEXTUALIZED NEURAL AUTOREGRESSIVE TOPIC MODELS OF LANGUAGE WITH DISTRIBUTED COMPOSITIONAL PRIOR
We address two challenges of probabilistic topic modelling in order to better estimate the probability of a word in a given context, i.e., P (word|context) : (1) No language structure in context: Probabilistic topic models ignore word order by summarizing a given context as a "bag-of-word" and consequently the semantics of words in the context is lost. In this work, we incorporate language structure by combining a neural autoregressive topic model (TM) (e.g., DocNADE) with a LSTM based language model (LSTM-LM) in a single probabilistic framework. The LSTM-LM learns a vector-space representation of each word by accounting for word order in local collocation patterns, while the TM simultaneously learns a latent representation from the entire document. In addition, the LSTM-LM models complex characteristics of language (e.g., syntax and semantics), while the TM discovers the underlying thematic structure in a collection of documents. We unite two complementary paradigms of learning the meaning of word occurrences by combining a topic model and a language model in a unified probabilistic framework, named as ctx-DocNADE. (2) Limited context and/or smaller training corpus of documents: In settings with a small number of word occurrences (i.e., lack of context) in short text or data sparsity in a corpus of few documents, the application of TMs is challenging. We address this challenge by incorporating external knowledge into neural autoregressive topic models via a language modelling approach: we use word embeddings as input of a LSTM-LM with the aim to improve the word-topic mapping on a smaller and/or short-text corpus. The proposed Doc-NADE extension is named as ctx-DocNADEe. We present novel neural autoregressive topic model variants coupled with neural language models and embeddings priors that consistently outperform state-of-theart generative topic models in terms of generalization (perplexity), interpretability (topic coherence) and applicability (retrieval and classification) over 7 long-text and 8 short-text datasets from diverse domains.

The paper extends an existing topic model - DocNADE - by replacing the feedforward part of the network which combines the textual context with an LSTM sequence model. Hence this paper fits in a long tradition of work which aims to extend the bad-of-words model from the original LDA paper with some sequence information.
The authors do a commendable job in thoroughly evaluating the proposed extension, using a number of evaluations based on perplexity, topic coherence, and text retrieval and categorization.
My main problem with the paper as it stands is that it a) arguably oversells the contribution, and b) is unclear when explaining certain crucial aspects of the model.
It would also help to have a clearer statement of whether the contribution here is on the document modeling side, or the language modeling side. Motivation is provided from both angles, but the evaluation focuses largely on the topic modeling (which is fine, just need to say it).
More specific comments:
--
The abstract should mention that the DocNADE model already exists, and that the contribution of the current work is to extend that existing model in a particular way. For those readers unfamiliar with DocNADE, this will help situate the work with regard to the existing literature.
Using existing word embeddings as a 'prior' for the LSTM word embeddings is a completely standard alternative now to learning those embeddings from scratch. I'm not sure that can count as a second, major contribution of the paper. (I'm also not sure that either extension to DocNADE warrants a new name, but I'll leave that to the authors' judgement.)
I'm confused by one aspect of the DocNADE model: "the topic assigned ...equally depends all the other words appearing in the same document". But the model is generative, no? And eqn 1 suggests that each word is generated conditioned on the *previous* words in the document, or did I miss something?
Related to this point, DocNADE transforms its BoWs into a sequence. But what's the order? Is it just the order of the words in the document? In which case it's very similar to the LSTM extension, except the LSTM keeps the order in the history, whereas the bag-of-words model doesn't.
Relation to generative models: LDA is a generative model with a generative story. It's not completely obvious to me what the generative story is in the new model. Talking about "distributed compositional priors" doesn't help, since I'm assuming these aren't priors in a Bayesian modeling sense? (It's also not clear in what sense these "priors" are compositional, but that's a separate question.)
Equation 2: what's the motivation for mixing the LSTM history with the bag-of-words (esp. if the history is from the same bag of words in each case). Why not just use the LSTM?
It would be useful to state in the main body of the text what the value of lambda ends up being. In 3.1 there's a suggestion this might be 0.01, but that effectively ignores the LSTM?
Similar question: how can the DocNADE model provide a *global* context if the model is generative?
Perplexity is a reasonable thing to measure, but presumably the auto-regressive nature of the LSTM means that it's more-or-less guaranteed to do better than a bag-of-words model? I wonder if it's worth acknowledging this fact?
I don't understand why lambda has to be zero "to compute the exact log-likelihood".
The first line of the conclusion doesn't say much: it's pretty obvious that the ordering of the words is going to help better estimate the probability of a word in a given context; 50 years of language modeling research has already taught us that.
Minor presentational comments:
--
Some of the hyphenation looks odd, eg in the title. Are you using the standard LaTeX hyphenation settings?
Strictly speaking, I'm not sure that 'bear' in the example is a proper noun.
"orderless sets of words": bags, not sets, since the counts matter, no?
The tables are too small, with a lot of numbers in them. One option is to move some of the details to the Appendix. Either way there needs to be more summary in the main body explaining what the numbers tell us.


This paper explores a simple regularizer for reinforcement learning by proposing Generative Adversarial Self-Imitation Learning (GASIL), which encourages the agent to imitate past good trajectories via generative adversarial imitation learning framework. Instead of directly maximizing rewards, GASIL focuses on reproducing past good trajectories, which can potentially make long-term credit assignment easier when rewards are delayed. GASIL can be easily combined with any policy gradient objective by using GASIL as a learned reward shaping function. Our experimental results show that GASIL improves the performance of proximal policy optimization on 2D Point Mass and MuJoCo environments with delayed reward and stochastic dynamics.

[Paper Summary]:
This paper proposes a regularization technique for existing RL algorithms by encouraging them to learn to reproduce the best past trajectories which obtained higher reward than that of current policy. The proposed method in the paper has the same high-level idea as "Self-imitation learning" [Oh et.al. ICML 2018] with a different objective. Instead of performing imitation learning to distill the knowledge from past best trajectories, this proposes to use inverse reinforcement learning via GAIL objective [Ho and Ermano, 2016]. The best k trajectories from past experience are stored to train a discriminator which is then used to augment the external reward function with a discriminator reward.
[Paper Strengths]:
The paper combines ideas from GAIL and self-imitation learning to propose a method that leverages past best trajectories via inverse-RL. This combination allows one to interpret self-imitation of best trajectories as a mechanism for "reward shaping" where learned discriminator shapes the environmental reward using past experiences. This is an exciting perspective and needs further discussion.
[Paper Weaknesses and Clarifications]:
=> This paper is very closely related to self-imitation learning [Oh et.al.], however, there is no theoretical justification provided (unlike [Oh et. al.]) whether the policy learned by optimizing Equation-11 is in anyway related to the optimal policy -- which was the case as shown in [Oh et.al.]. That being said, this is not a requirement for a paper to show theoretical justification as long as the paper justifies given approach with ample empirical evidence.
=> The main comparison point for the proposed approach, "GASIL", is "SIL" [Oh et.al.]. This paper provides a good comparison on continuous control tasks on Mujoco where "GASIL" performs slightly better than "SIL" in 3 out of 6 environments. However, "SIL" [Oh et. al.] showed extensive experiments on all Atari Games + Mujoco tasks. Since the proposed approach is mainly empirically motivated, the experiments should at least show a comparison on all the environments of the closely-related prior work. It would be much more convincing to see a bar chart across all 48 Atari Games showing relative improvement of "GASIL" over "SIL", as shown in Figure-4 of [Oh et. al.].
=> Other concerns:
    - The paper mentions on multiple occasions that the proposed method would handle delayed and "sparse" reward. However, it is not clear how can past best trajectories help with "sparse" rewards ("delayed-dense-rewards" seems alright, but they are not the same as "sparse"!). For instance, suppose the agent gets only terminal-reward in a maze. In such a case, the agent would need to rely on some form of exploration bonus (count-based, curiosity etc.) to reach the sparse-goal even once.
    - What prevents the learned policy from over-fitting to the local minima of the "locally" best trajectories seen so far?
[Final Recommendation]:
I request the authors to address the comments raised above. The paper has good potential, but sufficient empirical evidence is needed to justify the proposed technique. If the results on all Atari games can be included and shown to improve over "SIL", I would update my final rating.


Recurrent networks are a widely used class of neural architectures. They have, however, two shortcomings. First, it is difficult to understand what exactly they learn. Second, they tend to work poorly on sequences requiring long-term memorization, despite having this capacity in principle. We aim to address both shortcomings with a class of recurrent networks that use a stochastic state transition mechanism between cell applications. This mechanism, which we term stateregularization, makes RNNs transition between a finite set of learnable states. We show that state-regularization (a) simplifies the extraction of finite state automata modeling an RNN's state transition dynamics, and (b) forces RNNs to operate more like automata with external memory and less like finite state machines.

The paper proposes an RNN architecture inspired from deterministic pushdown automata. An RNN is extended to use soft attention at every time step to choose from several learnable centroids.
In general, the paper is well written and the proposed model is theoretically grounded. Unfortunately, the proposed approach shines only on specifically designed benchmarks. It is not a surprise that a CF can be learned by an architecture very similar to DPDA (with addition of learnable parameters). There is a number of specifically designed tasks to test long-term memorization, such as copy/addition, etc. Furthermore, RNNs are mostly used for natural language processing tasks. This paper only conducts experiments on IMDB sentiment analysis ignoring better benchmarked tasks, such as language modelling.
It is not absolutely clear why authors claim that cell is playing the role of memory. It is always possible to rewrite LSTM formulas with h' which is concatenation of hidden state h and cell c. Results on "peephole connection"-inspired SR-LSTM-p should be benchmarked against an LSTM with peephole connections.
The claim repeated several times that RNNs operate like DFAs, not DPDAs. This is an important point in the paper and should be verbalized more. Does it mean that it is easier to learn regular languages with RNNs?
While intuitive, theorems 3.1-3.2 are very vague to be theorems. Otherwise, they should be proven or provided a sketch of proof. For example, how do you formalize "state dynamics"?
The quality of writing of the related work section is worse that the rest of the paper. Authors should explore more other hidden state regularization methods. And, perhaps, give less attention to stochastic RNNs since the final version of the proposed model is not stochastic.
To summarize, this paper provides an interesting direction but lacks in terms of experimentation and global coherence of what is claimed and what is shown.
Minor points:
- Citation of Theano is missing
- Give a sentence explaining what is hidden state "drifting"
- a-priori -> a priori


There exist several stochastic optimization algorithms. However in most cases, it is difficult to tell for a particular problem which will be the best optimizer to choose as each of them are good. Thus, we present a simple and intuitive technique, when applied to first order optimization algorithms, is able to improve the speed of convergence and reaches a better minimum for the loss function compared to the original algorithms. The proposed solution modifies the update rule, based on the variation of the direction of the gradient during training. We conducted several tests with Adam and AMSGrad on two different datasets. The preliminary results show that the proposed technique improves the performance of existing optimization algorithms and works well in practice.

The paper considers a simplistic extension of first order methods typically used for neural network training. Apart from the basic idea the paper's actual algorithm is hard to read because it is full of lacking definitions. I have tried to piece together whatever I could by reading the proof. The algorithm box is very unclear. For instance the * operator is undefined.
To the best of my understanding which the paper changes the update by first checking whether the gradient has the same direction as the previous gradient if yes it uses the component wise maximum of the new gradient and the previous gradient in the update and otherwise it uses the new gradient. Now whether this if condition is checked component wise or an angle between the two vectors is completely unclear.
I will really suggest the authors to at least write their algorithm with clarity. Further while stating the theorem there are undefined parameter and even the objective Regret has not been defined anywhere. Further the theorem which I could not verify due to similar unclarity shows I believe the same convergence result as AMSGrad and hence there is no theoretical advantage for the proposed algorithm. In terms of practice further I do not see a significant advantage and it could result be a step size issue . The authors do not say that they do a search over the hyper parameters.
On a philosophical level it is unclear what the motivation behind this particular change to any algorithm is. It would be good to discuss what additional advantage is added on top of acceleration. Note that the method feels very much like acceleration.


Automatic Chemical Design provides a framework for generating novel molecules with optimized molecular properties. The current model suffers from the pathology that it tends to produce invalid molecular structures. By reformulating the search procedure as a constrained Bayesian optimization problem, we showcase improvements in both the validity and quality of the generated molecules. We demonstrate that the model consistently produces novel molecules ranking above the 90th percentile of the distribution over training set scores across a range of objective functions. Importantly, our method suffers no degradation in the complexity or the diversity of the generated molecules.

The authors proposed a new method improving a previous Bayesian optimization approach for chemical design (Gomez-Bombarelli et al., 2016b) by addressing the problem that data points need to have valid molecular structures. The main contribution is a constrained Bayesian optimization approach that take into account the constraint on the probability of being valid.
My biggest concern of this paper is that it is not really using sequential evaluations to do automatic design of experiments on molecules. The experiments do not seem to fully support the proposed approach in terms of being able to adaptively do sequential evaluations.
Detailed comments:
1. The term inside the expectation of the EI criterion in Sec 2.5 should be max(0, f(m)-\eta) rather than max(0, \eta - f(m)).
2. The EIC criterion the authors adopted uses Pr(C(m)) if the constraint is violated everywhere with high probability. It seems like Pr(C(m)) does not have the ability to explore regions with high uncertainty. How does this approach compare to Bayesian level set estimation approaches like
B. Bryan, R. C. Nichol, C. R. Genovese, J. Schneider, C. J. Miller, and L. Wasserman, “Active learning for identifying function threshold boundaries,” in NIPS, 2006
I. Bogunovic, J. Scarlett, A. Krause, and V. Cevher, “Truncated variance reduction: A unified approach to bayesian optimization and level-set estimation,” in NIPS, 2016.
3. It would be good to explain in more detail how a constraint is labeled to be valid or invalid.
4. What is the space of m in Sec 2.3 and the space of m in Sec 2.4? It seems like there is a discrepancy: Sec 2.3 is talking about m as a molecule but Sec 2.4 describes f as a function on the latent variable? It would be good to be clear about it.
5. In the experiment, I think it is very necessary to show the effectiveness of the constrained BO approach in terms of how the performance varies as a function of the number of evaluations on the constraint and the function. The current empirical results only support the claim that the constrained BO approach is able to output more valid latent variables and the function values from constrained BO is higher than vanilla BO under the same number of training data. It is also strange why there is a set of test data.
Typos/format:
1. citation format problems across the paper, e.g.
"Gomez-Bombarelli et al. (Gomez-Bombarelli et al., 2016b) presented Automatic Chem"
"SMILES strings Weininger (1988) are a means of representing molecules as a character sequence."
It's likely a problem of misuse of \cite, \citep.
2. no period at the end of Sec 2.4


We propose a new algorithm to learn a one-hidden-layer convolutional neural network where both the convolutional weights and the outputs weights are parameters to be learned. Our algorithm works for a general class of (potentially overlapping) patches, including commonly used structures for computer vision tasks. Our algorithm draws ideas from (1) isotonic regression for learning neural networks and (2) landscape analysis of non-convex matrix factorization problems. We believe these findings may inspire further development in designing provable algorithms for learning neural networks and other complex models. While our focus is theoretical, we also present experiments that illustrate our theoretical findings.

This paper gives a new algorithm for learning a two layer neural network which involves a single convolutional filter and a weight vector for different locations. The algorithm works on any symmetric input data. The techniques in this paper combines two previous approaches: 1. the algorithm Convotron for learning a single convolutional filter (while the second layer has fixed weight) on any symmetric input distributions; 2. non-convex optimization for low rank matrix factorization.
The main observation in the paper is that if the overlap in the convolutions is not large (in the sense that each location of the convolution has at least one input coordinate that is not used in any other locations), then the weight that corresponds to the non-overlapping part and the weights in the second layer can be computed by a matrix factorization step (the paper gives a way to estimate a gradient that is similar to the gradient for a linear neural network, and then the problem is very similar to a rank-1 matrix factorization). After this step, we know the second layer and the algorithm can generalize the previous Convotron algorithm to learn the full convolutional filter.
This is an interesting observation that allows the algorithm to learn a two-layer neural network. On the other hand this two layer neural network is still a bit limited as there is still only one convolutional filter, and in particular there is only one local and global optimum (up to scaling the two layers). The observation also limited how much the patches can overlap which was not a problem in the original convotron algorithm.
Overall I feel the paper is interesting but a bit incremental.


AN ANALYTIC THEORY OF GENERALIZATION DYNAM- ICS AND TRANSFER LEARNING IN DEEP LINEAR NET- WORKS
Much attention has been devoted recently to the generalization puzzle in deep learning: large, deep networks can generalize well, but existing theories bounding generalization error are exceedingly loose, and thus cannot explain this striking performance. Furthermore, a major hope is that knowledge may transfer across tasks, so that multi-task learning can improve generalization on individual tasks. However we lack analytic theories that can quantitatively predict how the degree of knowledge transfer depends on the relationship between the tasks. We develop an analytic theory of the nonlinear dynamics of generalization in deep linear networks, both within and across tasks. In particular, our theory provides analytic solutions to the training and testing error of deep networks as a function of training time, number of examples, network size and initialization, and the task structure and SNR. Our theory reveals that deep networks progressively learn the most important task structure first, so that generalization error at the early stopping time primarily depends on task structure and is independent of network size. This suggests any tight bound on generalization error must take into account task structure, and explains observations about real data being learned faster than random data. Intriguingly our theory also reveals the existence of a learning algorithm that proveably out-performs neural network training through gradient descent. Finally, for transfer learning, our theory reveals that knowledge transfer depends sensitively, but computably, on the SNRs and input feature alignments of pairs of tasks.

This paper builds on the long recent tradition of analyzing deep linear neural networks. In addition to an ample appendix bringing the page total to 20, the authors went over the recommended eight pages, hitting the hard limit of 10 and thus per reviewing directions will be held to a higher standard than the other (mostly 8-page) papers.
The recent literature on deep linear networks has explored many paths with the hope of producing insights that might help explain the performance of deep neural networks. A recent line of papers by Soudry and Srebro among others focuses on the behavior of stochastic gradient descent. This paper’s analysis comes from a different angle, following the work by Saxe et al (2013) whose analysis considers a (classic one hidden layer) linear teacher network that generates labels and a corresponding student trained to match those labels. The analysis hinges on the singular value decomposition of the composite weight matrix USV^T = W = W^{32} W^{21}.
One aim of the present work, that appears to be a unique contribution above the prior work is to focus on the role played by task structure, suggesting that certain notions of task structure may play a more significant role than architecture and that any bounds which consider architecture but not task structure are doomed to be excessively loose.
To facilitate their analysis, the authors consider an artificial setup that requires some specific properties. For example, the number of inputs are equal to the input dimension of the network, with the inputs themselves being orthonormal. The labeling function includes a noise term and the singular values of the teacher model admit an interpretation as signal to noise ratios. Given their setup, the authors can express the train and test errors analytically in terms of the weight matrices of the teacher and student and the input-output covariance matrix. The authors then analyze the gradient descent dynamics in what appears to follow the work of Saxe 2013 although I am not an expert on that paper. The analysis focuses on the time dependent evolution of the singular values of the student model, characterized via a set of differential equations.
The next analysis explores a condition that the authors dub “training aligned” initial conditions. This involves initializing the student weights to have the same singular vectors as the training data input-output covariance but with all singular values equal to some amount epsilon. The authors show that the learning dynamics give rise to what they characterize as a singular value “detection wave”. Detecting the modes in descending order by their corresponding singular values.
A set of synthetic experiments show close alignment between theory and experiment.
Section 3.5 offers just one paragraph on a “qualitative comparison to nonlinear networks”. A few issues here are that aesthetically, one-paragraph subsections are not ideal. More problematic is that this theory presumably is building towards insights that might actually be useful towards understanding deep non-linear networks. Since the present material is only interesting as an analytic instrument, I would have hoped for greater emphasis on these connections, with perhaps some hypotheses about the behavior of nonlinear nets driven by this analysis that might subsequently be confirmed or refuted.
The paper concludes with two sections discussing what happens when nets are trained on randomly labeled data and knowledge transfer across related tasks respectively.
Overall I think the paper is well-written and interesting, and while I haven’t independently verified every proof, the technical analysis appears to be interesting and sound. The biggest weaknesses of this paper---for this audience, which skews empirical---concern the extent to which the work addresses or provides insight about real neural networks. One potential weakness in this line of work may be that it appears to rely heavily on the linearity of the deep net. While some other recent theories seem more plausibly generalized to more general architectures, it’s not clear to me how this analysis, which hinges so crucially on the entire mapping being expressible as a linear operator, can generalize.
On the other hand, I am personally of the opinion that the field is in the unusual position of possessing too many tools that “work” and too few new ideas. So I’m inclined to give the authors some license, even if I’m unsure of the eventual utility of the work.
One challenge in reviewing this paper is that it builds tightly on a number of recent papers and without being an authority on the other works, while it’s possible to assess the insights in this paper, it’s difficult to say conclusively which among them can rightly be considered the present paper’s contributions (vs those of the prior work).


Aligning knowledge graphs from different sources or languages, which aims to align both the entity and relation, is critical to a variety of applications such as knowledge graph construction and question answering. Existing methods of knowledge graph alignment usually rely on a large number of aligned knowledge triplets to train effective models. However, these aligned triplets may not be available or are expensive to obtain for many domains. Therefore, in this paper we study how to design fully-unsupervised methods or weakly-supervised methods, i.e., to align knowledge graphs without or with only a few aligned triplets. We propose an unsupervised framework based on adversarial training, which is able to map the entities and relations in a source knowledge graph to those in a target knowledge graph. This framework can be further seamlessly integrated with existing supervised methods, where only a limited number of aligned triplets are utilized as guidance. Experiments on real-world datasets prove the effectiveness of our proposed approach in both the weakly-supervised and unsupervised settings.

The authors propose KAGAN, a novel method for knowledge graph (KG) alignments using GANs. In contrast to most other methods, KAGAN does not rely on a supervised setting where a set of already aligned triples is used as seed. In addition, the authors propose modifications such that their method can also integrate information about aligned triples.
In my opinion the biggest strength of the paper is the novelty of the proposed method. The standard framework of GANs is adjusted elegantly to the KG alignment setting. Further, the basic approach as well as the proposed modification to deal with practical issues such as mode collapse are well motivated and comprehensible. The ability to perform well in the unsupervised/weakly supervised setting is another plus. Up to my knowledge, this paper constitutes the first approach using GANs for the KG alignment task. However, there are several methods that use GANs directly to produce KG embeddings (e.g. KBGAN). These methods are related to KAGAN since they employ TransE as an underlying embedding method. Therefore, I strongly believe the authors should include [1] and [2] in their references and discuss these methods briefly.
The authors conduct experiments on benchmark datasets for the KG alignment tasks. Thereby, they aim to show that their KAGAN outperforms other state of the art methods. However, I have one major concern: In the subsection 'Parameter Settings' the authors state without further explanation that they set the embedding size to 512 for all compared methods. I do not understand the rationale behind this decision. In particular, since the embedding sizes in the original publications (e.g., in Chen et al. (2017a)) are very different. For KG embedding methods the dimensionality of the embeddings is probably the most important hyperparameter. In my opinion, picking a global value for all methods instead of tuning it for each method individually does not guarantee a fair comparison.
On a side note, the authors might also consider the mean rank as performance measure which is frequently employed in the KG alignment setting.
Overall, the paper is well written and organized. A minor point: I would get rid of the formulas in the introduction; in particular, since the notation is not introduced at this point.
While the novelty and good structure of the paper are reasons to accept it, I have doubts concerning the soundness of the results due to the experimental setup.
Reasons to accept the paper
- novelty
- works in an unsupervised setting
- well written/structured
Reasons to reject
-  Doubts concerning the experimental setting
- (Minor) related work is not complete
- (Minor) not all common performance measures are reported
----------------------------
[1] Wang, Peifeng, Shuangyin Li, and Rong Pan. "Incorporating GAN for Negative Sampling in Knowledge Representation Learning." (2018).
[2] Cai, Liwei, and William Yang Wang. "KBGAN: Adversarial Learning for Knowledge Graph Embeddings." Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). Vol. 1. 2018.


Interpretability has largely focused on local explanations, i.e. explaining why a model made a particular prediction for a sample. These explanations are appealing due to their simplicity and local fidelity. However, they do not provide information about the general behavior of the model. We propose to leverage model distillation to learn global additive explanations that describe the relationship between input features and model predictions. These global explanations take the form of feature shapes, which are more expressive than feature attributions. Through careful experimentation, we show qualitatively and quantitatively that global additive explanations are able to describe model behavior and yield insights about models such as neural nets. A visualization of our approach applied to a neural net as it is trained is available at https://youtu.be/ATNcgurNHhc.

Summary: This paper makes an interesting contribution of providing global explanations of black box models (such as neural nets) using a special class of models called generalized additive models.  While the paper is well written and experiments are quite detailed, I have some problems with some of the basic premises of this work.
1. The concept of using simpler models to approximate other complex models (model distillation) is not new and has been explored quite a bit already in ML literature. The only new proposition of this work is to use generalized additive models to approximate other complex models. This seems rather incremental.
2. The premise behind using generalized additive models (GAMs) to explain other complex models is that GAMs are interpretable. I am not convinced about this premise. While I can intuitively see that GAMs might be able to better approximate complex models compared to rules and trees, I highly doubt if they are even interpretable.
Pros:
1. The paper is well written
2. Experiments are very detailed and thorough
Cons:
1. The proposed approach lacks novelty
2. Experimentation lacks a user study which helps understand if and when GAMs are at least as interpretable as rule-based approaches.
Detailed Comments:
I actually like the way this paper is written and executed. The writing is very clear and experiments are quite thorough. But, as discussed earlier, I have some issues with the basic premises of this paper i.e., novelty of the proposed approach and justification for the claim that GAMs are interpretable. I would encourage the authors to discuss these two aspects in their rebuttal. I would strongly encourage the authors to carry out at least a simple user study which compares the interpretability aspect of GAMs with rule-based or prototype based approaches.


There are great interests as well as many challenges in applying reinforcement learning (RL) to recommendation systems. In this setting, an online user is the environment; neither the reward function nor the environment dynamics are clearly defined, making the application of RL challenging.
In this paper, we propose a novel model-based reinforcement learning framework for recommendation systems, where we develop a generative adversarial network to imitate user behavior dynamics and learn her reward function. Using this user model as the simulation environment, we develop a novel DQN algorithm to obtain a combinatorial recommendation policy which can handle a large number of candidate items efficiently. In our experiments with real data, we show this generative adversarial user model can better explain user behavior than alternatives, and the RL policy based on this model can lead to a better long-term reward for the user and higher click rate for the system.

This paper proposes to frame the recommendation problem as one of (model-based) RL. The two main innovations are: 1) a model and objective for learning the environment and reward models; 2) a cascaded DQN framework for reasoning about a combinatorial number of actions (i.e., which subset of items to recommend to the user).
The problem is clearly important and the authors' approach focuses on solving some of the current issue with deployment of RL-based recommenders. Overall the paper is relatively easy to follow, but the current version is not the easiest to understand and, in particular, it may be worth providing more intuitions (e.g., about the GAN-like setup). I also found that several decisions are not properly justified. The novelty of this paper seems reasonably high but my impression is that other/stronger baselines would make the study more convincing. Copy-editing the paper would also greatly improve readability.
Detailed comments:
- I am not clear on whether or not in the proposed model, users are "allowed" to not click on a recommendation. It sounds like the authors in fact allow it but I think that could be made clearer.
- Section 4. I am not sure that using the Generative Adversarial Network terminology is useful here. Specifically, it is not clear what is your generative model over (I imagine next state and reward?).

- Remark in Section 4.1: It seems like a user not clicking on something is also useful information. Why not model it?

- I am a bit unclear on the value of Lemma 1. Further, what are the assumptions behind it? (also what is this temperature parameter eta?)

- In Section 4.2, the size of your model seems to grow linearly with the number of user interactions. That seems like a major advantage of RNNs/LSTMs. In practice, I imagine you cull the history at some fixed point?

- What is the advantage of learning a reward? E.g., a very simple reward would be to give a positive reward if a user clicks on a recommended item and a negative reward otherwise. What does your learned reward allow beyond this?

- Section 4.3. I also found Section 4.3 to be relatively unclear. I find that more intuition would be helpful.
  Also, if Eq. 7 is equivalent to Eq. 8, then why is the solution of 8 used only to initialize 7? I guess it may have to do with not finding the global optimum.
- Your cascading DQN idea seems like a good one. It would be nice to check if the constraints are correctly learned. If not, this seems like it would do not better than a greedy action-by-action solution. Is that correct?

- In Section 6.1, it would be good to discuss the pre-processing in the main text since it's pretty important to understand the study (e.g., evaluate is impact).

- In 6.2, your baselines seem a bit weak. Why not compare to more recent CF models (e.g., including Session-Based RNNs which you cite earlier)?

- Related work: it would probably be good to survey some of the multi-arm bandit literature. There is also some CF-RL work which should be cited (perhaps there are a few things in there that should be compared to in Section 6.3 & 6.4).
- Section 6.2 and Table 1. I believe that Recall@k is most common in recommendation-systems-for-implicit-data literature. Or, are you assuming that what people do not click on are true negatives? This doesn't seem quite right as users are only allowed to click on a single item.
- In Section 6.3, could you clarify how do you learn your reward model that is used to train the various methods?
- There are many typos and grammatical errors in the paper. I would suggest that the authors carefully copy-edit the manuscript.


AUGMENTED CYCLIC ADVERSARIAL LEARNING FOR LOW RESOURCE DOMAIN ADAPTATION
Training a model to perform a task typically requires a large amount of data from the domains in which the task will be applied. However, it is often the case that data are abundant in some domains but scarce in others. Domain adaptation deals with the challenge of adapting a model trained from a data-rich source domain to perform well in a data-poor target domain. In general, this requires learning plausible mappings between domains. CycleGAN is a powerful framework that efficiently learns to map inputs from one domain to another using adversarial training and a cycle-consistency constraint. However, the conventional approach of enforcing cycle-consistency via reconstruction may be overly restrictive in cases where one or more domains have limited training data. In this paper, we propose an augmented cyclic adversarial learning model that enforces the cycle-consistency constraint via an external task specific model, which encourages the preservation of task-relevant content as opposed to exact reconstruction. We explore digit classification in a low-resource setting in supervised, semi and unsupervised situation, as well as high resource unsupervised. In low-resource supervised setting, the results show that our approach improves absolute performance by 14% and 4% when adapting SVHN to MNIST and vice versa, respectively, which outperforms unsupervised domain adaptation methods that require high-resource unlabeled target domain. Moreover, using only few unsupervised target data, our approach can still outperforms many high-resource unsupervised models. Our model also outperforms on USPS to MNIST and synthetic digit to SVHN for high resource unsupervised adaptation. In speech domains, we similarly adopt a speech recognition model from each domain as the task specific model. Our approach improves absolute performance of speech recognition by 2% for female speakers in the TIMIT dataset, where the majority of training samples are from male voices.

The authors propose an extension of cycle-consistent adversarial adaptation methods in order to tackle domain adaptation in settings where a limited amount of supervised target data is available (though they also validate their model in the standard unsupervised setting as well). The method appears to be a natural generalization/extension of CycleGAN/CyCADA. It uses the ideas of the semantic consistency loss and training on adapted data from CyCADA, but "fills out" the model by applying these techniques in both directions (whereas CyCADA only applied them in the source-to-target direction).
The writing in this paper is a little awkward at times (many omitted articles such as "the" or "a'), but, with a few exceptions, it is generally easy to understand what the authors are saying. They provide experiments in a variety of settings in order to validate their model, including both visual domain adaptation and speech domain adaptation. The experiments show that their model is effective both in low-resource supervised adaptation settings as well as high-resource unsupervised adaptation settings. An ablation study, provided in Section 4.1, helps to understand how well the various instantiations of the authors' model perform, indicating that enforcing consistency in both methods is crucial to achieving performance beyond the simple baselines.
It's a little hard to understand how this method stands in comparison to existing work. Table 3 helps to show that the model can scale up to the high-resource setting, but it would also be nice to see the reverse: comparisons against existing work run in the limited data setting, to better understand how much limited data negatively impacts the performance of models that weren't designed with this setting in mind.
I would've also liked to see more comparisons against the simple baseline of a classifier trained exclusively on the available supervised target data, or with the source and target data together—in my experience, these baselines can prove to be surprisingly strong, and would give a better sense of how effective this paper's contributions are. This corresponds to rows 2 and 3 of Table 1, and inspection of the numbers in that table shows that the baseline performance is quite strong even relative to the proposed method, so it would be nice to see these numbers in Table 2 as well, since that table is intended to demonstrate the model's effectiveness across a variety of different domain shifts.
While it's nice that the model is experimentally validated on the speech domain, the experiment itself is not explained well. The speech experiments are hard to understand—it's unclear what the various training sets are, such as "Adapted Male" or "All Data," making it hard to understand exactly what numbers should be compared. Why is there no CycleGAN result for "Female + Adapted Male," or "All Data + Adapted Male," for example? The paper would greatly benefit from a more careful explanation and analysis of this experimental setting.
Ultimately, I think the idea is a nice generalization of previous work, and the experiments seem to indicate that the model is effective, but the limited scope of the experiments prevent me from being entirely convinced. The inclusion of additional baselines and a great deal of clarification on the speech experiments would improve the quality of this paper enormously.
---
Update: After looking over the additional revisions and experiments, I'm bumping this to a weak accept. I agree with reviewer 3 that novelty is not the greatest, but there is a useful contribution here, and the demonstration of its effectiveness on low resource settings is valuable, since in a practical setting it is usually feasible to manually label a few examples.
I'm still not convinced by the TIMIT experiments, now that I better understand them, since the F+M baseline is quite strong and very simple to run. It simply doesn't seem worthwhile to introduce all of this extra machinery for such a marginal improvement, but the experiment does serve the job of at least demonstrating an improvement over existing methods.


We propose a general-purpose approach to discovering active learning (AL) strategies from data. These strategies are transferable from one domain to another and can be used in conjunction with many machine learning models. To this end, we formalize the annotation process as a Markov decision process, design universal state and action spaces and introduce a new reward function that precisely model the AL objective of minimizing the annotation cost We seek to find an optimal (non-myopic) AL strategy using reinforcement learning. We evaluate the learned strategies on multiple unrelated domains and show that they consistently outperform state-of-the-art baselines.

Summary:
This paper presents an RL approach to active learning that is generic across ML model being learned, and across dataset being used. The paper formulates the standard active learning problem as an MDP with the objective of minimizing the number of annotated labels required to meet a pre-specified prediction quality.
The MDP state proposed by this paper is the current performance score on each sample in a hold-out set. The actions are specified by selecting a datapoint from the set of all un-annotated datapoints. The action feature vector consists of the current performance score of the model on the datapoint, and the average distance of that datapoint from every datapoint in the labeled set and every datapoint in the unlabeled set.
Review:
I do not recommend this paper for publication in ICLR because I believe:
1) the work is too incremental
2) the comparison to baseline and competing methods is incomplete
3) some design decisions of the proposed method are not well motivated.
I appreciated the clarity of the writting, and the paper organization. I also believe that the proposed method is quite intuitive, and is a good addition to the field. Finally, I appreciate that sufficient experimental details are available within the paper to be able to easily reproduce the results.
Details:
My points (1) and (2) are highly related, so I will discuss both simultaneously. I find that this paper makes only incremental forward progress from the Pang 2018 paper and the Konyushkova 2017 paper. The methodology here looks very similar to the SingleRL method, which Pang 2018 notes can be considered a special case of Konyushkova 2017's method. I think that the work in this paper would be sufficient to stand on its own if it performed a convincing comparison to SingleRL and/or MLP-GAL from Pang 2018. I recognize that this paper references why no such comparison currently exists, but I think this comparison would be extremely valuable to the paper.
A further comment on my point (2), I do not find the comparisons to baseline methods to be entirely convincing. Of note, only the average performance for each method is reported. I'm curious of the variance---and more specifically the standard error and number of independent runs---of each of the reported results. On many of the datasets, the performance difference between the proposed method and uncertainty sampling is quite small in table 1.
A final comment on point (2): I would have liked to see more exploration of different models. I think table 2 is quite informative, showing notable differences between simple baseline AL methods. I would have liked to see table 2 with more classifiers and with more competing AL methods. Because logistic regression is a simple model, the differences between AL methods may be more subtle. Perhaps a more complex model (say a single hidden layer NN) would show more notable differences.
For point (3), I would have liked to see either an exploration of other design decisions or an explanation of given design decisions. For instance, why only use 30 hold-out samples for the state? I imagine the proposed method would be fairly sensitive to this choice.  Another unexplained design decision was using a maximum budget of 100 datapoints. Table 2 shows some extremely interesting interactions with this budget in its comparison between LogReg-100 and LogReg-200, and further explanation would have been useful. Finally, I would have liked to see some motivation for choice of stopping condition. Using the stopping condition of 98% of maximum performance may have some biasing effect of each method, and it would helpful to have some motivation behind this choice.
Questions:
 - Why did uncertainty sampling have such limited benefits on LogReg-200 in table 2? This was a surprising result to me, as uncertainty sampling consistently outperformed most other methods.
 - Why is there a disparity between the results for the SVM in table 2 and the discussion in the first paragraph of section 4.3?
 - How does choice of final performance metric affect all methods? Choosing final performance to be 98% of maximum performance could have a major effect on each method. Because the proposed method is non-myopic, I would expect that it performs well when this value is large but would perform poorly with a smaller percentage of maximum performance.
 - Is the proposed method sensitive to number of samples used to compute the state?
 - What does figure 1 show? Are the same 30 samples used for all three subfigures? Perhaps this would more interpretable if, instead of showing the predicted class, this figure showed the prediction error.
Minor nitpicks (did not influence decision):
 - The datasets are 1-based indexed sometimes and 0-based indexed sometimes, even with disparities within a single paragraph.
 - Figure 1 appears a long time before it is discussed, which made it difficult to understand what was going on.


Many deep reinforcement learning algorithms contain inductive biases that sculpt the agent's objective and its interface to the environment. These inductive biases can take many forms, including domain knowledge and pretuned hyperparameters. In general, there is a trade-off between generality and performance when we use such biases. Stronger biases can lead to faster learning, but weaker biases can potentially lead to more general algorithms that work on a wider class of problems. This trade-off is relevant because these inductive biases are not free; substantial effort may be required to obtain relevant domain knowledge or to tune hyper-parameters effectively. In this paper, we re-examine several domain-specific components that modify the agent's objective and environmental interface. We investigated whether the performance deteriorates when all these fixed components are replaced with adaptive solutions from the literature. In our experiments, performance sometimes decreased with the adaptive components, as one might expect when comparing to components crafted for the domain, but sometimes the adaptive components performed better. We then investigated the main benefit of having fewer domain-specific components, by comparing the learning performance of the two systems on a different set of continuous control problems, without additional tuning of either system. As hypothesized, the system with adaptive components performed better on many of the tasks.
The deep reinforcement learning (RL) community has demonstrated that well-tuned deep RL algorithms can master a wide range of challenging tasks. Human-level performance has been approached or surpassed on board-games such as Go and Chess (Silver et al., 2017), video-games such as Atari (Mnih et al., 2015;Hessel et al., 2018a;Horgan et al., 2018), and custom 3D navigation tasks (Johnson et al., 2016;Kempka et al., 2016;Beattie et al., 2016;Jaderberg et al., 2016;Espeholt et al., 2018). These results are a testament to the generality of the overall approach. At times, however, the excitement for the constant stream of new domains being mastered by suitable RL algorithms may have over-shadowed the dependency on inductive biases of these agents, and the amount of tuning that is often required for these to perform effectively in new domains. A clear example of the benefits of generality is the AlphaZero algorithm (Silver et al., 2017). AlphaZero differs from the earlier AlphaGo algorithm  by removing all dependencies on Go-specific inductive biases and human data. After removing these biases, AlphaZero could not just surpass human performance on Go but also learn to play Chess and Shogi.
In general, there is a trade off between generality and performance when we inject inductive biases into our algorithms. Inductive biases take many forms, including domain knowledge and pretuned learning parameters. If applied carefully, such biases can lead to faster and better learning. On the other hand, fewer biases can potentially lead to more general algorithms that work out of the box on a wider class of problems. Crucially, most inductive biases are not free: for instance, substantial effort can be required to attain the relevant domain knowledge or pretune parameters. This cost is often hidden-for instance, one might use hyperparameters established as good in prior work on the same domain, without knowing how much data or time was spent optimising these, nor how specific the settings are to the given domain. Systematic studies about the impact of different inductive biases are rare and the generality of these different biases is often unclear. Another consideration is that inductive biases may mask the generality of other parts of the system as a whole; if a learning algorithm tuned for a specific domain does not generalize out of the box to a new domain, it can be unclear whether it is due to the having the wrong inductive biases or whether the underpinning learning algorithm is lacking something important.
In this paper, we consider several commonly used domain heuristics, and investigate if (and by how much) performance deteriorates when we replace these with more general adaptive components, and we assess the impact of such replacements on the generality of the agent. We consider two broad ways of injecting inductive biases in RL agents: 1) sculpting the agent's objective (e.g., clipping and discounting rewards), 2) sculpting the agent-environment interface (e.g., repeating each selected action a hard-coded fixed number of times, or crafting of the learning agent's observation). Our contributions show that all the carefully crafted heuristics commonly used in Atari, and often considered essential for good performance on this domain, can be safely replaced with adaptive components, while preserving competitive performance across the benchmark. Furthermore, we show that this results in increased generality for an actor critic agent; the resulting fully adaptive system can be applied with no additional tuning on a separate suite of continuous control tasks that have different environmental interfaces. Here the adaptive system achieved much higher performance than a comparable system using the Atari tuned heuristics, and also higher performance than an actor-critic agent that was tuned for this benchmark (Tassa et al., 2018).

This paper focuses on deep reinforcement learning methods and discusses the presence of inductive biases in the existingRL algorithm. Specifically, they discuss biases that take the form of domain knowledge or hyper-parameter tuning. The authors state that such biases rise the tradeoff between generality and performance wherein strong biases can lead to efficient performance but deteriorate generalization across domains. Further, it motivates that most inductive biases has a cost associated to it and hence it is important to study and analyze the effect of such biases.
To support their insights, the authors investigate the performance of well known actor-critic model in the Atari environment after replacing domain specific heuristics with the adaptive components. The author considers two ways of injecting biases: i) sculpting agents objective and ii) sculpting agent's environment. They show empirical evidence that replacing carefully designed heuristics to induce biases with more adaptive counterparts preserves performance and generalizes without additional fine tuning.
The paper focuses on an important concept and problem of inductive biases in deep reinforcement learning techniques.
Analysis of such biases and methods to use them judiciously is an interesting future direction. The paper covers a lot of related work in terms of various algorithms and corresponding biases.
However, this paper only discusses such concepts at high level and provides short empirical evidences in a single environment to support their arguments. Further, both the heuristics used in practice and the adaptive counterparts that the paper uses to replace those heuristics are all available in existing approaches and there is no novel contribution in that direction too.
Finally, the adaptive methods based on parallel environment and RNNs have several limitation, as per author's own admission.
Overall, the paper does not have any novel technical contributions or theoretical analysis on the effect of such inductive biases which makes it very weak. Further, there is nothing surprising about the author's claims and many of the outcomes from the analysis are expected. The authors are recommended to consider this task more rigorously and provide stronger and concrete analysis on the effects of inductive biases on variety of algorithms and variety of environments.