Published as a conference paper at ICLR 2018 LEARNING TO COUNT OBJECTS IN NATURAL IMAGES FOR VISUAL QUESTION ANSWERING
Visual Question Answering (VQA) models have struggled with counting objects in natural images so far. We identify a fundamental problem due to soft attention in these models as a cause. To circumvent this problem, we propose a neural network component that allows robust counting from object proposals. Experiments on a toy task show the effectiveness of this component and we obtain state-of-theart accuracy on the number category of the VQA v2 dataset without negatively affecting other categories, even outperforming ensemble models with our single model. On a difficult balanced pair metric, the component gives a substantial improvement in counting over a strong baseline by 6.6%.

Summary:
- This paper proposes a hand-designed network architecture on a graph of object proposals to perform soft non-maximum suppression to get object count.
Contribution:
- This paper proposes a new object counting module which operates on a graph of object proposals.
Clarity:
- The paper is well written and clarity is good. Figure 2 & 3 helps the readers understand the core algorithm.
Pros:
- De-duplication modules of inter and intra object edges are interesting.
- The proposed method improves the baseline by 5% on counting questions.
Cons:
- The proposed model is pretty hand-crafted. I would recommend the authors to use something more general, like graph convolutional neural networks (Kipf & Welling, 2017) or graph gated neural networks (Li et al., 2016).
- One major bottleneck of the model is that the proposals are not jointly finetuned. So if the proposals are missing a single object, this cannot really be counted. In short, if the proposals don’t have 100% recall, then the model is then trained with a biased loss function which asks it to count all the objects even if some are already missing from the proposals. The paper didn’t study what is the recall of the proposals and how sensitive the threshold is.
- The paper doesn’t study a simple baseline that just does NMS on the proposal domain.
- The paper doesn’t compare experiment numbers with (Chattopadhyay et al., 2017).
- The proposed algorithm doesn’t handle symmetry breaking when two edges are equally confident (in 4.2.2 it basically scales down both edges). This is similar to a density map approach and the problem is that the model doesn’t develop a notion of instance.
- Compared to (Zhou et al., 2017), the proposed model does not improve much on the counting questions.
- Since the authors have mentioned in the related work, it would also be more convincing if they show experimental results on CL
Conclusion:
- I feel that the motivation is good, but the proposed model is too hand-crafted. Also, key experiments are missing: 1) NMS baseline 2) Comparison with VQA counting work  (Chattopadhyay et al., 2017). Therefore I recommend reject.
References:
- Kipf, T.N., Welling, M., Semi-Supervised Classification with Graph Convolutional Networks. ICLR 2017.
- Li, Y., Tarlow, D., Brockschmidt, M., Zemel, R. Gated Graph Sequence Neural Networks. ICLR 2016.
Update:
Thank you for the rebuttal. The paper is revised and I saw NMS baseline is added. I understood the reason not to compare with certain related work. The rebuttal is convincing and I decided to increase my rating, because adding the proposed counting module achieve 5% increase in counting accuracy. However, I am a little worried that the proposed model may be hard to reproduce due to its complexity and therefore choose to give a 6.


We present Tensor-Train RNN (TT-RNN), a novel family of neural sequence architectures for multivariate forecasting in environments with nonlinear dynamics. Long-term forecasting in such systems is highly challenging, since there exist long-term temporal dependencies, higher-order correlations and sensitivity to error propagation. Our proposed tensor recurrent architecture addresses these issues by learning the nonlinear dynamics directly using higher order moments and high-order state transition functions. Furthermore, we decompose the higherorder structure using the tensor-train (TT) decomposition to reduce the number of parameters while preserving the model performance. We theoretically establish the approximation properties of Tensor-Train RNNs for general sequence inputs, and such guarantees are not available for usual RNNs. We also demonstrate significant long-term prediction improvements over general RNN and LSTM architectures on a range of simulated environments with nonlinear dynamics, as well on real-world climate and traffic data.

The paper proposes Tensor-Train RNN and Tensor-Train LSTM (TT-RNN/TLSTM), a RNN/LSTM architecture whose hidden unit at time t h_t is computed from the tensor-vector product between a tensor of weights and a concatenation of hidden units from the previous L time steps. The motivation is to incorporate previous hidden states and high-order correlations among them to better predict long-term temporal dependencies for seq2seq problems. To address the issue of the number of parameters growing exponentially in the rank of the tensor, the model uses a low rank decomposition called the ‘tensor-train decomposition’ to make the number of parameters linear in the rank. Some theoretical analysis on the number of hidden units required for a given estimation error, and experimental results have been provided for synthetic and real sequential data.
First of all, the presentation of the method in section 2.1 is confusing and there seem to be various ambiguities in the notation that harms understanding of the method. The tensor-vector product in equation (6) appears problematic. The notation that I think is standard is as follows: given a tensor W \in R^{n_1 \times … \times n_P} and vectors v_p \in R^{n_p}, the tensor-vector product W \times_{p=1}^P v_p = vec(W) \otimes_{p=1}^P v_p = \sum{i_1,...,i_P} \prod_{p=1}^P v_{p,i_p}. So I’m guessing you want to get rid of the \otimes signs (the kronecker products) in (6) or you want to remove the summation and write W \times_{p=1}^P s_{t-1}. Also \alpha that appears in (6) is never defined. Is it another index? This is confusing because you say W is P-dimensional but have P+1 indices for it including alpha (W_{\alpha i_1 … i_p}). Moreover the dimensionality of W^{hx} x_t in (6) is R^H judging from the notation in page 2, but isn’t the tensor-vector product a scalar? Also am I correct in thinking that s_{t-1} should be [1, h_{t-1}^T, …, h_{t-L}^T], i.e. a vector of length LH+1 rather than a matrix? The notation from page 2 implies that you are using column vectors, so the definition of s_{t-1} makes it appear as an (L+1) by H matrix, which could make the reader interpret s_{t-1;i_1} in (6) as vectors instead of scalars (this is reinforced by the kronecker product between these s_{t-1;i_p}). I had to work this out from the number of parameters (HL+1)^P in section 2.2. The diagram of s_{t-1} in Figure 3 is also confusing, because it isn’t obvious that the unlabelled grey bars are copies of s_{t-1}. Also I notice that the name ‘Tensor Train RNN/LSTM’ has been used in Yang et al, 2017. You probably want to avoid using the same name since the models are different. It would be nice if you could explain in a bit more detail about how they are different in the related work section.
Assuming I have understood the method correctly, the idea of using tensor products to incorporate higher order interactions between the hidden states at different times appears sensible. From the theoretical analysis, you claim that 1) smoother f is easier to approximate, and 2) polynomial interactions are more efficient than linear ones. The first point seems fairly self-explanatory and doesn’t seem to require a proof. The second point isn’t so convincing because you have two additive terms on the right hand side of the inequality in Theorem 3.1 (btw I’m guessing you want the inequality to be the other way round): the first term is independent of p, and the second decreases exponentially with p. Your second point would only hold if this first term is reasonably small, but this doesn’t seem obvious to me.
Regarding the experiments, I’m sceptical as to whether a grid search over hyperparameters for TLSTM vs grid search over the same hyperparameters for (M)LSTM provides a fair comparison. You probably want to compare the models given the same number of parameters, since given the same state size, TLSTM will have many more parameters than (M)LSTM. A plot of x-axis: # parameters, y-axis: average RMSE at convergence would be informative. Moreover for figure 8, you probably want to control the time taken for training instead of just comparing validation loss at the same number of steps. I imagine the best performing TLSTM model will have many more parameters and hence take much longer to train than the best performing LSTM model.
Moreover, it seems as though the increased prediction accuracy from LSTM is marginal considering you have 3 more hyperparameters to tune (L,S,P - what was the value of P used for the experiments?) and that tuning them is important to prevent overfitting.
I’m also curious as to how TLSTM compares to hierarchical RNN approaches for modelling long-term dependencies. It will be interesting to compare against models like Stacked LSTM (Graves, 2013), Grid LSTM (Kalchbrenner, 2015) and HM LSTM (Chung, 2017). These models have mostly been evaluated on text, but I don’t see any reason they can’t be extended to sequential forecasting on time series data. Also regularisation techniques such as batch-norm for LSTMs (Cooijmans et al, 2016) and layer-norm (Ba et al, 2016) seem to help a lot for increasing prediction accuracy. Did you investigate these techniques to control overfitting?
Other minor comments on presentation:
For figure 6, the legends are inconsistent with the caption. Also you might want to overlay predictions on top of the ground truth for better comparison and also to save space.
Overall, I think there are vast scopes for improvement in presentation and comparisons with other methods, and hence find the paper not yet ready for publication.


We note that common implementations of adaptive gradient algorithms, such as Adam, limit the potential benefit of weight decay regularization, because the weights do not decay multiplicatively (as would be expected for standard weight decay) but by an additive constant factor. We propose a simple way to resolve this issue by decoupling weight decay and the optimization steps taken w.r.t. the loss function. We provide empirical evidence that our proposed modification (i) decouples the optimal choice of weight decay factor from the setting of the learning rate for both standard SGD and Adam, and (ii) substantially improves Adam's generalization performance, allowing it to compete with SGD with momentum on image classification datasets (on which it was previously typically outperformed by the latter). We also demonstrate that longer optimization runs require smaller weight decay values for optimal results and introduce a normalized variant of weight decay to reduce this dependence. Finally, we propose a version of Adam with warm restarts (AdamWR) that has strong anytime performance while achieving state-ofthe-art results on CIFAR-10 and ImageNet32x32. Our source code will become available after the review process.

At the heart of the paper, there is a single idea: to decouple the weight decay from the number of steps taken by the optimization process (the paragraph at the end of page 2 is the key to the paper). This is an important and largely overlooked area of implementation and most off-the-shelf optimization algorithms, unfortunately, miss this point, too. I think that the proposed implementation should be taken seriously, especially in conjunction with the discussion that has been carried out with the work of Wilson et al., 2017 (https://arxiv.org/abs/1705.08292).
The introduction does a decent job explaining why it is necessary to pay attention to the norm of the weights as the training progresses within its scope. However, I would like to add a couple more points to the discussion:
- "Optimal weight decay is a function (among other things) of the total number of epochs / batch passes." in principle, it is a function of weight updates. Clearly, it depends on the way the decay process is scheduled. However, there is a bad habit in DL where time is scaled by the number of epochs rather than the number of weight updates which sometimes lead to misleading plots (for instance, when comparing two algorithms with different batch sizes).
- Another ICLR 2018 submission has an interesting take on the norm of the weights and the algorithm (https://openreview.net/forum?id=HkmaTz-0W&noteId=HkmaTz-0W). Figure 3 shows the histograms of SGD/ADAM with and without WD (the *un-fixed* version), and it clearly shows how the landscape appear misleadingly different when one doesn't pay attention to the weight distribution in visualizations.
- In figure 2, it appears that the training process has three phases, an initial decay, a steady progress, and a final decay that is more pronounced in AdamW. This final decay also correlates with the better test error of the proposed method. This third part also seems to correspond to the difference between Adam and AdamW through the way they branch out after following similar curves. One wonders what causes this branching and whether the key the desired effects are observed at the bottom of the landscape.
- The paper concludes with "Advani & Saxe (2017) analytically showed that in the limited data regime of deep networks the presence of eigenvalues that are zero forms a frozen subspace in which no learning occurs and thus smaller (e.g., zero) initial weight norms should be used to achieve best generalization results." Related to this there is another ICLR 2018 submission (https://openreview.net/forum?id=rJrTwxbCb), figure 1 shows that the eigenvalues of the Hessian of the loss have zero forms at the bottom of the landscape, not at the beginning. Back to the previous point, maybe that discussion should focus on the second and third phases of the training, not the beginning.
- Finally, it would also be interesting to discuss the relation of the behavior of the weights at the last parts of the training and its connection to pruning.
I'm aware that one can easily go beyond the scope of the paper by adding more material. Therefore, it is not completely reasonable to expect all such possible discussions to take place at once. The paper as it stands is reasonably self-contained and to the point. Just a minor last point that is irrelevant to the content of the work: The slash punctuation mark that is used to indicate 'or' should be used without spaces as in 'epochs/batch'.
Edit: Thanks very much for the updates and refinements. I stand by my original score and would like to indicate my support for this style of empirical work in scientific conferences.


Published as a conference paper at ICLR 2018 MEMORY-BASED PARAMETER ADAPTATION
Deep neural networks have excelled on a wide range of problems, from vision to language and game playing. Neural networks very gradually incorporate information into weights as they process data, requiring very low learning rates. If the training distribution shifts, the network is slow to adapt, and when it does adapt, it typically performs badly on the training distribution before the shift. Our method, Memory-based Parameter Adaptation, stores examples in memory and then uses a context-based lookup to directly modify the weights of a neural network. Much higher learning rates can be used for this local adaptation, reneging the need for many iterations over similar data before good predictions can be made. As our method is memory-based, it alleviates several shortcomings of neural networks, such as catastrophic forgetting, fast, stable acquisition of new knowledge, learning with an imbalanced class labels, and fast learning during evaluation. We demonstrate this on a range of supervised tasks: large-scale image classification and language modelling.

This article introduces a new method to improve neural network performances on tasks ranging from continual learning (non-stationary target distribution, appearance of new classes, adaptation to new tasks, etc) to better handling of class imbalance, via a hybrid architecture between nearest neighbours and neural net.
After an introduction summarizing their goal, the authors introduce their Model-based parameter adaptation: this hybrid architecture enriches classical deep architectures with a non-parametric “episodic” memory, which is filled at training time with (possibly learned) encodings of training examples and then polled at inference time to refine the neural network parameters with a few steps of gradient in a direction determined by the closest neighbours in memory to the input being processed.  The authors justify this inference-time SGD update with three different interpretations: one linked in Maximum A Posteriori optimization, another to Elastic Weight Regularisation (the current state of the art in continual learning), and one generalising attention mechanisms (although to be honest that later was more elusive to this reviewer). The mandatory literature review on the abundant recent uses of memory in neural networks is then followed by experiments on continual learning tasks involving permuted MNIST tasks, ImageNET incremental inclusion of classes, ImageNet unbalanced, and two language modeling tasks.
This is an overall very interesting idea, which has the merit of being rather simple in its execution and can be combined with many other methods: it is fully compatible with any optimiser (e.g. ADAM) and can be tacked on top of EWC (which the authors do). The justification is clear, the examples reasonably thorough. It is a very solid paper, which this reviewer believes to be of real interest to the ICLR community.
The following important clarifications from the authors could make it even better:
*  Algorithm 1 in its current form seems to imply an infinite memory, which the experiments make clear is not the case. Therefore: how does the algorithm decide what entries to discard when the memory fills up?
* In most non-trivial settings, the parameter $gamma$ of the encoding is learned, and therefore older entries in the memory lose any ability to be compared to more recent encodings. How do the authors handle this obsolescence of the memory, other than the trivial scheme of relying on KNN to only match recent entries?
* Because gamma needs to be “recent”, this means “theta” is also recent: could the authors give a good intuition on how the two sets of parameters can evolve at different enough timescales to really make the episodic memory relevant? Is it anything else than relying on the fact that the lower levels of a neural net converge before the upper levels?
* Table 1:  could the authors explain why the pre-trained Parametric (and then Mixture) models have the best  AUC in the low-data regime, whereas MbPA was designed very much to be superior in such regimes?
* Paragraph below equation (5), page 3: why not including the regularisation term, whereas the authors just went to great pain to explain it? Rationale? Not including it is also akin to using an improper non-information prior on theta^x independent of theta, which is quite a strong choice to be made “by default”.
* The extra complexity of choosing the learning rate alpha_M and the number of  MpAB steps is worrying this reviewer somewhat. In practice, in Section 4.1the authors explain using grid search to tune the parameters. Is this reviewer correct in understanding that this search is done across all tasks, as opposed to only the first task? And if so, doesn’t this grid search introduce an information leak by bringing information from the whole pre-determined set of task, therefore undermining the very “continuous learning” aim? How do the algorithm performs if the grid search is done only on the first task?
* Figure 3:  the text could clarify that the accuracy is measured across all tasks seen so far. It would be interesting to add a figure (in the Appendix) showing the evolution of the accuracy *per task*, not just the aggregated accuracy.
* In the related works linking neural networks to encoded episodic memory, the authors might want to include the stream of research on HMAX of Anselmi et al 2014 (https://arxiv.org/pdf/1311.4158.pdf) , Leibo et al 2015 (https://arxiv.org/abs/1512.08457), and Blundell et al 2016 (https://arxiv.org/pdf/1606.04460.pdf ).
Minor typos:
* Figure 4: the title of the key says “New/Old” but then the lines read, in order, “Old” then “New” -- it would be nicer to have them in the same order.
* Section 5: missing period between "ephemeral gradient modifications" and "Further".
* Section 4.2, parenthesis should be "perform well across all 1000 classes", not "all 100 classes".

With the above clarifications, this article could become a very remarked contribution.


During the last years, a remarkable breakthrough has been made in AI domain thanks to artificial deep neural networks that achieved a great success in many machine learning tasks in computer vision, natural language processing, speech recognition, malware detection and so on. However, they are highly vulnerable to easily crafted adversarial examples. Many investigations have pointed out this fact and different approaches have been proposed to generate attacks while adding a limited perturbation to the original data. The most robust known method so far is the so called C&W attack [1]. Nonetheless, a countermeasure known as feature squeezing coupled with ensemble defense showed that most of these attacks can be destroyed [6]. In this paper, we present a new method we call Centered Initial Attack (CIA) whose advantage is twofold : first, it insures by construction the maximum perturbation to be smaller than a threshold fixed beforehand, without the clipping process that degrades the quality of attacks. Second, it is robust against recently introduced defenses such as feature squeezing, JPEG encoding and even against a voting ensemble of defenses. While its application is not limited to images, we illustrate this using five of the current best classifiers on ImageNet dataset among which two are adversarialy retrained on purpose to be robust against attacks. With a fixed maximum perturbation of only 1.5% on any pixel, around 80% of attacks (targeted) fool the voting ensemble defense and nearly 100% when the perturbation is only 6%. While this shows how it is difficult to defend against CIA attacks, the last section of the paper gives some guidelines to limit their impact.

This paper presents a reparametrization of the perturbation applied to features in adversarial examples based attacks. It tests this attack variation on against Inception-family classifiers on ImageNet. It shows some experimental robustness to JPEG encoding defense.
Specifically about the method: Instead of perturbating a feature x_i by delta_i, as in other attacks, with delta_i in range [-Delta_i, Delta_i], they propose to perturbate x_i^*, which is recentered in the domain of x_i through a heuristic ((x_i ± Delta_i + domain boundary that would be clipped)/2), and have a similar heuristic for computing a Delta_i^*. Instead of perturbating x_i^* directly by delta_i, they compute the perturbed x by x_i^* + Delta_i^* * g(r_i), so they follow the gradient of loss to misclassify w.r.t. r (instead of delta).
+/-:
+ The presentation of the method is clear.
+ ImageNet is a good dataset to benchmark on.
- (!) The (ensemble) white-box attack is effective but the results are not compared to anything else, e.g. it could be compared to (vanilla) FGSM nor C&W.
- The other attack demonstrated is actually a grey-box attack, as 4 out of the 5 classifiers are known, they are attacking the 5th, but in particular all the 5 classifiers are Inception-family models.
- The experimental section is a bit sloppy at times (e.g. enumerating more than what is actually done, starting at 3.1.1.).
- The results on their JPEG approximation scheme seem too explorative (early in their development) to be properly compared.
I think that the paper need some more work, in particular to make more convincing experiments that the benefit lies in CIA (baselines comparison), and that it really is robust across these defenses shown in the paper.


Published as a conference paper at ICLR 2018 STOCHASTIC ACTIVATION PRUNING FOR ROBUST ADVERSARIAL DEFENSE
Neural networks are known to be vulnerable to adversarial examples. Carefully chosen perturbations to real images, while imperceptible to humans, induce misclassification and threaten the reliability of deep learning systems in the wild. To guard against adversarial examples, we take inspiration from game theory and cast the problem as a minimax zero-sum game between the adversary and the model. In general, for such games, the optimal strategy for both players requires a stochastic policy, also known as a mixed strategy. In this light, we propose Stochastic Activation Pruning (SAP), a mixed strategy for adversarial defense. SAP prunes a random subset of activations (preferentially pruning those with smaller magnitude) and scales up the survivors to compensate. We can apply SAP to pretrained networks, including adversarially trained models, without fine-tuning, providing robustness against adversarial examples. Experiments demonstrate that SAP confers robustness against attacks, increasing accuracy and preserving calibration.

This paper investigates a new approach to prevent a given classifier from adversarial examples. The most important contribution is that the proposed algorithm can be applied post-hoc to already trained networks. Hence, the proposed algorithm (Stochastic Activation Pruning) can be combined with algorithms which prevent from adversarial examples during the training.
The proposed algorithm is clearly described. However there are issues in the presentation.
In section 2-3, the problem setting is not suitably introduced.
In particular one sentence that can be misleading:
“Given a classifier, one common way to generate an adversarial example is to perturb the input in direction of the gradient…”
You should explain that given a classifier with stochastic output, the optimal way to generate an adversarial example is to perturb the input proportionally to the gradient. The practical way in which the adversarial examples are generated is not known to the player. An adversary could choose any policy. The only thing the player knows is the best adversarial policy.
In section 4, I do not understand why the adversary uses only the sign and not also the value of the estimated gradient. Does it come from a high variance? If it is the case, you should explain that the optimal policy of the adversary is approximated by “fast gradient sign method”.
In comparison to dropout algorithm, SAP shows improvements of accuracy against adversarial examples. SAP does not perform as well as adversarial training, but SAP could be used with a trained network.
Overall, this paper presents a practical method to prevent a classifier from adversarial examples, which can be applied in addition to adversarial training. The presentation could be improved.


As machine learning becomes ubiquitous, deployed systems need to be as accurate as they can. As a result, machine learning service providers have a surging need for useful, additional training data that benefits training, without giving up all the details about the trained program. At the same time, data owners would like to trade their data for its value, without having to first give away the data itself before receiving compensation. It is difficult for data providers and model providers to agree on a fair price without first revealing the data or the trained model to the other side. Escrow systems only complicate this further, adding an additional layer of trust required of both parties. Currently, data owners and model owners don't have a fair pricing system that eliminates the need to trust a third party and training the model on the data, which 1) takes a long time to complete, 2) does not guarantee that useful data is paid valuably and that useless data isn't, without trusting in the third party with both the model and the data. Existing improvements to secure the transaction focus heavily on encrypting or approximating the data, such as training on encrypted data, and variants of federated learning. As powerful as the methods appear to be, we show them to be impractical in our use case with real world assumptions for preserving privacy for the data owners when facing black-box models. Thus, a fair pricing scheme that does not rely on secure data encryption and obfuscation is needed before the exchange of data. This paper proposes a novel method for fair pricing using data-model efficacy techniques such as influence functions, model extraction, and model compression methods, thus enabling secure data transactions. We successfully show that without running the data through the model, one can approximate the value of the data; that is, if the data turns out redundant, the pricing is minimal, and if the data leads to proper improvement, its value is properly assessed, without placing strong assumptions on the nature of the model. Future work will be focused on establishing a system with stronger transactional security against adversarial attacks that will reveal details about the model or the data to the other party.

Summary
The paper addresses the issues of fair pricing and secure transactions between model and data providers in the context of machine learning real-world application.
Major
The paper addresses an important issue regarding the real-world application of machine learning, that is, the transactions between data and model provider and the associated aspects of fairness, pricing, privacy, and security.
The originality and significance of the work reported in this paper are difficult to comprehend. This is largely due to the lack of clarity, in general, and the lack of distinction between what is known and what is proposed. I failed to find any clear description of the proposed approach and any evaluation of the main idea.
Most of the discussions in the paper are difficult to follow due to that many of the statements are vague or unclear. There are some examples of this vagueness illustrated under “minor issues”. Together, the many minor issues contribute to a major communication issue, which significantly reduces readability of the paper. A majority of the references included in the reference section lack some or all of the required meta data.
In my view, the paper is out of scope for ICLR. Neither the CFP overview nor the (non-exhaustive) list of relevant topics suggest otherwise. In very general terms, the paper could of course be characterised as dealing with machine learning implementation/platform/application but the issues discussed are more connected to privacy, security, fair transactions, and pricing.
In summary; although there is no universal rule on how to structure research papers, a more traditional structure (introduction, aim & scope, background, related work, method, results, analysis, conclusions & future work) would most certainly have benefitted the paper through improved clarity and readability. Although some interesting works on adversarial learning, federated learning, and privace-preserving training are cited in the paper, the review and use of these references did not contribute to a better understanding of the topic or the significance of the contribution in this paper. I was unable to find any support in the paper for the strong general result stated in the abstract (“We successfully show that without running the data through the model, one can approximate the value of the data”).
Minor issues (examples)
- “Models trained only a small scale of data” (missing word)
- “to prevent useful data from not being paid” (unclear meaning)
- “while the company may decline reciprocating gifts such as academic collaboration, while using the data for some other service in the future” (unclear meaning)
- “since any data given up is given up ” (unclear meaning)
- “a user of a centralized service who has given up their data will have trouble telling if their data exchange was fair at all (even if their evaluation was purely psychological)” (unclear meaning)
- “For a generally deployed model, it can take any form. Designing a transaction strategy for each one can be time-consuming and difficult to reason about” (unclear meaning)
- “(et al., 2017)” (unknown reference)
- “Osbert Bastani, Carolyn Kim, and Hamsa Bastani. Interpreting blackbox models via model extraction, 2017” (incomplete reference data)
- “Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding, 2015.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network, 2015.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions, 2017.” (Incomplete reference data)
- “H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agera y Arcas. Communication-efficient learning of deep networks from decentralized data. 2016.” (Incomplete reference data)
- “et al. Richard Craid.” (Incorrect author reference style)
- “Ryo Yonetani, Vishnu Naresh Boddeti, Kris M. Kitani, and Yoichi Sato. Privacy-preserving visual learning using doubly permuted homomorphic encryption, 2017.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization, 2016.” (Incomplete reference data)


Online healthcare services can provide the general public with ubiquitous access to medical knowledge and reduce the information access cost for both individuals and societies. To promote these benefits, it is desired to effectively expand the scale of high-quality yet novel relational medical entity pairs that embody rich medical knowledge in a structured form. To fulfill this goal, we introduce a generative model called Conditional Relationship Variational Autoencoder (CRVAE), which can discover meaningful and novel relational medical entity pairs without the requirement of additional external knowledge. Rather than discriminatively identifying the relationship between two given medical entities in a free-text corpus, we directly model and understand medical relationships from diversely expressed medical entity pairs. The proposed model introduces the generative modeling capacity of variational autoencoder to entity pairs, and has the ability to discover new relational medical entity pairs solely based on the existing entity pairs. Beside entity pairs, relationship-enhanced entity representations are obtained as another appealing benefit of the proposed method. Both quantitative and qualitative evaluations on real-world medical datasets demonstrate the effectiveness of the proposed method in generating relational medical entity pairs that are meaningful and novel.

In the medical context, this paper describes the classic problem of "knowledge base completion" from structured data only (no text).  The authors argue for the advantages of a generative VAE approach (but without being convincing).  They do not cite the extensive literature on KB completion.  They present experimental results on their own data set, evaluating only against simpler baselines of their own VAE approach, not the pre-existing KB methods.
The authors seem unaware of a large literature on "knowledge base completion."  E.g. [Bordes, Weston, Collobert, Bengio, AAAI, 2011],  [Socher et al 2013 NIPS], [Wang, Wang, Guo 2015 IJCAI], [Gardner, Mitchell 2015 EMNLP], [Lin, Liu, Sun, Liu, Zhu AAAI 2015], [Neelakantan, Roth, McCallum 2015],
The paper claims that operating on pre-structured data only (without using text) is an advantage.  I don't find the argument convincing.  There are many methods that can operate on pre-structured data only, but also have the ability to incorporate text data when available, e.g. "universal schema" [Riedel et al, 2014].
The paper claims that "discriminative approaches" need to iterate over all possible entity pairs to make predictions.  In their generative approach they say they find outputs by "nearest neighbor search."  But the same efficient search is possible in many of the classic "discriminatively-trained" KB completion models also.
It is admirable that the authors use an interesting (and to my knowledge novel) data set.  But the method should also be evaluated on multiple now-standard data sets, such as FB15K-237 or NELL-995.  The method is evaluated only against their own VAE-based alternatives.  It should be evaluated against multiple other standard KB completion methods from the literature, such as Jason Weston's Trans-E, Richard Socher's Tensor Neural Nets, and Neelakantan's RNNs.


discovering-order-unordered
The assumption that data samples are independently identically distributed is the backbone of many learning algorithms. Nevertheless, datasets often exhibit rich structures in practice, and we argue that there exist some unknown orders within the data instances. Aiming to find such orders, we introduce a novel Generative Markov Network (GMN) which we use to extract the order of data instances automatically. Specifically, we assume that the instances are sampled from a Markov chain. Our goal is to learn the transitional operator of the chain as well as the generation order by maximizing the generation probability under all possible data permutations. One of our key ideas is to use neural networks as a soft lookup table for approximating the possibly huge, but discrete transition matrix. This strategy allows us to amortize the space complexity with a single model and make the transitional operator generalizable to unseen instances. To ensure the learned Markov chain is ergodic, we propose a greedy batch-wise permutation scheme that allows fast training. Empirically, we evaluate the learned Markov chain by showing that GMNs are able to discover orders among data instances and also perform comparably well to state-of-the-art methods on the one-shot recognition benchmark task.

The authors deal with the problem of implicit ordering in a dataset and the challenge of recovering it, i.e. when given a random dataset with no explicit ordering in the samples, the model is able to recover an ordering. They propose to learn a distance-metric-free model that assumes a Markov chain as the generative mechanism of the data and learns not only the transition matrix but also the optimal ordering of the observations.
> Abstract
“Aiming to find such orders, we introduce a novel Generative Markov Network (GMN) which we use to extract the order of data instances automatically. ”
I am not sure what automatically refers here to. Do the authors mean that the GMN model does not explicitly assume any ordering in the observed dataset? This needs to be better stated here.
“Aiming to find such orders, we introduce a novel Generative Markov Network (GMN) which we use to extract the order of data instances automatically; given an unordered dataset, it outputs the best -most possible- ordering.”
Most of the models assume an explicit ordering in the dataset and use it as an integral modelling assumption. Contrary to that they propose a model where no ordering assumption is made explicitly, but the model itself will recover it if any.
> Introduction
The introduction is fairly well structured and the example of the joint locations in different days helps the reader.
In the last paragraph of page 1, “we argue that … a temporal model can generate it.”, the authors present very good examples where ordered observations (ballerina poses, video frames) can be shuffled and then the proposed model can recover a temporal ordering out of them. What I would like to think also here is about an example where the recovered ordering will also be useful as such. An example where the recovered ordering will increase the importance of the inferred solution would be more interesting..
2. Related work
This whole section is not clear how it relates to the proposed model GMN. Rewriting is strongly suggested.
The authors mention Deep Generative models and One-shot learning methods as related work but the way this section is constructed makes it hard for the reader to see the relation. It is important that first the authors discuss the characteristics of GMN that makes it similar to Deep generative models and the one-shot learning models. They should briefly explain the characteristics of DGN and one-shot learning so that the readers see the relationship.
Also, the authors never mention that the architecture they propose is deep.

Regarding the last paragraph of page 2, “Our approach can be categorised … can be computed efficiently.”:
Not sure why the authors assume that the samples can be sampled from an unmixed chain. An unmixed chain can also result in observing data that do not exhibit the real underlying relationships. Also the authors mention couple of characteristics of the GMN but without really explaining them.  What are the explicit and implicit models [1] … this needs more details.
[1] P. J. Diggle and R. J. Gratton. Monte Carlo methods of inference for implicit statistical models. Journal of the Royal Statistical Society. Series B (Methodological), pages 193–227, 1984.
“Second, prior approaches were proposed based on the notion of denoising models. In other words, their goal was generating high-quality images; on the other hand, we aim at discovering orders in datasets.” —>this bit is confusing. Do the authors mean that prior approaches were considering the observed ordering as part of the model assumptions and were just focusing on the denoising?
3. Generative Markov models
First, I would like to draw the attention of the authors on the terminology they use. The states here are not the latent states usually referred in the literature of Markov chains. The states here are observed and should not be confused with the emissions also usually stated in the corresponding literature. There are as many states as the number of observations and not differentiation is made for ties. All these are based on my understanding of the model.
In  the Equation just before equation (1),  on the left hand side, shouldn’t \pi be after the `;’. It’s an average over the possible \pi.  We cannot  consider the average over \pi when we also want to find the optimal \pi.  The sum doesn’t need to be there. Shouldn’t it just be  max_{\theta, \pi} log P({s_i}^{n}_{i=1}; \pi, \theta) ?
Equation (1), same. The summation over the possible \pi is confusing. It’s an optimisation problem…
page 4, section 3.1: The discussion about the use of Neural Net for the construction of the transition matrix needs expansion. It is unclear how the matrix is constructed. Please add more details. E.g. use of soft-max non-linear transformation so that the output of the Neural Net can be interpreted as the probabilities of jumping to one of the possible states. In this fashion, we map the input (current state) and transform it to the probability gf occupying states at the next time step.
Why this needs expansion: The construction of the transition matrix is the one that actually plays the role of the distance metric in the related models. More specifically, the choice of the non-linear function that outputs the transition probability is crucial; e.g. a smooth function will output comparable transition probabilities to similar inputs (i.e. similar states).
section 3.2:
My concern about averaging over \pi applies on the equations here too.
“However, without further assumption on the structure of the transitional operator..”—> I think the choice of the nonlinear function in the output node of the NN is actually related to the transition matrix and defines the probabilities. It is a confusing statement to make and authors need to discuss more about it. After all, what is the driving force of the inference? This is a problem/task where the observations are considered in a number of different permutations. As such, the ordering is not fixed and the main driving force regarding the best choice of ordering should come from the architecture of the transition matrix; what kind of transitions does the Neural Net architecture favour? Distance free metric but still assumptions are made that favour specific transitions over others.
“At first, Alg. 1 enumerates all the possible states appearing in the first time step. For each of the following steps, it finds the next state by maximizing the transition probability at the current step, i.e., a local search to find the next state. ” —>  local search in the sense that the algorithm chooses as the next state the state with the biggest transition probability (to it) as defined in the Neural Net (transition operator) output? This is a deterministic step, right?
4.1 DISCOVERING ORDERS IN DATASETS
Nice description of the datasets. In the <MSR_SenseCam> the choice of one of the classes needs to be supported.  Why? What do the authors expect to happen if a number of instances from different classes are chosen?
4.1.1 IMPLICIT ORDERS IN DATASETS
The explanation of the inferred orderings for the GMN and Nearest Neighbour model is not clear. In figure 2, what forces the GMN to make distinguishable transitions as opposed to the Nearest neighbour approach that prefers to get stuck to similar states? Is it the transition matrix architecture as defined by the neural network?
>> Figure 10: why use of X here? Why not keep being consistent by using s?
*** DO the authors test the model performance on a ordered dataset (after shuffling it…) ?  Is the model able of recovering the order? **


We introduce a neural network that represents sentences by composing their words according to induced binary parse trees. We use Tree-LSTM as our composition function, applied along a tree structure found by a fully differentiable natural language chart parser. Our model simultaneously optimises both the composition function and the parser, thus eliminating the need for externally-provided parse trees which are normally required for Tree-LSTM. It can therefore be seen as a tree-based RNN that is unsupervised with respect to the parse trees. As it is fully differentiable, our model is easily trained with an off-the-shelf gradient descent method and backpropagation. We demonstrate that it achieves better performance compared to various supervised Tree-LSTM architectures on a textual entailment task and a reverse dictionary task. Finally, we show how performance can be improved with an attention mechanism which fully exploits the parse chart, by attending over all possible subspans of the sentence.

The paper presents a model titled the "unsupervised tree-LSTM," in which the authors mash up a dynamic-programming chart and a recurrent neural network. As far as I can glean, the topology of the neural network is constructed using the chart of a CKY parser. When combining different constituents, an energy function is computed (equation 6) and the resulting energies are passed through a softmax. The architecture achieves impressive results on two tasks: SNLI and the reverse dictionary of Hill et al. (2016).
Overall, I found the paper deeply uninspired. The authors downplay the similarity of their paper to that of Le and Zuidema (2015), which  I did not appreciate. It's true that Le and Zuidema take a parse forest from an existing parser, but it still contains an exponential number of trees, as does the work in here. Note that exposition in Le and Zuidema (2015) discusses the pruned case as well, i.e., a compete parse forest. The authors of this paper simply write "Le and Zuidema (2015) propose a model that takes as input a parse forest from an external parser, in order to deal with uncertainty." I would encourage the authors to revisit Le and Zuidema (2015), especially section 3.2, and consider the technical innovations over the existing work. I believe the primary difference (other using an LSTM instead of a convnet) is to replace max-pooling with softmax-pooling. Do these two architectural changes matter? The experiments offer no empirical comparison. In short, the insight of having an end-to-end differentiable function based on a dynamic-programming chart is pretty common -- the idea is in the air. The authors provide yet another instantiation of such an approach, but this time with an LSTM.
The technical exposition is also relatively poor. The authors could have expressed their network using a clean recursion, following the parse chart, but opted not to, and, instead,  provided a round-about explanation in English. Thus, despite the strong results, I would not like to see this work in the proceedings, due to the lack of originality and poor technical discussion. If the paper were substantially cleaned-up, I would be willing to increase my rating.


Deep reinforcement learning algorithms that estimate state and state-action value functions have been shown to be effective in a variety of challenging domains, including learning control strategies from raw image pixels. However, algorithms that estimate state and state-action value functions typically assume a fully observed state and must compensate for partial or non-Markovian observations by using finite-length frame-history observations or recurrent networks. In this work, we propose a new deep reinforcement learning algorithm based on counterfactual regret minimization that iteratively updates an approximation to a cumulative clipped advantage function and is robust to partially observed state. We demonstrate that on several partially observed reinforcement learning tasks, this new class of algorithms can substantially outperform strong baseline methods: on Pong with single-frame observations, and on the challenging Doom (ViZDoom) and Minecraft (Malmö) first-person navigation benchmarks.

This paper presents Advantage-based Regret Minimization, somewhat similar to advantage actor-critic with REINFORCE.
The main focus of the paper seems to be the motivation/justification of this algorithm with connection to the regret minimization literature (and without Markov assumptions).
The claim that ARM is more robust to partially observable domains is supported by experiments where it outperforms DQN.
There are several things to like about this paper:
- The authors do a good job of reviewing/referencing several papers in the field of "regret minimization" that would probably be of interest to the ICLR community + provide non-obvious connections / summaries of these perspectives.
- The issue of partial observability is good to bring up, rather than simply relying on the MDP framework that is often taken as a given in "deep reinforcement learning".
- The experimental results show that ARM outperforms DQN on a suite of deep RL tasks.
However, there are also some negatives:
- Reviewing so much of the CFR-literature in a short paper means that it ends up feeling a little rushed and confused.
- The ultimate algorithm *seems* like it is really quite similar to other policy gradient methods such as A3C, TRPO etc. At a high enough level, these algorithms can be written the same way... there are undoubtedly some key differences in how they behave, but it's not spelled out to the reader and I think the connections can be missed.
- The experiment/motivation I found most compelling was 4.1 (since it clearly matches the issue of partial observability) but we only see results compared to DQN... it feels like you don't put a compelling case for the non-Markovian benefits of ARM vs other policy gradient methods. Yes A3C and TRPO seem like they perform very poorly compared to ARM... but I'm left wondering how/why?
I feel like this paper is in a difficult position of trying to cover a lot of material/experiments in too short a paper.
A lot of the cited literature was also new to me, so it could be that I'm missing something about why this is so interesting.
However, I came away from this paper quite uncertain about the real benefits/differences of ARM versus other similar policy gradient methods... I also didn't feel the experimental evaluations drove a clear message except "ARM did better than all other methods on these experiments"... I'd want to understand how/why and whether we should expect this universally.
The focus on "regret minimization perspectives" didn't really get me too excited...
Overall I would vote against acceptance for this version.


We propose and evaluate new techniques for compressing and speeding up dense matrix multiplications as found in the fully connected and recurrent layers of neural networks for embedded large vocabulary continuous speech recognition (LVCSR). For compression, we introduce and study a trace norm regularization technique for training low rank factored versions of matrix multiplications. Compared to standard low rank training, we show that our method leads to good accuracy versus number of parameter trade-offs and can be used to speed up training of large models. For speedup, we enable faster inference on ARM processors through new open sourced kernels optimized for small batch sizes, resulting in 3x to 7x speed ups over the widely used gemmlowp library. Beyond LVCSR, we expect our techniques and kernels to be more generally applicable to embedded neural networks with large fully connected or recurrent layers.

The authors propose a strategy for compressing RNN acoustic models in order to deploy them for embedded applications. The technique consists of first training a model by constraining its trace norm, which allows it to be well-approximated by a truncated SVD in a second fine-tuning stage. Overall, I think this is interesting work, but I have a few concerns which I’ve listed below:
1. Section 4, which describes the experiments of compressing server sized acoustic models for embedded recognition seems a bit “disjoint” from the rest of the paper. I had a number of clarification questions spefically on this section:
- Am I correct that the results in this section do not use the trace-norm regularization at all? It would strengthen the paper significantly if the experiments presented on WSJ in the first section were also conducted on the “internal” task with more data.
- How large are the training/test sets used in these experiments (for test sets, number of words, for training sets, amount of data in hours (is this ~10,000hrs), whether any data augmentation such as multi-style training was done, etc.)
- What are the “tier-1” and “tier-2” models in this section? It would also aid readability if the various models were described more clearly in this section, with an emphasis on structure, output targets, what LMs are used, how are the LMs pruned for the embedded-size models, etc. Also, particularly given that the focus is on embedded speech recognition, of which the acoustic model is one part, I would like a few more details on how decoding was done, etc.
- The details in appendix B are interesting, and I think they should really be a part of the main paper. That being said, the results in Section B.5, as the authors mention, are somewhat preliminary, and I think the paper would be much stronger if the authors can re-run these experiments were models are trained to convergence.
- The paper focuses fairly heavily on speech recognition tasks, and I wonder if it would be more suited to a conference on speech recognition.
2. Could the authors comment on the relative training time of the models with the trace-norm regularizer, L2-regularizer and the unconstrained model in terms of convergence time.
3. Clarification question: For the WSJ experiments was the model decoded without an LM? If no LM was used, then the choice of reporting results in terms of only CER is reasonable, but I think it would be good to also report WERs on the WSJ set in either case.
4. Could the authors indicate the range of values of \lambda_{rec} and \lambda_{nonrec} that were examined in the work? Also, on a related note, in Figure 2, does each point correspond to a specific choice of these regularization parameters?
5. Figure 4: For the models in Figure 4, it would be useful to indicate the starting CER of the stage-1 model before stage-2 training to get a sense of how stage-2 training impacts performance.
6. Although the results on the WSJ set are interesting, I would be curious if the same trends and conclusions can be drawn from a larger dataset -- e.g., the internal dataset that results are reported on later in the paper, or on a set like Switchboard. I think these experiments would strengthen the paper.
7. The experiments in Section 3.2.3 were interesting, since they demonstrate that the model can be warm-started from a model that hasn’t fully converged. Could the authors also indicate the CER of the model used for initialization in addition to the final CER after stage-2 training in Figure 5.
8. In Section 4, the authors mention that quantization could be used to compress models further although this is usually degrades WER by 2--4% relative. I think the authors should consider citing previous works which have examined quantization for embedded speech recognition [1], [2]. In particular, note that [2] describes a technique for training with quantized forward passes which results in models that have smaller performance degradation relative to quantization after training.
References:
[1] Vincent Vanhoucke, Andrew Senior, and Mark Mao, “Improving the speed of neural networks on cpus,” in Deep Learning and Unsupervised Feature Learning Workshop, NIPS, 2011.
[2] Raziel Alvarez, Rohit Prabhavalkar, Anton Bakhtin, “On the efficient representation and execution of deep acoustic models,” Proc. of Interspeech, pp. 2746 -- 2750, 2016.
9. Minor comment: The authors use the term “warmstarting” to refer to the process of training NNs by initializing from a previous model. It would be good to clarify this in the text.


A variety of learning objectives have been recently proposed for training generative models. We show that many of them, including InfoGAN, ALI/BiGAN, ALICE, CycleGAN, VAE, β-VAE, adversarial autoencoders, AVB, and InfoVAE, are Lagrangian duals of the same primal optimization problem. This generalization reveals the implicit modeling trade-offs between flexibility and computational requirements being made by these models. Furthermore, we characterize the class of all objectives that can be optimized under certain computational constraints. Finally, we show how this new Lagrangian perspective can explain undesirable behavior of existing methods and provide new principled solutions.

Update after rebuttal
==========
Thanks for your response on my questions. The stated usefulness of the method unfortunately do not answer my worry about the significance. It remains unclear to me how much "real" difference the presented results would make to advance the existing work on generative models. Also, the authors did not promised any major changes in the final version in this direction, which is why I have reduced my score.
I do believe that this work could be useful and should be resubmitted. There are two main things to improve. First, the paper need more work on improving the clarity. Second, more work needs to be added to show that the paper will make a real difference to advance/improve existing methods.
==========
Before rebuttal
==========
This paper proposes an optimization problem whose Lagrangian duals contain many existing objective functions for generative models. Using this framework, the paper tries to generalize the optimization problems by defining computationally-tractable family which can be expressed in terms of existing objective functions.
The paper has interesting elements and the results are original. The main issue is that the significance is unclear. The writing in Section 3 is unclear for me, which further made it challenging to understand the consequences of the theorems presented in that section.
Here is a big-picture question that I would like to know answer for. Do the results of sec 3 help us identify a more useful/computationally tractable model than exiting approaches? Clarification on this will help me evaluate the significance of the paper.
I have three main clarification points. First, what is the importance of T1, T2, and T3 classes defined in Def. 7, i.e., why are these classes useful in solving some problems? Second, is the opposite relationship in Theorem 1, 2, and 3 true as well, e.g., is every linear combination of beta-ELBO and VMI is equivalent to a likelihood-based computable-objective of KL info-encoding family? Is the same true for other theorems?
Third, the objective of section 3 is to show that "only some choices of lambda lead to a dual with a tractable equivalent form". Could you rewrite the theorems so that they truly reflect this, rather than stating something which only indirectly imply the main claim of the paper.
Some small comments:
- Eq. 4. It might help to define MI to remind readers.
- After Eq. 7, please add a proof (may be in the Appendix). It is not that straightforward to see this. Also, I suppose you are saying Eq. 3 but with f from Eq. 4.
- Line after Eq. 8, D_i is "one" of the following... Is it always the same D_i for all i or it could be different? Make this more clear to avoid confusion.
- Last line in Para after Eq. 15, "This neutrality corresponds to the observations made in.." It might be useful to add a line explaining that particular "observation"
- Def. 7, the names did not make much sense to me. You can add a line explaining why this name is chosen.
- Def. 8, the last equation is unclear. Does the first equivalence impy the next one?
- Writing in Sec. 3.3 can be improved. e.g., "all linear operations on log prob." is very unclear, "stated computational constraints" which constraints?


Generative Adversarial Networks (GANs), when trained on large datasets with diverse modes, are known to produce conflated images which do not distinctly belong to any of the modes. We hypothesize that this problem occurs due to the interaction between two facts: (1) For datasets with large variety, it is likely that the modes lie on separate manifolds. (2) The generator (G) is formulated as a continuous function, and the input noise is derived from a connected set, due to which G's output is a connected set. If G covers all modes, then there must be some portion of G's output which connects them. This corresponds to undesirable, conflated images. We develop theoretical arguments to support these intuitions. We propose a novel method to break the second assumption via learnable discontinuities in the latent noise space. Equivalently, it can be viewed as training several generators, thus creating discontinuities in the G function. We also augment the GAN formulation with a classifier C that predicts which noise partition/generator produced the output images, encouraging diversity between each partition/generator. We experiment on MNIST, celebA, STL-10, and a difficult dataset with clearly distinct modes, and show that the noise partitions correspond to different modes of the data distribution, and produce images of superior quality.

Summary:
The paper studies the problem of learning distributions with disconnected support. The paper is very well written, and the analysis is mostly correct, with some important exceptions. However, there are a number of claims that are unverified, and very important baselines are missing. I suggest improving the paper taking into account the following remarks and I will strongly consider improving the score.
Detailed comments:
- The paper is very well written, which is a big plus.
- There are a number of claims in the paper that are not supported by experiments, citations, or a theorem.
- Sections 3.1 - 3.3 can be summarized to "Connected prior + continuous generator => connected support". Thus, to allow for disconnected support, the authors propose to have a discontinuous generator. However to me it seems that a trivial and important attack to this problem is to allow a simple disconnected prior, such as a mixture between uniforms, or at least an approximately disconnected (given the superexponential decay of the gaussian pdf) of a mixture of gaussians, which is very common. The authors fail to mention this obvious alternative, or explore it further, which I think weakens the paper.
- Another standard approach to attacking diverse datasets such as imagenet is adding noise in the intermediate layers of the generator (this was done by EBGAN and the Improved GAN paper by Salimans et al.). It seems to me that this baseline is missing.
- Section 3.4, paragraph 3, "the outputs corresponding to vectors linearly interpolated from z_1 to z_2 show a smooth". Actually, this is known to not perform very well often, indeed the interpolations are done through great circles in z_1 and z_2. See https://www.youtube.com/watch?v=myGAju4L7O8 for example.
- Lemma 1 is correct, but the analysis on the paragraph following is flat out wrong. The fact that a certain z has high density doesn't imply that the sample g_\theta(z) has high density! You're missing the Jacobian term appearing in the change of variables. Indeed, it's common to see neural nets spreading appart regions of high probability to the extent that each individual output point has low density (this is due in its totallity to the fact that ||\nabla_x g_\theta(z)|| can be big.
- Borrowing from the previous comment, the evidence to support result 5 is insufficient. I think the authors have the right intuition, but no evidence or citation is presented to motivate result 5. Indeed, DCGANs are known to have extremely sharp interpolations, suggesting that small jumps in z lead to large jumps in images, thus having the potential to assign low probability to tunnels.
- A citation, experiment or a theorem is missing showing that the K of a generator is small enough in an experiment with separated manifolds. Until that evidence is presented, section 3.5 is anecdotal.
- The second paragraph of section 3.6 is a very astute observation, but again it is necessary to show some evidence to verify this intuition.
- The authors then propose to partition the prior space by training separate first layers for the generator in a maximally discriminative way, and then at inference time just sampling which layer to use uniformly. It's important to note that this has a problem when the underlying separated manifolds in the data are not equiprobable. For example, if we use N = 2 in CelebRoom but we use 30% faces and 70% bedrooms, I would still expect tunneling due to the fact that one of the linear layers has to cover both faces and bedrooms.
- MNIST is known to be a very poor benchmark for image generation, and it should be avoided.
- I fail to see an improvement in quality in CelebA. It's nice to see some minor form on clustering when using generator's prediction, but this has been seen in many other algorithms (e.g. ALI) with much better results long before. I have to say also the official baseline for 64x64 images in wgangp (that I've used several times) gives much better results than the ones presented in this paper https://github.com/igul222/improved_wgan_training/blob/master/gan_64x64.py .
- The experiments in celebRoom are quite nice, and a good result, but we are still missing a detailed analysis for most of the assumptions and improvements claimed in the paper. It's very hard to make very precise claims about the improvements of this algorithm in such a complex setting without having even studied the standard baselines (e.g. noise at every layer of the generator, which has very public and well established code https://github.com/openai/improved-gan/blob/master/imagenet/generator.py).
- I would like to point a lot of tunneling issues can be seen and studied in toy datasets. The authors may want to consider doing targeted experiments to evaluate their assumptions.
=====================
After the rebuttal I've increased my score. The authors did a great job at addressing some of the concerns. I still think there is more room to be done as to justifying the approach, dealing properly with tunneling when we're not in the somewhat artificial case of equiprobable partitions, and primarily at understanding the extent to which tunneling is a problem in current methods. The revision is a step forward in this direction, but still a lot remains to be done. I would like to see simple targeted experiments aimed at testing how much and in what way tunneling is a problem in current methods before I see high dimensional non quantitative experiments.
In the case where the paper gets rejected I would highly recommend the acceptance at the workshop due to the paper raising interesting questions and hinting to a partial solution, even though the paper may not be at a state to be published at a conference venue like ICLR.


Application of deep learning has been successful in various domains such as image recognition, speech recognition and natural language processing. However, the research on its application in graph mining is still in an early stage. Here we present the first generic deep learning approach to the graph link weight prediction problem based on node embeddings. We evaluate this approach with three different node embedding techniques experimentally and compare its performance with two state-of-the-art non deep learning baseline approaches. Our experiment results suggest that this deep learning approach outperforms the baselines by up to 70% depending on the dataset and embedding technique applied. This approach shows that deep learning can be successfully applied to link weight prediction to improve prediction accuracy.

Although this paper aims at an interesting and important task, the reviewer does not feel it is ready to be published.
Below are some detailed comments:
Pros
- Numerous public datasets are used for the experiments
- Good introductions for some of the existing methods.
Cons
- The novelty is limited. The basic idea of the proposed method is to simply concatenate the embeddings of two nodes (via activation separately) from both side of edges, which is straightforward and produces only marginal improvement over existing methods (the comparison of Figure 1 and Figure 3 would suggest this fact). The optimization algorithm is not novel either.
- Lack of detailed description and analysis for the proposed model S. In Section 5.2, only brief descriptions are given for the proposed approach.
- The selected baseline methods are too weak as competitors, some important relevant methods are also missing in the comparisons. For the graph embedding learning task, one of the state-of-the-art approach is conducting Graph Convolutional Networks (GCNs), and GCNs seem to be able to tackle this problem as well. Moreover, the target task of this paper is mathematically identical to the rating prediction problem (if we treat the weight matrix of the graph as the rating matrix, and consider the nodes as users, for example), which can be loved by a classic collaborative filtering solution such as matrix factorization. The authors probably need to survey and compared against the proposed approach.


Convolution Neural Network (CNN) has gained tremendous success in computer vision tasks with its outstanding ability to capture the local latent features. Recently, there has been an increasing interest in extending CNNs to the general spatial domain. Although various types of graph convolution and geometric convolution methods have been proposed, their connections to traditional 2D-convolution are not well-understood. In this paper, we show that depthwise separable convolution is a path to unify the two kinds of convolution methods in one mathematical view, based on which we derive a novel Depthwise Separable Graph Convolution that subsumes existing graph convolution methods as special cases of our formulation. Experiments show that the proposed approach consistently outperforms other graph convolution and geometric convolution baselines on benchmark datasets in multiple domains.

The paper presents a Depthwise Separable Graph Convolution network that aims
at generalizing Depthwise convolutions, that exhibit a nice performance in image
related tasks, to the graph domain. In particular it targets
Graph Convolutional Networks.
In the abstract the authors mention that the Depthwise Separable Graph Convolution
that they propose is the key to understand the connections between geometric
convolution methods and traditional 2D ones. I am afraid I have to disagree as
the proposed approach is not giving any better understanding of what needs to be
done and why. It is an efficient way to mimic what has worked so far for the planar
domain but I would not consider it as fundamental in "closing the gap".
I feel that the text is often redundant and that it could be simplified a lot.
For example the authors state in various parts that DSC does not work on
non-Euclidean data. Section 2 should be clearer and used to better explain
related approaches to motivate the proposed one.
In fact, the entire motivation, at least for me, never went beyond the simple fact
that this happens to be a good way to improve performance. The intuition given
is not sufficient to substantiate some of the claims on generality and understanding
of graph based DL.
In 3.1, at point (2), the authors mention that DSC filters are learned from the
data whereas GC uses a constant matrix. This is not correct, as also reported in
equation 2. The matrix U is learned from the data as well.
Equation (4) shows that the proposed approach would weight Q different GC
layers. In practical terms this is a linear combination of these graph
convolutional layers.
What is not clear is the \Delta_{ij} definition. It is first introduced in 2.3
and described as the relative position of pixel i and pixel j on the image, but
then used in the context of a graph in (4). What is the coordinate system used
by the authors in this case? This is a very important point that should be made
clearer.
Why is the Related Work section at the end? I would put it at the front.
The experiments compare with the recent relevant literature. I think that having
less number of parameters is a good thing in this setting as the data is scarce,
however I would like to see a more in-depth comparison with respect to the number
of features produced by the model itself. For example GCN has a representation
space (latent) much smaller than DSCG.
No statistics over multiple runs are reported, and given the high variance of
results on these datasets I would like them to be reported.
I think the separability of the filters in this case brings the right level of
simplification to the learning task, however as it also holds for the planar case
it is not clear whether this is necessarily the best way forward.
What are the underlying mathematical insights that lead towards selecting
separable convolutions?
Overall I found the paper interesting but not ground-breaking. A nice application
of the separable principle to GCN. Results are also interesting but should be
further verified by multiple runs.


Published as a conference paper at ICLR 2018 DYNAMIC NEURAL PROGRAM EMBEDDINGS FOR PRO- GRAM REPAIR
Neural program embeddings have shown much promise recently for a variety of program analysis tasks, including program synthesis, program repair, codecompletion, and fault localization. However, most existing program embeddings are based on syntactic features of programs, such as token sequences or abstract syntax trees. Unlike images and text, a program has well-defined semantics that can be difficult to capture by only considering its syntax (i.e. syntactically similar programs can exhibit vastly different run-time behavior), which makes syntaxbased program embeddings fundamentally limited. We propose a novel semantic program embedding that is learned from program execution traces. Our key insight is that program states expressed as sequential tuples of live variable values not only capture program semantics more precisely, but also offer a more natural fit for Recurrent Neural Networks to model. We evaluate different syntactic and semantic program embeddings on the task of classifying the types of errors that students make in their submissions to an introductory programming class and on the CodeHunt education platform. Our evaluation results show that the semantic program embeddings significantly outperform the syntactic program embeddings based on token sequences and abstract syntax trees. In addition, we augment a search-based program repair system with predictions made from our semantic embedding and demonstrate significantly improved search efficiency.

Summary of paper: The paper proposes an RNN-based neural network architecture for embedding programs, focusing on the semantics of the program rather than the syntax. The application is to predict errors made by students on programming tasks. This is achieved by creating training data based on program traces obtained by instrumenting the program by adding print statements. The neural network is trained using this program traces with an objective for classifying the student error pattern (e.g. list indexing, branching conditions, looping bounds).
---
Quality: The experiments compare the three proposed neural network architectures with two syntax-based architectures. It would be good to see a comparison with some techniques from Reed & De Freitas (2015) as this work also focuses on semantics-based embeddings.
Clarity: The paper is clearly written.
Originality: This work doesn't seem that original from an algorithmic point of view since Reed & De Freitas (2015) and Cai et. al (2017) among others have considered using execution traces. However the application to program repair is novel (as far as I know).
Significance: This work can be very useful for an educational platform though a limitation is the need for adding instrumentation print statements by hand.
---
Some questions/comments:
- Do we need to add the print statements for any new programs that the students submit? What if the structure of the submitted program doesn't match the structure of the intended solution and hence adding print statements cannot be automated?
---
References
Cai, J., Shin, R., & Song, D. (2017). Making Neural Programming Architectures Generalize via Recursion. In International Conference on Learning Representations (ICLR).


Generative adversarial networks (GANs) are a powerful framework for generative tasks. However, they are difficult to train and tend to miss modes of the true data generation process. Although GANs can learn a rich representation of the covered modes of the data in their latent space, the framework misses an inverse mapping from data to this latent space. We propose Invariant Encoding Generative Adversarial Networks (IVE-GANs), a novel GAN framework that introduces such a mapping for individual samples from the data by utilizing features in the data which are invariant to certain transformations. Since the model maps individual samples to the latent space, it naturally encourages the generator to cover all modes. We demonstrate the effectiveness of our approach in terms of generative performance and learning rich representations on several datasets including common benchmark image generation tasks.

This paper presents the IVE-GAN, a model that introduces en encoder to the Generative Adversarial Network (GAN) framework. The model is evaluated qualitatively through samples and reconstructions on a synthetic dataset, MNIST and CelebA.
Summary:
The evaluation is superficial, no quantitative evaluation is presented and key aspects of the model are not explored. Overall, there just is not enough innovation or substance to warrant publication at this point.
Impact:
The motivation given throughout the introduction -- to add an encoder (inference) network to GANs -- is a bit odd in the light of the existing literature. In addition to the BiGAN/ALI models that were cited, there are a number of (not cited) papers with various ways of combining GANs with VAE encoders to accomplish exactly this. If your goal was to improve reconstructions in ALI, one could simply add an reconstruction (or cycle) penalty to the ALI objective as advocated in the (not cited) ALICE paper (Li et al., 2017 -- "ALICE: Towards Understanding Adversarial Learning for Joint Distribution Matching").
The training architecture presented here is novel as far as I know, though I am unconvinced that it represents an optimum in model design space. The model presented in the ALICE paper would seem to be a more elegant solution to the motivation given in this paper.
Model Feature:
The authors should discuss in detail the interaction between the regular GAN pipeline and the introduced variant (with the transformations). Why is the standard GAN objective thrown in? I assume it is to allow you to sample directly from the noise in z (as opposed to z' which is used for reconstruction), but this is not discussed in much detail. The GAN objective and the added IVE objective seem like they will interact in not all together beneficial ways, with the IVE component pushing to make the distribution in z complicated. This would result in a decrease in sample quality. Does it? Exploration of this aspect of the model should be included in the empirical evaluation.
Also the addition of the transformations added to the proposed IVE pipeline seem to cause the latent variations z' to encode these variations rather than the natural variations that exist in the dataset. They would seem to make it difficult to encode someone face and make some natural manipulates (such as adjusting the smile) that are not included in this transformations.
Empirical Evaluation:
Comparison to BiGAN/ALI: The authors motivate their work by drawing comparisons to BiGAN/ALI, showing CelebA reconstructions from the ALI paper in the appendix. The comparison is not fair for two reasons, (1) authors should state that their reconstructions are made at a higher resolution (seems like 128x128, which is now standard but was not so when the BiGAN/ALI papers came out, they were sampled at 64x64), also, unlike the ALI results, they authors cut the background away from the CelebA faces. This alone could account for the difference between the two models, as ICE-GAN only has to encode the variability extant in faces and hair, ALI had to additionally encode the much greater variability in the background. The failure to control the experimental conditions makes this comparison inappropriate.
There is no quantitative evaluations at all. While many GAN papers do not place an emphasis on quantitative evaluations, at this point, I consider the complete lack of such an evaluation as a weakness of the paper.
Finally, based on just the samples offered in the paper, which is admittedly a fairly weak standard, the model does not seem to be among the state-of-the-art on CelebA that have been reported in the literature. Given the rapid progress that is being made, I do not feel this should be could against this particular paper, but the quality of samples cannot be considered a compelling reason to accept the paper.
Minor comment:
The authors appear to be abusing the ICLR style file by not leaving a blank line  between paragraphs. This is annoying and not at all necessary since ICLR does not have a strict page limit.
Figure 1 is not consistent with the model equations (in Eqns. 3). In particular, Figure 1 is missing the standard GAN component of the model.
I assume that the last term in Eqns 3 should have G(z) as opposed to G(z',E(x)). Is that right?


Attention models have been intensively studied to improve NLP tasks such as machine comprehension via both question-aware passage attention model and selfmatching attention model. Our research proposes phase conductor (PhaseCond) for attention models in two meaningful ways. First, PhaseCond, an architecture of multi-layered attention models, consists of multiple phases each implementing a stack of attention layers producing passage representations and a stack of inner or outer fusion layers regulating the information flow. Second, we extend and improve the dot-product attention function for PhaseCond by simultaneously encoding multiple question and passage embedding layers from different perspectives. We demonstrate the effectiveness of our proposed model PhaseCond on the SQuAD dataset, showing that our model significantly outperforms both stateof-the-art single-layered and multiple-layered attention models. We deepen our results with new findings via both detailed qualitative analysis and visualized examples showing the dynamic changes through multi-layered attention models.

This paper proposes a new machine comprehension model, which integrates several contributions like different embeddings for gate function and passage representation function, self-attention layers and highway network based fusion layers. The proposed method was evaluated on the SQuAD dataset only, and marginal improvement was observed compared to the baselines.
(1) One concern I have for this paper is about the evaluation. The paper only evaluates the proposed method on the SQuAD data with systems submitted in July 2017, and the improvement is not very large. As a result, the results are not suggesting significance or generalizability of the proposed method.
(2) The paper gives some ablation tests like reducing the number of layers and removing the gate-specific question embedding, which help a lot for understanding how the proposed methods contribute to the improvement. However, the results show that the deeper self-attention layers are indeed useful (but still not improving a lot, about 0.7-0.8%). The other proposed components contribute less significant. As a result, I suggest the authors add more ablation tests regarding (1) replacing the outer-fusion with simple concatenation (it should work for two attention layers); (2) removing the inner-fusion layer and only use the final layer's output, and using residual connections (like many NLP papers did) instead of the more complicated GRU stuff.
(3) Regarding the ablation in Table 2, my first concern is that the improvement seems small (~0.5%). As a result, I am wondering whether this separated question embedding really brings new information, or the similar improvement can be achieved by increasing the size of LSTM layers. For example, if we use the single shared question embeddings, but increase the size from 128 to some larger number like 192, can we observe similar improvement. I suggest the authors try this experiment as well and I hope the answer is no, as separated input embeddings for gate functions was verified to be useful in some "old" works with syntactic features as gate values, like "Semantic frame identification with distributed word representations" and "Learning composition models for phrase embeddings" etc.
(4) Please specify which version of the SQuAD leaderboard is used in Table 3. Is it a snapshot of the Jul 14 one? Because this paper is not comparing to the state-of-the-art, no specification of the leaderboard version may confuse the other reviewers and readers. By the way, it will be better to compare to the snapshot of Oct 2017 as well, indicating the position of this work during the submission deadline.
Minor issues:
(1) There are typos in Figure 1 regarding the notations of Question Features and Passage Features.
(2) In Figure 1, I suggest adding an "N \times" symbol to the left of the Q-P Attention Layer and remove the current list of such layers, in order to be consistent to the other parts of the figure.
(3) What is the relation between the "PhaseCond, QPAtt+" in Table 2 and the "PhaseCond" in Table 3? I was assuming that those are the same system but did not see the numbers match each other.


We consider the problem of exploration in meta reinforcement learning. Two new meta reinforcement learning algorithms are suggested: E-MAML and E-RL 2 . Results are presented on a novel environment we call 'Krazy World' and a set of maze environments. We show E-MAML and E-RL 2 deliver better performance on tasks where exploration is important.

The paper proposes a trick of extending objective functions to drive exploration in meta-RL on top of two recent so-called meta-RL algorithms, Model-Agnostic Meta-Learning (MAML) and RL^2.
Pros:
+ Quite simple but promising idea to augment exploration in MAML and RL^2 by taking initial sampling distribution into account.
+ Excellent analysis of learning curves with variances across two different environments. Charts across different random seeds and hyperparameters indicate  reproducibility.
Cons/Typos/Suggestions:
- The brief introduction to meta-RL is missing lots of related work - see below.
- Equation (3) and equations on the top of page 4: Mathematically, it looks better to swap \mathrm{d}\tau and \mathrm{d}\bar{\tau}, to obtain a consistent ordering with the double integrals.
- In page 4, last paragraph before Section 5, “However, during backward pass, the future discounted returns for the policy gradient computation will zero out the contributions from exploratory episodes”:  I did not fully understand this - please explain better.
- It is not very clear if the authors use REINFORCE or more advanced approaches like TRPO/PPO/DDPG to perform policy gradient updates?
- I'd like to see more detailed hyperparameter settings.
- Figures 10, 11, 12, 13, 14: Too small to see clearly. I would propose to re-arrange the figures in either [2, 2]-layout, or a single column layout, particularly for Figure 14.
- Figures 5, 6, 9: Wouldn't it be better to also use log-scale on the x-axis for consistent comparison with curves in Krazy World experiments ?
3. It could be very interesting to benchmark also in Mujoco environments, such as modified Ant Maze.
Overall, the idea proposed in this paper is interesting. I agree with the authors that a good learner should be able to generalize to new tasks with very few trials compared with learning each task from scratch. This, however, is usually called transfer learning, not metalearning. As mentioned above, experiments in more complex, continuous control tasks with Mujoco simulators might be illuminating.
Relation to prior work:
p 2: Authors write: "Recently, a flurry of new work in Deep Reinforcement Learning has provided the foundations for tackling RL problems that were previously thought intractable. This work includes: 1) Mnih et al. (2015; 2016), which allow for discrete control in complex environments directly from raw images. 2) Schulman et al. (2015); Mnih et al. (2016); Schulman et al. (2017); Lillicrap et al. (2015), which have allowed for high-dimensional continuous control in complex environments from raw state information."
Here it should be mentioned that the first RL for high-dimensional continuous control in complex environments from raw state information was actually published in mid 2013:
(1) Koutnik, J., Cuccu, G., Schmidhuber, J., and Gomez, F. (July 2013). Evolving large-scale neural networks for vision-based reinforcement learning. GECCO 2013, pages 1061-1068, Amsterdam. ACM.
p2: Authors write: "In practice, these methods are often not used due to difficulties with high-dimensional observations, difficulty in implementation on arbitrary domains, and lack of promising results."
Not quite true - RL robots with high-dimensional video inputs and intrinsic motivation learned to explore in 2015:
(2) Kompella, Stollenga, Luciw, Schmidhuber. Continual curiosity-driven skill acquisition from high-dimensional video inputs for humanoid robots. Artificial Intelligence, 2015.
p2: Authors write: "Although this line of work does not explicitly deal with exploration in meta learning, it remains a large source of inspiration for this work."
p2: Authors write: "To the best of our knowledge, there does not exist any literature addressing the topic of exploration in meta RL."
But there is such literature - see the following meta-RL work where exploration is the central issue:
(3) J. Schmidhuber. Exploring the Predictable. In Ghosh, S. Tsutsui, eds., Advances in Evolutionary Computing, p. 579-612, Springer, 2002.
The RL method of this paper is the one from the original meta-RL work:
(4) J. Schmidhuber. On learning how to learn learning strategies. Technical Report FKI-198-94, Fakultät für Informatik, Technische Universität München, November 1994.
Which then led to:
(5) J. Schmidhuber, J.  Zhao, N. Schraudolph. Reinforcement learning with self-modifying policies. In S. Thrun and L. Pratt, eds., Learning to learn, Kluwer, pages 293-309, 1997.
p2: "In hierarchical RL, a major focus is on learning primitives that can be reused and strung together. These primitives will frequently enable better exploration, since they’ll often relate to better coverage over state visitation frequencies. Recent work in this direction includes (Vezhnevets et al., 2017; Bacon & Precup, 2015; Tessler et al., 2016; Rusu et al., 2016)."
These are very recent refs - one should cite original work on hierarchical RL including:
J.  Schmidhuber. Learning to generate sub-goals for action sequences. In T. Kohonen, K. Mäkisara, O. Simula, and J. Kangas, editors, Artificial Neural Networks, pages 967-972. Elsevier Science Publishers B.V., North-Holland, 1991.
M. B. Ring. Incremental Development of Complex Behaviors through Automatic Construction of Sensory-Motor Hierarchies. Machine Learning: Proceedings of the Eighth International Workshop, L. Birnbaum and G. Collins, 343-347, Morgan Kaufmann, 1991.
M. Wiering and J. Schmidhuber. HQ-Learning. Adaptive Behavior 6(2):219-246, 1997
References to original work on meta-RL are missing. How does the approach of the authors relate to the following approaches?
(6) J. Schmidhuber. Gödel machines: Fully Self-Referential Optimal Universal Self-Improvers. In B. Goertzel and C. Pennachin, eds.: Artificial General Intelligence, p. 119-226, 2006.
(7) J. Schmidhuber. Evolutionary principles in self-referential learning, or on learning how to learn: The meta-meta-... hook. Diploma thesis, TUM, 1987.

Papers (4,5) above describe a universal self-referential, self-modifying RL machine. It can implement and run all kinds of learning algorithms on itself, but cannot learn them by gradient descent (because it's RL). Instead it uses what was later called the success-story algorithm (5) to handle all the meta-learning and meta-meta-learning etc.
Ref (7) above also has a universal programming language such that the system can learn to implement and run all kinds of computable learning algorithms, and uses what's now called Genetic Programming (GP), but applied to itself, to recursively evolve better GP methods through meta-GP and meta-meta-GP etc.
Ref (6) is about an optimal way of learning or the initial code of a learning machine through self-modifications, again with a universal programming language such that the system can learn to implement and run all kinds of computable learning algorithms.
General recommendation: Accept, provided the comments are taken into account, and the relation to previous work is established.