Published as a conference paper at ICLR 2018 TOWARDS SYNTHESIZING COMPLEX PROGRAMS FROM INPUT-OUTPUT EXAMPLES
In recent years, deep learning techniques have been developed to improve the performance of program synthesis from input-output examples. Albeit its significant progress, the programs that can be synthesized by state-of-the-art approaches are still simple in terms of their complexity. In this work, we move a significant step forward along this direction by proposing a new class of challenging tasks in the domain of program synthesis from input-output examples: learning a context-free parser from pairs of input programs and their parse trees. We show that this class of tasks are much more challenging than previously studied tasks, and the test accuracy of existing approaches is almost 0%. We tackle the challenges by developing three novel techniques inspired by three novel observations, which reveal the key ingredients of using deep learning to synthesize a complex program. First, the use of a non-differentiable machine is the key to effectively restrict the search space. Thus our proposed approach learns a neural program operating a domain-specific non-differentiable machine. Second, recursion is the key to achieve generalizability. Thus, we bake-in the notion of recursion in the design of our non-differentiable machine. Third, reinforcement learning is the key to learn how to operate the non-differentiable machine, but it is also hard to train the model effectively with existing reinforcement learning algorithms from a cold boot. We develop a novel two-phase reinforcement learningbased search algorithm to overcome this issue. In our evaluation, we show that using our novel approach, neural parsing programs can be learned to achieve 100% test accuracy on test inputs that are 500× longer than the training samples.

This paper presents a reinforcement learning based approach to learn context-free
parsers from pairs of input programs and their corresponding parse trees. The main
idea of the approach is to learn a neural controller that operates over a discrete
space of programmatic actions such that the controller is able to produce the
desired parse trees for the input programs. The neural controller is trained using
a two-phase reinforcement learning approach where the first phase is used to find
a set of candidate traces for each input-output example and the second phase is
used to find a satisfiable specification comprising of 1 unique trace per example
such that there exists a program that is consistent with all the traces. The
approach is evaluated on two datasets comprising of learning parsers for an
imperative WHILE language and a functional LAMBDA language. The results show that
the proposed approach is able to achieve 100% generalization on test sets with
programs upto 100x longer than the training programs, while baseline approaches
such as seq2seq and stack LSTM do not generalize at all.
The idea to decompose the synthesis task into two sub-tasks of first learning
a set of individual traces for each example, and then learning a program consistent
with a satisfiable subset of traces is quite interesting and novel. The use of
reinforcement learning in the two phases of finding candidate trace sets with
different reward functions for different operators and searching for a satisfiable
subset of traces is also interesting. Finally, the results leading to perfect
generalization on parsing 100x longer input programs is also quite impressive.
While the presented results are impressive, a lot of design decisions such as
designing specific operators (Call, Reduce,..) and their specific semantics seem
to be quite domain-specific for the parsing task. The comparison with general
approaches such as seq2seq and stack LSTM might not be that fair as they are
not restricted to only those operators and this possibly also explains the low
generalization accuracies. Can the authors comment on the generality of the
presented approach to some other program synthesis tasks?
For comparison with the baseline networks such as seq2seq and stack-LSTM, what
happens if the number of training examples is 1M (say programs upto size 100)?
10k might be too small a number of training examples and these networks can
easily overfit such a small dataset.
The paper mentions that developing a parser can take upto 2x/3x more time than
developing the training set. How large were the 150 examples that were used for
training the models and were they hand-designed or automatically generated by a
parsing algorithm? Hand generating parse trees for complex expressions seems to
be more tedious and error-prone that writing a modular parser.
The reason there are only 3 to 5 candidate traces per example is because the training
examples are small? For longer programs, I can imagine there can be thousands of bad
traces as it only needs one small mistake to propagate to full traces. Related to this
question, what happens to the proposed approach if it is trained with 1000 length programs?
What is the intuition behind keeping M1, M2 and M3 constants? Shouldn’t they be adaptive
values with respect to the number of candidate traces found so far?
For phase-1 of learning candidate traces, what happens if the algorithm was only using the
outside loop (M2) and performing REINFORCE without the inside loop?
The current paper presentation is a bit too dense to clearly understand the LL machine
model and the two-phase algorithm. A lot of important details are currently in the
appendix section with several forward references. I would suggest moving Figure 3
from appendix to the main paper, and also add a concrete example in section 4 to
better explain the two-phase strategy.


Published as a conference paper at ICLR 2018 DETECTING STATISTICAL INTERACTIONS FROM NEURAL NETWORK WEIGHTS
Interpreting neural networks is a crucial and challenging task in machine learning. In this paper, we develop a novel framework for detecting statistical interactions captured by a feedforward multilayer neural network by directly interpreting its learned weights. Depending on the desired interactions, our method can achieve significantly better or similar interaction detection performance compared to the state-of-the-art without searching an exponential solution space of possible interactions. We obtain this accuracy and efficiency by observing that interactions between input features are created by the non-additive effect of nonlinear activation functions, and that interacting paths are encoded in weight matrices. We demonstrate the performance of our method and the importance of discovered interactions via experimental results on both synthetic datasets and real-world application datasets.

Based on a hierarchical hereditary assumption, this paper identifies pairwise and high-order feature interactions by re-interpreting neural network weights, assuming higher-order interactions exist only if all its induced lower-order interactions exist. Using a multiplication of the absolute values of all neural network weight matrices on top of the first hidden layer, this paper defines the aggregated strength z_r of each hidden unit r contributing to the final target output y. Multiplying z_r by some statistics of weights connecting a subset of input features to r and summing over r results in final interaction strength of each feature interaction subsets, with feature interaction order equal to the size of each feature subset.
Main issues:
1. Aggregating neural network weights to identify feature interactions is very interesting. However, completely ignoring
activation functions makes the method quite crude.
2. High-order interacting features must share some common hidden unit somewhere in a hidden layer within a deep neural network. Restricting to the first hidden layer in Algorithm 1 inevitably misses some important feature interactions.
3. The neural network weights heavily depends on the l1-regularized neural network training, but a group lasso penalty makes much more sense. See Group Sparse Regularization for Deep Neural Networks (https://arxiv.org/pdf/1607.00485.pdf).
4. The experiments are only conducted on some synthetic datasets with very small feature dimensionality p. Large-scale experiments are needed.
5. There are some important references missing. For example, RuleFit is a good baseline method for identifying feature interactions based on random forest and l1-logistic regression (Friedman and Popescu, 2005, Predictive learning via rule ensembles); Relaxing strict hierarchical hereditary constraints, high-order l1-logistic regression based on tree-structured feature expansion identifies pairwise and high-order multiplicative feature interactions (Min et al. 2014, Interpretable Sparse High-Order Boltzmann Machines); Without any hereditary constraint, feature interaction matrix factorization with l1 regularization identifies pairwise feature interactions on datasets with high-dimensional features (Purushotham et al. 2014, Factorized Sparse Learning Models with Interpretable High Order Feature Interactions).
6. At least, RuleFit (Random Forest regression for getting rules + l1-regularized regression) should be used as a baseline in the experiments.
Minor issues:
Ranking of feature interactions in Algorithm 1 should be explained in more details.
On page 3: b^{(l)} \in R^{p_l}, l should be from 1, .., L. You have b^y.
In summary, the idea of using neural networks for screening pairwise and high-order feature interactions is novel, significant, and interesting.  However, I strongly encourage the authors to perform additional experiments with careful experiment design to address some common concerns in the reviews/comments for the acceptance of this paper.

========
The additional experimental results are convincing, so I updated my rating score.


Workshop track -ICLR 2018 LEARNING REPRESENTATIONS AND GENERATIVE MODELS FOR 3D POINT CLOUDS
Three-dimensional geometric data offer an excellent domain for studying representation learning and generative modeling. In this paper, we look at geometric data represented as point clouds. We introduce a deep autoencoder (AE) network with state-of-the-art reconstruction quality and generalization ability. The learned representations outperform existing methods on 3D recognition tasks and enable basic shape editing via simple algebraic manipulations, such as semantic part editing, shape analogies and shape interpolation. We perform a thorough study of different generative models including: GANs operating on the raw point clouds, significantly improved GANs trained in the fixed latent space of our AEs and, Gaussian mixture models (GMM). For our quantitative evaluation we propose measures of sample fidelity and diversity based on matchings between sets of point clouds. Interestingly, our careful evaluation of generalization, fidelity and diversity reveals that GMMs trained in the latent space of our AEs produce the best results.

Summary:
This paper proposes generative models for point clouds. First, they train an auto-encoder for 3D point clouds,  somewhat similar to PointNet (by Qi et al.). Then, they train generative models over the auto-encoder's latent space, both using a "latent-space GAN" (l-GAN) that outputs latent codes, and a Gaussian Mixture Model. To generate point clouds, they sample a latent code and pass it to the decoder. They also introduce a "raw point cloud GAN" (r-GAN) that, instead of generating a latent code, directly produces a point cloud.
They evaluate the methods on several metrics. First, they show that the autoencoder's latent space is a good representation for classification problems, using the ModelNet dataset. Second, they evaluate the generative model on several metrics (such as Jensen-Shannon Divergence) and study the benefits and drawbacks of these metrics, and suggest that one-to-one mapping metrics such as earth mover's distance are desirable over Chamfer distance. Methods such as the r-GAN score well on the latter by over-representing parts of an object that are likely to be filled.
Pros:
- It is interesting that the latent space models are most successful, including the relatively simple GMM-based model. Is there a reason that these models have not been as successful in other domains?
- The comparison of the evaluation metrics could be useful for future work on evaluating point cloud GANs. Due to the simplicity of the method, this paper could be a useful baseline for future work.
- The part-editing and shape analogies results are interesting, and it would be nice to see these expanded in the main paper.
Cons:
- How does a model that simply memorizes (and randomly samples) the training set compare to the auto-encoder-based models on the proposed metrics? How does the diversity of these two models differ?
- The paper simultaneously proposes methods for generating point clouds, and for evaluating them. The paper could therefore be improved by expanding the section comparing to prior, voxel-based 3D methods, particularly in terms of the diversity of the outputs. Although the performance on automated metrics is encouraging, it is hard to conclude much about under what circumstances one representation or model is better than another.
- The technical approach is not particularly novel. The auto-encoder performs fairly well, but it is just a series of MLP layers that output a Nx3 matrix representing the point cloud, trained to optimize EMD or Chamfer distance. The most successful generative models are based on sampling values in the auto-encoder's latent space using simple models (a two-layer MLP or a GMM).
- While it is interesting that the latent space models seem to outperform the r-GAN, this may be due to the relatively poor performance of r-GAN than to good performance of the latent space models, and directly training a GAN on point clouds remains an important problem.
- The paper could possibly be clearer by integrating more of the "background" section into later sections. Some of the GAN figures could also benefit from having captions.
Overall, I think that this paper could serve as a useful baseline for generating point clouds, but I am not sure that the contribution is significant enough for acceptance.


We capitalize on the natural compositional structure of images in order to learn object segmentation with weakly labeled images. The intuition behind our approach is that removing objects from images will yield natural images, however removing random patches will yield unnatural images. We leverage this signal to develop a generative model that decomposes an image into layers, and when all layers are combined, it reconstructs the input image. However, when a layer is removed, the model learns to produce a different image that still looks natural to an adversary, which is possible by removing objects. Experiments and visualizations suggest that this model automatically learns object segmentation on images labeled only by scene better than baselines.

This paper creates a layered representation in order to better learn segmentation from unlabeled images. It is well motivated, as Fig. 1 clearly shows the idea that if the segmentation was removed properly, the result would still be a natural image. However, the method itself as described in the paper leaves many questions about whether they can achieve the proposed goal.
I cannot see from the formulation why would this model work as it is advertised. The formulation (3-4) looks like a standard GAN, with some twist about measuring the GAN loss in the z space (this has been used in e.g. PPGN and CVAE-GAN). I don't see any term that would guarantee:
1) Each layer is a natural image. This was advertised in the paper, but the loss function is only on the final product G_K. The way it is written in the paper, the result of each layer does not need to go through a discriminator. Nothing seems to have been done to ensure that each layer outputs a natural image.
2) None of the layers is degenerate. There does not seem to be any constraint either regularizing the content in each layer, or preventing any layer to be non-degenerate.
3) The mask being contiguous. I don't see any term ensuring the mask being contiguous, I imagine normally without such terms doing such kinds of optimization would lead to a lot of fragmented small areas being considered as the mask.
The claim that this paper is for unsupervised semantic segmentation is overblown. A major problem is that when conducting experiments, all the images seem to be taken from a single category, this implicitly uses the label information of the category. In that regard, this cannot be viewed as an unsupervised algorithm.
Even with that, the results definitely looked too good to be true. I have a really difficult time believing why such a standard GAN optimization would not generate any of the aforementioned artifacts and would perform exactly as the authors advertised. Even if it does work as advertised, the utilization of implicit labels would make it subject to comparisons with a lot of weakly-supervised learning papers with far better results than shown in this paper. Hence I am pretty sure that this is not up to the standards of ICLR.
I have read the rebuttal and still not convinced. I don't think the authors managed to convince me that this method would work the way it's advertised. I also agree with Reviewer 2 that there is a lack of comparison against baselines.


We present DANTE, a novel method for training neural networks, in particular autoencoders, using the alternating minimization principle. DANTE provides a distinct perspective in lieu of traditional gradient-based backpropagation techniques commonly used to train deep networks. It utilizes an adaptation of quasi-convex optimization techniques to cast autoencoder training as a bi-quasi-convex optimization problem. We show that for autoencoder configurations with both differentiable (e.g. sigmoid) and non-differentiable (e.g. ReLU) activation functions, we can perform the alternations very effectively. DANTE effortlessly extends to networks with multiple hidden layers and varying network configurations. In experiments on standard datasets, autoencoders trained using the proposed method were found to be very promising and competitive to traditional backpropagation techniques, both in terms of quality of solution, as well as training speed.

After reading the rebuttal:
The authors addressed some of my theoretical questions. I think the paper is borderline, leaning towards accept.
I do want to note my other concerns:
I suspect the theoretical results obtained here are somewhat restricted to the least-squares, autoencoder loss.
And note that the authors show that the proposed algorithm performs comparably to SGD, but not significantly better. The classification result (Table 1) was obtained on the autoencoder features instead of training a classifier on the original inputs. So it is not clear if the proposed algorithm is better for training the classifier, which may be of more interest.
=============================================================
This paper presents an algorithm for training deep neural networks. Instead of computing gradient of all layers and perform updates of all weight parameters at the same time, the authors propose to perform alternating optimization on weights of individual layers.
The theoretical justification is obtained for single-hidden-layer auto-encoders. Motivated by recent work by Hazan et al 2015, the authors developed the local-quasi-convexity of the objective w.r.t. the hidden layer weights for the generalized RELU activation. As a result, the optimization problem over the single hidden layer can be optimized efficiently using the algorithm of Hazan et al 2015. This itself can be a small, nice contribution.
What concerns me is the extension to multiple layers. Some questions are not clear from section 3.4:
1. Do we still have local-quasi-convexity for the weights of each layer, when there are multiple nonlinear layers above it? A negative answer to this question will somewhat undermine the significance of the single-hidden-layer result.
2. Practically, even if the authors can perform efficient optimization of weights in individual layers, when there are many layers, the alternating optimization nature of the algorithm can possibly result in overall slower convergence. Also, since the proposed algorithm still uses gradient based optimizers for each layer, computing the gradient w.r.t. lower layers (closer to the inputs) are still done by backdrop, which has pretty much the same computational cost of the regular backdrop algorithm for updating all layers at the same time. As a result, I am not sure if the proposed algorithm is on par with / faster than the regular SGD algorithm in actual runtime. In the experiments, the authors plotted the training progress w.r.t. the minibatch iterations, I do not know if the minibatch iteration is a proxy for actual runtime (or number of floating point operations).
3. In the experiments, the authors found the network optimized by the proposed algorithm generalize better than regular SGD. Is this result consistent (across dataset, random initializations, etc), and can the authors elaborate the intuition behind?


A widely observed phenomenon in deep learning is the degradation problem: increasing the depth of a network leads to a decrease in performance on both test and training data. Novel architectures such as ResNets and Highway networks have addressed this issue by introducing various flavors of skip-connections or gating mechanisms. However, the degradation problem persists in the context of plain feed-forward networks. In this work we propose a simple method to address this issue. The proposed method poses the learning of weights in deep networks as a constrained optimization problem where the presence of skip-connections is penalized by Lagrange multipliers. This allows for skip-connections to be introduced during the early stages of training and subsequently phased out in a principled manner. We demonstrate the benefits of such an approach with experiments on MNIST, fashion-MNIST, CIFAR-10 and CIFAR-100 where the proposed method is shown to greatly decrease the degradation effect and is often competitive with ResNets.

EDIT: The rating has been changed. See thread below for explanation / further comments.
ORIGINAL REVIEW: In this paper, the authors present a new training strategy, VAN, for training very deep feed-forward networks without skip connections (henceforth called VDFFNWSC) by introducing skip connections early in training and then gradually removing them.
I think the fact that the authors demonstrate the viability of training VDFFNWSCs that could have, in principle, arbitrary nonlinearities and normalization layers, is somewhat valuable and as such I would generally be inclined towards acceptance, even though the potential impact of this paper is limited because the training strategy proposed is (by deep learning standards) relatively complicated, requires tuning two additional hyperparameters in the initial value of \lambda as well as the step size for updating \lambda, and seems to have no significant advantage over just using skip connections throughout training. So my rating based on the message of the paper would be 6/10.
However, there appear to be a range of issues. As long as those issues remain unresolved, my rating is at is but if those issues were resolved it could go up to a 6.
+++ Section 3.1 problems +++
- I think the toy example presented in section 3.1 is more confusing than it is helpful because the skip connection you introduce in the toy example is different from the skip connection you introduce in VANs. In the toy example, you add (1 - \alpha)wx whereas in the VANs you add (1 - \alpha)x. Therefore, the type of vanishing gradient that is observed when tanh saturates, which you combat in the toy model, is not actually combated at all in the VAN model. While it is true that skip connections combat vanishing gradients in certain situations, your example does not capture how this is achieved in VANs.
- The toy example seems to be an example where Lagrangian relaxation fails, not where it succeeds. Looking at figure 1, it appears that you start out with some alpha < 1 but then immediately alpha converges to 1, i.e. the skip connection is eliminated early in training, because wx is further away from y than tanh(wx). Most of the training takes place without the skip connection. In fact, after 10^4 iterations, training with and without skip connection seem to achieve the same error. It appears that introducing the skip connection was next to useless and the model failed to recognize the usefulness of the skip connection early in training.
- Regarding the optimization algorithm involving \alpha^* at the end of section 3: It looks to me like a hacky, unprincipled method with no guarantees that just happened to work in the particular example you studied. You motivate the choice of \alpha^* by wanting to maximize the reduction in the local linear approximation to \mathcal{C} induced by the update on w. However, this reduction grows to infinity the larger the update is. Does that mean that larger updates are always better? Clearly not. If we wanted to reduce the size of the objective according to the local linear approximation, why wouldn't we choose infinitely large step sizes? Hence, the motivation for the algorithm you present is invalid. Here is an example where this algorithm fails: consider the point (x,y,w,\alpha,\lambda) = (100, \sigma(100), 1.0001, 1, 1). Here, w has almost converged to its optimum w* = 1. Correspondingly, the derivative of C is a small negative value. However, \alpha* is actually 0, and this choice would catapult w far away from w*.
If I haven't made a mistake in my criticisms above, I strongly suggest removing section 3.1 entirely or replacing it with a completely new example that does not suffer from the above issues.
+++ ResNet scaling +++
There is a crucial difference between VANs and ResNets. In the VAN initial state (alpha = 0.5), both the residual path and the skip path are multiplied by 0.5 whereas for ResNet, neither is multiplied by 0.5. Because of this, the experimental results between the two architectures are incomparable.
In a question I posed earlier, you claimed that this scaling makes no difference when batch normalization is used. I disagree. Let's look at an example. Consider ResNet first. It can be written as x + r_1 + r_2 + .. + r_B, where r_b is the value computed by residual block b. Now let's assume we insert a scaling constant after each residual block, say c = 0.5. Then the result is c^{B}x + c^{B-1}r_1 + c^{B-2}r_2 + .. + r_B. Therefore, contributions of lower blocks vanish exponentially. This effect is not combated by batch normalization.
So the learning dynamics for VAN and ResNet are very different because of this scaling. Therefore, there is an open question: are the differences in results between VAN and ResNet in your experiments caused by the removal of skip connections during training or by this scaling? Without this information, the experiments have limited value. In fact, I suspect that the vanishing of the contribution of lower blocks bears more responsibility for the declining performance of VAN at higher depths than the removal of skip connections.
If my assessment of the situation is correct, I would like to ask you to repeat your experiments with the following two settings:
- ResNet where after each block you multiply the result of the addition by 0.5, i.e. x_{l+1} = 0.5\mathcal{F}(x_l) + 0.5x_l
- VAN with the following altered equation: x_{l+1} = \mathcal{F}(x_l) + (1-\alpha)x_l, i.e. please remove the alpha in front of \mathcal{F}. Also, initialize \alpha to zero. This ensures that VAN starts out as a regular ResNet.
+++ writing issues +++
Title:
- "VARIABLE ACTIVATION NETWORKS: A SIMPLE METHOD TO TRAIN DEEP FEED-FORWARD NETWORKS WITHOUT SKIP-CONNECTIONS" This title can be read in two different ways. (A) [Train] [deep feed-forward networks] [without skip-connections] and (B) [Train] [deep feed-forward networks without skip connections]. In (A), the `without skip-connections' modifies the `train' and suggests that training took place without skip connections. In (B), the `without skip-connections' modifies `deep feed-forward networks' and suggests that the network trained has no skip connections. You must mean (B), because (A) is false. Since it is not clear from reading the title whether (A) or (B) is true, please reword it.
Abstract:
- "Part of the success of ResNets has been attributed to improvements in the conditioning of the optimization problem (e.g., avoiding vanishing and shattered gradients). In this work we propose a simple method to extend these benefits to the context of deep networks without skip-connections." Again, this is ambiguous. To me, this sentence implies that you extend the benefit of avoiding vanishing and exploding gradients to fully-connected networks without skip connections. However, nowhere in your paper do you show that trained VANs have less exploding / vanishing gradients than fully-connected networks trained the old-fashioned way. Again, please reword or include evidence.
- "where the proposed method is shown to outperform many architectures without skip-connections" Again, this sentence makes no sense to me. It seems to imply that VAN has skip connections. But in the abstract you defined VAN as an architecture without skip connections. Please make this more clear.
Introduction:
- "Indeed, Zagoruyko & Komodakis (2016) demonstrate that it is better to increase the width of ResNets than the depth, suggesting that perhaps only a few layers are learning useful representations." Just because increasing width may be better than increasing depth does not mean that deep layers don't learn useful representations. In fact, the claim that deep layers don't learn useful representations is directly contradicted by the paper.
section 3.1:
- replace "to to" by "to" in the second line
section 4:
- "This may be a result of the ensemble nature of ResNets (Veit et al., 2016), which does not play a significant role until the depth of the network increases." The ensemble nature of ResNet is a drawback, not an advantage, because it causes a lack of high-order co-adaptataion of layers. Therefore, it cannot contribute positively to the performance or ResNet.
As mentioned in earlier comments, please reword / clarify your use of "activation function". It is generally used a synonym for "nonlinearity", so please use it in this way. Change your claim that VAN is equivalent to PReLU. Please include your description of how your method can be extended to networks which do allow for skip connections.
+++ Hyperparameters +++
Since the initial values of \lambda and \eta' are new hyperparameters, include the values you chose for them, explain how you arrived at those values and plot the curve of how \lambda evolves for at least some of the experiments.


A bottleneck problem in machine learning-based relationship extraction (RE) algorithms, and particularly of deep learning-based ones, is the availability of training data in the form of annotated corpora. For specific domains, such as biomedicine, the long time and high expertise required for the development of manually annotated corpora explain that most of the existing one are relatively small (i.e., hundreds of sentences). Beside, larger corpora focusing on general or domain-specific relationships (such as citizenship or drug-drug interactions) have been developed. In this paper, we study how large annotated corpora developed for alternative tasks may improve the performances on biomedicine related tasks, for which few annotated resources are available. We experiment two deep learning-based models to extract relationships from biomedical texts with high performance. The first one combine locally extracted features using a Convolutional Neural Network (CNN) model, while the second exploit the syntactic structure of sentences using a Recursive Neural Network (RNN) architecture. Our experiments show that, contrary to the former, the latter benefits from a cross-corpus learning strategy to improve the performance of relationship extraction tasks. Indeed our approach leads to the best published performances for two biomedical RE tasks, and to state-of-the-art results for two other biomedical RE tasks, for which few annotated resources are available (less than 400 manually annotated sentences). This may be particularly impactful in specialized domains in which training resources are scarce, because they would benefit from the training data of other domains for which large annotated corpora does exist.

SUMMARY.
The paper presents a cross-corpus approach for relation extraction from text.
The main idea is complementing small training data for relation extraction with training data with different relation types.
The model is also connected with multitask learning approaches where the encoder for the input is the same but the output layer is different for each task. In this work, the output/softmax layer is different for each data type, while the encoder is shared.
The authors tried two different sentence encoders (cnn-based and tree-lstm), and final results are calculated on the low resource dataset.
Experimental results show that the tree-rnn encoder is able to capture valuable information from auxiliary data, while the cnn based does not.
----------
OVERALL JUDGMENT
The paper shows an interesting approach to data augmentation with data of different type for relation extraction.
I would have appreciated a section where the authors explain briefly what relation extraction is maybe with an example.
The paper is overall clear, although the experimental section has to be improved I believe.
From section 5.2 I am not able to understand the experimental setting the authors used, is it 10-fold CV? Did the authors tune the hyperparameters for each fold?
Are the results in table 3 obtained with tree-lstm?
What kind of ensembling did the authors chose for those experiments?
The author overstates that their model outperforms the state-of-the-art models they compare to, but that is not true for the EU-ADR dataset where in 2 out of 3 relation types the proposed model performs on par with the state-of-the-art model.
Finally, the authors used only one auxiliary dataset at the time, it would be interesting to see whether using all the auxiliary dataset together would improve results even more.
I would suggest the author also to check and revise citations (CNN's are not Collobert et al. invention, the same thing for the maximum likelihood objective) and more in general to improve the reference on relation extraction literature.


Spiking neural networks are being investigated both as biologically plausible models of neural computation and also as a potentially more efficient type of neural network. While convolutional spiking neural networks have been demonstrated to achieve near state-of-the-art performance, only one solution has been proposed to convert gated recurrent neural networks, so far. Recurrent neural networks in the form of networks of gating memory cells have been central in state-of-the-art solutions in problem domains that involve sequence recognition or generation. Here, we design an analog gated LSTM cell where its neurons can be substituted for efficient stochastic spiking neurons. These adaptive spiking neurons implement an adaptive form of sigma-delta coding to convert internally computed analog activation values to spike-trains. For such neurons, we approximate the effective activation function, which resembles a sigmoid. We show how analog neurons with such activation functions can be used to create an analog LSTM cell; networks of these cells can then be trained with standard backpropagation. We train these LSTM networks on a noisy and noiseless version of the original sequence prediction task from Hochreiter & Schmidhuber (1997), and also on a noisy and noiseless version of a classical working memory reinforcement learning task, the T-Maze. Substituting the analog neurons for corresponding adaptive spiking neurons, we then show that almost all resulting spiking neural network equivalents correctly compute the original tasks.

First the authors suggest an adaptive analog neuron (AAN) model which can be trained by back-propagation and then mapped to an Adaptive Spiking Neuron (ASN). Second, the authors suggest a network module called Adaptive Analog LSTM Cell (AA-LSTM) which contains input cells, input gates, constant error carousels (CEC) and output cells. Jointly with the AA-LSTM, the authors describe a spiking model (AS-LSTM) that is meant to reproduce its transfer function. It is shown quantitatively that the transfer functions of isolated AAN and AA-LSTM units are well approximated by their spiking counterparts. Two sets of experiments are reported, a sequence prediction task taken from the original LSTM paper and a T-maze task solved with reward based learning.
In general, the paper presents an interesting idea. However, it seems that the main claims of the introduction are not sufficiently well proven later. Also, I believe that the tasks are rather simple and therefore it is not demonstrated that the approach performs well on practically relevant tasks.
On general level, it should be clarified whether the model is meant to reproduce features of biology or whether the model is meant to be efficient. If the model is meant to reproduce biology, some features of the model are problematic. In particular, that the CEC is modeled with an infinitely long integration time constant of the input current. This would produce infinitely long EPSPs. However, I think there is a chance that minor changes of the model could still work while being more realistic. For example, I would find it more convincing to put the CEC into the adaptation time constants by using a large tau_gamma or tau_eta.
If the model is meant to provide efficient spiking neural networks, I find the tasks too simple and too artificial. This is particularly true in comparison to the speech recognition tasks VAD and TIMIT which were already solved in Esser et al. with spiking and efficient feedforward networks.
The authors say in the introduction that they target to model recurrent neural networks. This is an important open question. The usage of the CEC is an interesting idea toward this goal.
However, beside the presence of CEC I do not see any recurrence in the used networks. This seems in contradiction with what is implicitly claimed in the introduction, title and abstract. There are only input-output neuron connections in the sequence prediction task, and a single hidden layer for the T-maze (which does not seem to be recurrently connected). This is problematic as the authors mention that their goal is to reproduce the functionality of LSTMs with spiking neurons for which the network recurrence is an important feature.
Regarding more low-level comments:
- The authors used a truncated version of RTRL to train LSTMs and standard back-propagation for single neurons. I wonder why two different algorithms were used, as, in principle, they compute the same gradient
either forward or backward.
Is there a reason for this? Did the truncated RTRL bring any
additional benefit compared to the exact backpropagation already
implemented in automatic differentiation software?
- The sigma-delta neuron model seems quite ad-hoc and incompatible
with most simulators and dedicated hardware. I wonder whether the
AS-LSTM model would still be valid if the ASN model is replaced with a
standard SRM model for instance.
- The authors claim in the introduction that they made an analytical conversion from discrete to continuous time. I did not find this in the main text.
- The axes in Figure 1 are not defined (what is Delta S?) and the
caption does not match. "Average output signal [...] as a function of its incoming PSC I" output signal is not defined, and S is presented in the graph, but not I.


Workshop track -ICLR 2018 GRADIENTS EXPLODE -DEEP NETWORKS ARE SHALLOW -RESNET EXPLAINED
Whereas it is believed that techniques such as Adam, batch normalization and, more recently, SeLU nonlinearities "solve" the exploding gradient problem, we show that this is not the case in general and that in a range of popular MLP architectures, exploding gradients exist and that they limit the depth to which networks can be effectively trained, both in theory and in practice. We explain why exploding gradients occur and highlight the collapsing domain problem, which can arise in architectures that avoid exploding gradients. ResNets have significantly lower gradients and thus can circumvent the exploding gradient problem, enabling the effective training of much deeper networks, which we show is a consequence of a surprising mathematical property. By noticing that any neural network is a residual network, we devise the residual trick, which reveals that introducing skip connections simplifies the network mathematically, and that this simplicity may be the major cause for their success.

Summary of paper - The paper introduces the Gradient Scale Coefficient and uses it to demonstrate issues with the current understanding of where and why exploding gradients occur.
Review - The paper attempts to contribute to the discussion about the exploding gradient problem by both introducing a metric for discussing this issue and by showing that current understanding of the exploding gradient problem may be incorrect. It is admirable that the authors are seeking to add to the understanding about theory of neural nets instead of contributing a new architecture with better error rates but without understanding why said error rates are lower. While the authors list 7 contributions, the current version of the text is a challenge to read and makes it challenging to distill an overarching theme or narrative to these contributions.
The authors do mention experiments on page 8, but confess that some of the results are somewhat underwhelming. Unfortunately, all tables with the experimental results are left to the appendix. As this is a mostly theoretical paper, pushing experimental results to the appendix does make sense, but the repeated references to these tables suggest that these experimental results are crucial for the authors’ overall points.
While the authors do attempt to accomplish a lot in these nearly 16 pages of text, the authors' main points and overall narrative gets lost due to the writing that is a bit jumbled at times and that relies heavily on the supplement. There are several places where it is not immediately clear why a certain block of text is included (i.e. the proof outlines on pages 8 and 10). At other points the authors default to an chronological narrative that can be useful at times (i.e. page 9), but here seems to distract from their overall narrative.
This paper has a lot of content, but not all of it appears to be relevant to the authors’ central points. Furthermore, the paper is nearly double the recommended page length and has a nearly 30 page supplement. My biggest recommendations for this paper are for the authors to 1) articulate one theme and then 2) look at each part (whether that be section, paragraph, or sentence) and ask what does that part contribute to that theme.
Pros -
* This paper attempts to add the understanding of neural nets instead of only contributing better error rates on benchmark datasets.
* At several points, the authors seek to make the work accessible by offering lay explanations for their more technical points.
* The practical suggestions on page 16 are a true highlight and could provide an outline for possible revisions.
Cons -
* The main narrative is lost in the text, leaving a reader unsure of the authors main points and contributions as they read. For example, the authors’ first contribution is hidden among the text presentation of section 2.
* The paper relies heavily on the supplement to make their central points.
* It is nearly double the recommended page length with a nearly 30 page supplement
Minor issues -
* Use one style for introducing and defining terms either use italics or single quotes. The latter is not recommended since the authors use double quotes in the abstract to express that the exploding gradient problem is not solved.
* The citation style of Authors (YEAR) at times leads to awkward sentence parsing.
* Given that many figures have several subfigures, the authors should consider using a package that will denote subfigures with letters.
* The block quotes in the introduction may be quite important for points later in the paper, but summarizing the points of these quotes may be a better use of space. The authors more successfully did this in paragraph 2 of the introduction.
* All long descriptions of the appendix should be carefully revisited and possibly removed due to page length considerations.
* In the text, figure 4 (which is in the supplement) is referenced before figure 3 (which is in the text).
=-=-=-= Response to the authors
During the initial reviewing period, I was unable to distill the significance of the authors’ contributions from the current literature in large part due to the nature of the writing style. After reading the authors responses and consulting the differences between the versions of the paper, my review remains the same. It should be noted that all three reviewers pointed out the length of the paper as a weakness of the paper, and that in the most recent draft, the authors made the main text of the paper longer.
Consulting the differences between the paper revisions, I was initially intrigued with the volume of differences that shown in the summary bar. Upon closer inspection, I read a much stronger introduction and appreciated the summaries at the ends of sections 4.4 and 6. However, I did notice that the majority of these changes were superficial re-orderings of the original text. Given the limited substantive changes to the main text, I did not deeply re-read the text of the paper beyond the introduction.


In order for machine learning to be deployed and trusted in many applications, it is crucial to be able to reliably explain why the machine learning algorithm makes certain predictions. For example, if an algorithm classifies a given pathology image to be a malignant tumor, then the doctor may need to know which parts of the image led the algorithm to this classification. How to interpret black-box predictors is thus an important and active area of research. A fundamental question is: how much can we trust the interpretation itself? In this paper, we show that interpretation of deep learning predictions is extremely fragile in the following sense: two perceptively indistinguishable inputs with the same predicted label can be assigned very different interpretations. We systematically characterize the fragility of the interpretations generated by several widely-used feature-importance interpretation methods (saliency maps, integrated gradient, and DeepLIFT) on ImageNet and CIFAR-10. Our experiments show that even small random perturbation can change the feature importance and new systematic perturbations can lead to dramatically different interpretations without changing the label. We extend these results to show that interpretations based on exemplars (e.g. influence functions) are similarly fragile. Our analysis of the geometry of the Hessian matrix gives insight on why fragility could be a fundamental challenge to the current interpretation approaches.

The paper shows that interpretations for DNN decisions, e.g. computed by methods such as sensitivity analysis or DeepLift, are fragile: Visually (to a human) inperceptibly different image cause greatly different explanations (and also to an extent different classifier outputs). The authors perturb input images and create explanations using different methods. Even though the image is inperceptibly different to a human observer, the authors observe large changes in the heatmaps visualizing the explanation maps. This is true even for random perturbations.
The images have been modified wrt. to some noise, such that they deviate from the natural statistics for images of that kind. Since the explanation algorithms investigated in this papers merely react to the interactions of the model to the input and thus are unsupervised processes in nature, the explanation methods merely show the model's reaction to the change.
For one, the model itself reacts to the perturbation, which can be measured by the (considarbly) increased class probability. Since the prediction score is given in probabilty values, the reviewer assumes the final layer of the model is a SoftMax activation. In order to see change in the softmax output score, especially if the already dominant prediction score is further increased, a lot of change has to happen to the outputs of the layer serving as input to the SoftMax layer.
It can thus be expected, that the input- and class specific explanations change as well, to an also not so small extent. The explanation maps mirror for the considered methods the model's reaction to the input. They are thus not meaningless, but are a measure to model reaction instead of an independent process. The excellent Figure 2 supports this point. Not the interpretation itself is fragile, but the model.
Adding a small delta to the sample x shifts its position in data space, completely altering the prediction rule applied by the model due to the change in proximity to another section of the decision hyperplane. The fragility of DNN models to marginally perturbed inputs themselves is well known.
This especially true for adversial perturbations, which have been used as test cases in this work. The explanation methods are expected to highlight highly important areas in an image, which have been targetet by these perturbation approaches.
The authors give an example of an adversary manipulating the input in order to draw the activation to specific features to draw confusing/malignant explanation maps. In a settig of model verification, the explanation via heatmaps is exactly what one wants to have: If tiny change to the image causes lots of change to the prediction (and explanation) we can visualize the instability of the model not the explanation method.
Further do targeted perturbations not show the fragility of explanation methods, but rather that the models actually find what is important to the model. It can be expected, that after a change to these parts of the input, the model will decide differently, albeit coming to the same conclusion (in terms of predicted class membership), which reflects in the explanation map computed for the perturbed input.
Further remarks:
It would be interesting to see the size and position of the center of mass attacks in the appendix. The reviewer closely follows and is experienced with various explanation methods, their application and the quality of the expected explanations. The reviewer is therefore surprised by the poor quality and lack of structure in the maps obtained from the DeepLift method. Can bugs and suboptimal configurations be ruled out during the experiments? The DeepLift explanations are almost as noisy as the ones obtained for Sensitivity Analysis (i.e. the gradient at the input point). However, recent work (e.g. Samek et al., IEEE TNNLS, 2017 or Montavon et al., Digital Signal Processing, 2017) showed that decomposition-based methods (such as DeepLift) provide less noisy explanations than Sensitivity Analysis.
Have the authors considered training the net with small random perturbations added to the samples, to compare the "vanilla" model to the more robust one, which has seen noisy samples, and compared explanations?
Why not train (finetune) the considered models using softplus activations instead of exchanging activation nodes?
Appendix B: Heatmaps through the different stages of perturbation should be normalized using a common factor, not individually, in order to better reflect the change in the explanation
Conclusion:
The paper follows an interesting approach, but ultimately takes the wrong view point:
The authors try to attribute fragility to explaining methods, which visualize/measure the reaction of the model to the perturbed inputs. A major rework should be considered.


Published as a conference paper at ICLR 2018 LEARNING DISCRETE WEIGHTS USING THE LOCAL REPARAMETERIZATION TRICK
Recent breakthroughs in computer vision make use of large deep neural networks, utilizing the substantial speedup offered by GPUs. For applications running on limited hardware, however, high precision real-time processing can still be a challenge. One approach to solving this problem is training networks with binary or ternary weights, thus removing the need to calculate multiplications and significantly reducing memory size. In this work, we introduce LR-nets (Local reparameterization networks), a new method for training neural networks with discrete weights using stochastic parameters. We show how a simple modification to the local reparameterization trick, previously used to train Gaussian distributed weights, enables the training of discrete weights. Using the proposed training we test both binary and ternary models on MNIST, CIFAR-10 and ImageNet benchmarks and reach state-of-the-art results on most experiments.

This paper proposes training binary and ternary weight distribution networks through the local reparametrization trick and continuous optimization. The argument is that due to the central limit theorem (CLT) the distribution on the neuron pre-activations is approximately Gaussian, with a mean given by the inner product between the input and the mean of the weight distribution and a variance given by the inner product between the squared input and the variance of the weight distribution. As a result, the parameters of the underlying discrete distribution can be optimized via backpropagation by sampling the neuron pre-activations with the reparametrization trick. The authors further propose appropriate initialisation schemes and regularization techniques to either prevent the violation of the CLT or to prevent underfitting. The method is evaluated on multiple experiments.
This paper proposed a relatively simple idea for training networks with discrete weights that seems to work in practice. My main issue is that while the authors argue about novelty, the first application of CLT for sampling neuron pre-activations at neural networks with discrete r.v.s is performed at [1]. While [1] was only interested in faster convergence and not on optimization of the parameters of the underlying distribution, the extension was very straightforward. I would thus suggest that the authors update the paper accordingly.
Other than that, I have some other comments:
- The L2 regularization on the distribution parameters for the ternary weights is a bit ad-hoc; why not penalise according to the entropy of the distribution which is exactly what you are trying to achieve?
- For the binary setting you mentioned that you had to reduce the entropy thus added a “beta density regulariser”. Did you add R(p) or log R(p) to the objective function? Also, with alpha, beta = 2 the beta density is unimodal with a peak at p=0.5; essentially this will force the probabilities to be close to 0.5, i.e. exactly what you are trying to avoid. To force the probability near the endpoints you have to use alpha, beta < 1 which results into a “bowl” shaped Beta distribution. I thus wonder whether any gains you observed from this regulariser are just an artifact of optimization.
- I think that a baseline (at least for the binary case) where you learn the weights with a continuous relaxation, such as the concrete distribution, and not via CLT would be helpful. Maybe for the network to properly converge the entropy for some of the weights needs to become small (hence break the CLT).
[1] Wang & Manning, Fast Dropout Training.
Edit: After the authors rebuttal I have increased the rating of the paper:
- I still believe that the connection to [1] is stronger than what the authors allude to; eg. the first two paragraphs of sec. 3.2 could easily be attributed to [1].
- The argument for the entropy was to include a term (- lambda * H(p)) in the objective function with H(p) being the entropy of the distribution p. The lambda term would then serve as an indicator to how much entropy is necessary.
- There indeed was a misunderstanding with the usage of the R(p) regularizer at the objective function (which is now resolved).
- The authors showed benefits compared to a continuous relaxation baseline.


Any autoencoder network can be turned into a generative model by imposing an arbitrary prior distribution on its hidden code vector. Variational Autoencoder uses a KL divergence penalty to impose the prior, whereas Adversarial Autoencoder uses generative adversarial networks. A straightforward modification of Adversarial Autoencoder is to replace the adversary by maximum mean discrepancy (MMD) test. This replacement leads to a new type of probabilistic autoencoder, which is also discussed in our paper.
However, an essential challenge remains in both of these probabilistic autoencoders, namely that the only source of randomness at the output of encoder, is the training data itself. Lack of enough stochasticity can make the optimization problem nontrivial. As a result, they can lead to degenerate solutions where the generator collapses into sampling only a few modes.
Our proposal is to replace the adversary of adversarial autoencoder by a space of stochastic functions. This replacement introduces a a new source of randomness, which can be considered as a continuous control for encouraging explorations. This prevents the adversary from fitting too closely to the generator and therefore leads to more diverse set of generated samples. Consequently, the decoder serves as a better generative network, which unlike MMD nets scales linearly with the amount of data. We provide mathematical and empirical evidence on how this replacement outperforms the pre-existing architectures.

Thank you for the feedback, and I have read it.
The authors claimed that they used techniques in [6] in which I am not an expert for this. However I cannot find the comparison that the authors mentioned in the feedback, so I am not sure if the claim is true.
I still recommend rejection for the paper, and as I said in the first review, the paper is not mature enough.
==== original review ===
The paper describes a generative model that replaces the GAN loss in the adversarial auto-encoder with MMD loss. Although the author claim the novelty as adding noise to the discriminator, it seems to me that at least for the RBF case it just does the following:
1. write down MMD as an integral probability metric (IPM)
2. say the test function, which originally should be in an RKHS, will be approximated using random feature approximations.
Although the authors explained the intuition a bit and showed some empirical results, I still don't see why this method should work better than directly minimising MMD. Also it is not preferred to look at the generated images and claim diversity, instead it's better to have some kind of quantitative metric such as the inception score.
Finally, given the fact that we have too many GAN related papers now, I don't think the innovation contained in the paper (which is using random features) is good enough to be published at ICLR. Also the paper is not clearly written, and I would suggest better not to copy-past paragraphs in the abstract and intro.
That said, I would welcome for the authors feedback and see if I have misunderstood something.


This article proposes to auto-encode text at byte-level using convolutional networks with a recursive architecture. The motivation is to explore whether it is possible to have scalable and homogeneous text generation at byte-level in a nonsequential fashion through the simple task of auto-encoding. We show that nonsequential text generation from a fixed-length representation is not only possible, but also achieved much better auto-encoding results than recurrent networks. The proposed model is a multi-stage deep convolutional encoder-decoder framework using residual connections (He et al., 2016), containing up to 160 parameterized layers. Each encoder or decoder contains a shared group of modules that consists of either pooling or upsampling layers, making the network recursive in terms of abstraction levels in representation. Results for 6 large-scale paragraph datasets are reported, in 3 languages including Arabic, Chinese and English. Analyses are conducted to study several properties of the proposed model.
Recently, generating text using convolutional networks (ConvNets) starts to become an alternative to recurrent networks for sequence-tosequence learning (Gehring et al., 2017). The dominant assumption for both these approaches is that texts are generated one word at a time. Such sequential generation process bears the risk of output or gradient vanishing or exploding problem (Bengio et al., 1994), which limits the length of its generated results. Such limitation in scalability prompts us to explore whether non-sequential text generation is possible.
Meanwhile, text processing from lower levels than words -such as characters (Zhang et al., 2015)  (Kim et al., 2016) and bytes (Gillick et al., 2016) (Zhang & LeCun, 2017) -is also being explored due to its promise in handling distinct languages in the same fashion. In particular, the work by Zhang & LeCun (2017) shows that simple one-hot encoding on bytes could give the best results for text classification in a variety of languages. The reason is that it achieved the best balance between computational performance and classification accuracy. Inspired by these results, this article explores auto-encoding for text using byte-level convolutional networks that has a recursive structure, as a first step towards low-level and non-sequential text generation.
For the task of text auto-encoding, we should avoid the use of common attention mechanisms like those used in machine translation Bahdanau et al. (2015), because they always provide a direct information path that enables the auto-encoder to directly copy from the input. This diminishes the Figure 2: Sequential and non-sequential decoders illustrated in graphical models. u is a vector containing encoded representation. y i 's are output entities. h i 's are hidden representations. Note that they both imply conditional independence between outputs conditioned by the representation u.
purpose of studying the representational ability of different models. Therefore, all models considered in this article would encode to and decode from a fixed-length vector representation.
The paper by  was an anterior result on using word-level convolutional networks for text auto-encoding. This article differs from it in several key ways of using convolutional networks. First of all, our models work from the level of bytes instead of words, which arguably makes the problem more challenging. Secondly, our network is dynamic with a recursive structure that scales with the length of input text, which by design could avoid trivial solutions for auto-encoding such as the identity function. Thirdly, by using the latest design heuristics such as residual connections (He et al., 2016), our network can scale up to several hundred of layers deep, compared to a static network that contains a few layers.
In this article, several properties of the auto-encoding model are studied. The following is a list.
1. Applying the model to 3 languages -Arabic, Chinese and English -shows that the model can handle all different languages in the same fashion with equally good accuracy. 2. Comparisons with long short-term memory (LSTM) (Hochreiter & Schmidhuber, 1997) show a significant advantage of using convolutional networks for text auto-encoding. 3. We determined that a recursive convolutional decoder like ours can accurately produce the end-of-string byte, despite that the decoding process is non-sequential. 4. By studying the auto-encoding error when the samples contain randomized noisy bytes, we show that the model does not degenerate to the identity function. However, it can neither denoise the input very well. 5. The recursive structure requires a pooling layer. We compared between average pooling, L2 pooling and max-pooling, and determined that max-pooling is the best choice. 6. The advantage of recursion is established by comparison against a static model that does not have shared module groups. This shows that linguistic heuristics such as recursion is useful for designing models for language processing. 7. We also explored models of different sizes by varying the maximum network depth from 40 to 320. The results show that deeper models give better results.

The paper aims to illustrated the representation learning ability of the convolutional autoencoder with residual connections is  proposed by to encode text at the byte level.  The authors apply the proposed architecture to 3 languages and run comparisons with an LSTM.  Experimental results  with different perturbation of samples, pooling layers, and sample lengths are presented.
The writing is fairly clear, however the presentation of tables and figures could be done better, for example, Fig. 2 is referred  to in page 3,  Table 2 which contains results is referred to on page 5, Fig 4 is referred to in page 6 and appears in page 5, etc.
What kind of minimal preprocessing is done on the text? Are punctuations removed? Is casing retained? How is the space character encoded?
Why was the encoded dimension always fixed at 1024?  What is the definition of a sample here?
The description of the various data sets could be moved to a table/Appendix, particularly since most of the results are presented on the enwiki dataset, which would lead to better readability of the paper.  Also results are presented only on a random 1M sample selected from these data sets, so the need for this whole page goes away.
Comparing Table 2 and Table 3, the LSTM is at 67% error on the test set while the proposed convolutional autoencoder is at 3.34%.  Are these numbers on the same test set?    While the argument that the LSTM does not generalize well due to the inherent memory learnt is reasonable, the differences in performance cannot be explained away with this. Can you please clarify this further?
It appears that the byte error shoot up for sequences of length 512+ (fig. 6 and fig. 7) and seems entirely correlated with the amount of data than recursion levels.
How do you expect these results to change for a different subset selection of training and test samples? Will Fig. 7 and Fig. 6 still hold?
In Fig, 8, unless the static train and test error are exactly on top of the  recursive errors, they are not visible.  What is the x-axis in Fig. 8?  Please also label axes on all figures.
While the datasets are large and would take a lot of time to process for each case study, a final result on the complete data set, to illustrate if the model does learn well with lots of data would have been useful.  A table showing generated sample text would also clarify the power of the model.
With the results presented,  with a single parameter setting, its hard to determine what exactly the model learns and why.


Ubuntu dialogue corpus is the largest public available dialogue corpus to make it feasible to build end-to-end deep neural network models directly from the conversation data. One challenge of Ubuntu dialogue corpus is the large number of out-of-vocabulary words. In this paper we proposed a method which combines the general pre-trained word embedding vectors with those generated on the taskspecific training set to address this issue. We integrated character embedding into Chen et al's Enhanced LSTM method (ESIM) and used it to evaluate the effectiveness of our proposed method. For the task of next utterance selection, the proposed method has demonstrated a significant performance improvement against original ESIM and the new model has achieved state-of-the-art results on both Ubuntu dialogue corpus and Douban conversation corpus. In addition, we investigated the performance impact of end-of-utterance and end-of-turn token tags.

The main contributions in this paper are:
1) New variants of a recent LSTM-based model ("ESIM") are applied to the task of response-selection in dialogue modeling -- ESIM was originally introduced and evaluated for natural language inference. In this new setting, the ESIM model (vanilla and extended) outperform previous models when trained and evaluated on two distinct conversational datasets.
2) A fairly trivial method is proposed to extend the coverage of pre-trained word embeddings to deal with the OOV problem that arises when applying them to these conversational datasets.
The method itself is to combine d1-dimensional word embeddings that were pretrained on a large unannotated corpus (vocabulary S) with distinct d2-dimensional word embeddings that are trained on the task-specific training data (vocabulary T). The enhanced (d1+d2)-dimensional representation for a word is constructed by concatenating its vectors from the two embeddings, setting either the d1- or d2-dimensional subvector to zeros when the word is absent from either S or T, respectively. This method is incorporated as an extension into ESIM and evaluated on the two conversation datasets.
The main results can be characterized as showing that this vocabulary extension method leads to performance gains on two datasets, on top of an ESIM-model extended with character-based word embeddings, which itself outperforms the vanilla ESIM model.
These empirical results are potentially meaningful and could justify reporting, but the paper's organization is very confusing, and too many details are too unclear, leading to low confidence in reproducibility.
There is basic novelty in applying the base model to a new task, and the analysis of the role of the special conversational boundary tokens is interesting and can help to inform future modeling choices. The embedding-enhancing method has low originality but is effective on this particular combination of model architecture, task and datasets. I am left wondering how well it might generalize to other models or tasks, since the problem it addresses shows up in many other places too...
Overall, the presentation switches back and forth between the Douban corpus and the Ubuntu corpus, and between word2vec and Glove embeddings, and this makes it very challenging to understand the details fully.
S3.1 - Word representation layer: This paragraph should probably mention that the character-composed embeddings are newly introduced here, and were not part of the original formulation of ESIM. That statement is currently hidden in the figure caption.
Algorithm 1:
- What set does P denote, and what is the set-theoretic relation between P and T?
- Under one possible interpretation, there may be items in P that are in neither T nor S, yet the algorithm does not define embeddings for those items even though its output is described as "a dictionary with word embeddings ... for P". This does not seem consistent? I think the sentence in S4.2 about initializing remaining OOV words as zeros is relevant and wonder if it should form part of the algorithm description?
S4.1 - What do the authors mean by the statement that response candidates for the Douban corpus were "collected by Lucene retrieval model"?
S4.2 - Paragraph two is very unclear. In particular, I don't understand the role of the Glove vectors here when Algorithm 1 is used, since the authors refer to word2vec vectors later in this paragraph and also in the Algorithm description.
S4.3 - It's insufficiently clear what the model definitions are for the Douban corpus. Is there still a character-based LSTM involved, or does FastText make it unnecessary?
S4.3 - "It can be seen from table 3 that the original ESIM did not perform well without character embedding." This is a curious way to describe the result, when, in fact, the ESIM model in table 3 already outperforms all the previous models listed.
S4.4 - gensim package -- for the benefit of readers unfamiliar with gensim, the text should ideally state explicitly that it is used to create the *word2vec* embeddings, instead of the ambiguous "word embeddings".


Background: Statistical mechanics results (Dauphin et al. (2014); Choromanska et al. (2015)) suggest that local minima with high error are exponentially rare in high dimensions. However, to prove low error guarantees for Multilayer Neural Networks (MNNs), previous works so far required either a heavily modified MNN model or training method, strong assumptions on the labels (e.g., "near" linear separability), or an unrealistically wide hidden layer with Ω (N ) units. Results: We examine a MNN with one hidden layer of piecewise linear units, a single output, and a quadratic loss. We prove that, with high probability in the limit of N → ∞ datapoints, the volume of differentiable regions of the empiric loss containing sub-optimal differentiable local minima is exponentially vanishing in comparison with the same volume of global minima, given standard normal input of dimension d 0 =Ω √ N , and a more realistic number of
hidden units. We demonstrate our results numerically: for example, 0% binary classification training error on CIFAR with only N/d 0 ≈ 16 hidden neurons.

## Summary
This paper aims to tackle the question: "why does standard SGD based algorithms on neural network converge to 'good' solutions?"
Pros:
Authors ask the question of convergence of optimization (ignoring generalization error): how "likely" is that an over-parameterized (d1d0 > N) single hidden layer binary classifier "find" a good (possibly over-fitted) local minimum. They make a set of assumptions (A1-A3) which are weaker (d1 > N^{1/2}) than the ones used earlier works. Previous works needed a wide hidden layer (d1 > N).
Assumptions (d0=input dim, d1=hidden dim, N=n of datapoints, X=datapoints matrix):
A1. Datapoints X come from a Gaussian distribution
A2. N^{1/2} < d0 =< N
A3. N polylog(N) < d0d1 (approximate n of. parameters)  and d1 =< N
This paper proves that total "angular volume" of "regions" (defined with respect to the piecewise linear regions of neuron activations) with differentiable bad-local minima are exponentially small when compared with to the total "angular volume" of "regions" containing only differentiable global-minimal. The proof boils down to counting arguments and concentration inequality.
Cons:
Non-differentiable stationary points are left as a challenging future work on this paper. Non-differentiability aside, authors show a possible way by which shallow neural networks might be over-fitting the data. But this is only half the story and does not completely answer the question. First, exponentially vanishing (in N) volume of the "regions" containing bad-local minima doesn't mean that the number of bad local minima are exponentially small when compared to number global minima.
Secondly, as the authors aptly pointed out in the discussion section, this results doesn't mean neural networks will converge to good local minima because these bad local minimas can have a large basins of attraction.
Lastly, appropriate comparisons with the existing literature is lacking. It is hinted that this paper is more general as the assumptions are more realistic. However, it comes at a cost of losing sharpness in the theoretical results. It is not well motivated why one should study the angular volume of the global and local minima.
## Questions and comments
1. How critical is Gaussian-datapoints assumption (A1)? Which part of the proof fails to generalize?
2. Can the proof be extended to scalar regression?  It seems hard to generalize to vector output neural networks. What about deep neural networks?
3. Can you relate the results to other more recent works like: https://arxiv.org/pdf/1707.04926.pdf.
4. Piecewise linear and positively homogeneous (https://arxiv.org/pdf/1506.07540.pdf) activation seem to be important assumption of the paper. It should probably be mentioned explicitly.
5. In the experiments section, it is mentioned that "...inputs to the hidden neurons converge to a distinctly non-zero value. This indicates we converged to DLMs." How can you guarantee that it is a local minimum and not a saddle point?


We study the control of symmetric linear dynamical systems with unknown dynamics and a hidden state. Using a recent spectral filtering technique for concisely representing such systems in a linear basis, we formulate optimal control in this setting as a convex program. This approach eliminates the need to solve the nonconvex problem of explicit identification of the system and its latent state, and allows for provable optimality guarantees for the control signal. We give the first efficient algorithm for finding the optimal control signal with an arbitrary time horizon , with sample complexity (number of training rollouts) polynomial only in log and other relevant parameters.

This paper proposes a new algorithm to generate the optimal control inputs for unknown linear dynamical systems (LDS) with known system dimensions.
The idea is exciting LDS by wave filter inputs and record the output and directly estimate the operator that maps the input to the output instead of estimating the hidden states. After obtaining this operator, this paper substitutes this operator to the optimal control problem and solve the optimal control problem to estimate the optimal control input, and show that the gap between the true optimal cost and the cost from applying estimated optimal control input is small with high probability.
I think estimating the operator from the input to the output is interesting, instead of constructing (A, B, C, D) matrices, but this idea and all the techniques are from Hazan et. el., 2017. After estimating this operator, it is straightforward to use this to generate the estimated optimal control input. So I think the idea is OK, but not a breakthrough.
Also I found the symmetric matrix assumption on A is quite limited. This limitation is from Hazan et. el., 2017, where the authors wants to predict the output. For prediction purposes, this restriction might be OK, but for control purposes, many interesting plants does not satisfy this assumption, even simple RL circuit. I agree with authors that this is an attempt to combine system identification with generating control inputs together, but I am not sure how to remove the restriction on A.
Dean et. el., 2017 also pursued this direction by combining system identification with robust controller synthesis to handle estimation errors in the system matrices (A, B) in the state-feedback case (LQR), and I can see that Dean et. el. could be extended to handle observer-feedback case (LQG) without any restriction.
Despite of this limitation I think the paper's idea is OK and the result is worth to be published but not in the current form. The paper is not clearly written and there are several areas need to be improved.
1. System identification.
Subspace identification (N4SID) won't take exponential time. I recommend the authors to perform either proper literature review or cite one or two papers on the time complexity and their weakness. Also note that subspace identification can estimate (A, B, C, D) matrices which is great for control purposes especially for the infinite horizon LQR.
2. Clarification on the unit ball constraints.
Optimal control inputs are restricted to be inside the unit ball and overall norm is bounded by L. Where is this restriction coming from? The standard LQG setup does not have this restriction.
3. Clarification on the assumption (3).
Where is this assumption coming from? I can see that this makes the analysis go through but is this a reasonable assumption?  Does most of system satisfy this constraint? Is there any? It's ok not to provide the answer if it's hard to analyze, but if that's the case the paper should provide some numerical case studies to show this bound either holds or the gap is negligible in the toy example.
4. Proof of theorem 3.3.
Theorem 3.3 is one of the key results in this paper, yet its proof is just "noted". The setup is slightly different from the original theorem in Hazan et. el., 2017 including the noise model, so I strongly recommend to include the original theorem in the appendix, and include the full proof in the appendix.
5. Proof of lemma 3.1.
I found it's hard to keep track of which one is inside the expectation. I recommend to follow the notation E[variable] the authors been using throughout the paper in the proof instead of dropping these brackets.

6. Minor typos
In theorem 2.4, ||Q||_op is used for defining rho, but in the text ||Q||_F is used. I think ||Q||_op is right.


A common way to speed up training of deep convolutional networks is to add computational units. Training is then performed using data-parallel synchronous Stochastic Gradient Descent (SGD) with a mini-batch divided between computational units. With an increase in the number of nodes, the batch size grows. However, training with a large batch often results in lower model accuracy. We argue that the current recipe for large batch training (linear learning rate scaling with warm-up) does not work for many networks, e.g. for Alexnet, Googlenet,... We propose a more general training algorithm based on Layer-wise Adaptive Rate Scaling (LARS). The key idea of LARS is to stabilize training by keeping the magnitude of update proportional to the norm of weights for each layer. This is done through gradient rescaling per layer. Using LARS, we successfully trained AlexNet and ResNet-50 to a batch size of 16K.

This paper proposes a training algorithm based on Layer-wise Adaptive Rate Scaling (LARS) to overcome the optimization difficulties for training with large batch size. The authors use a linear scaling and warm-up scheme to train AlexNet on ImageNet. The results show promising performance when using a relatively large batch size. The presented method is interesting. However, the experiments are poorly organized since some necessary descriptions and discussions are missing. My detailed comments are as follows.
Contributions：
1.	The authors propose a training algorithm based LARS with the adaptive learning rate for each layer, and train the AlexNet and ResNet-50 to a batch size of 16K.
2.	The training method shows stable performance and helps to avoid gradient vanishing or exploding.
Weak points:
The training algorithm does not overcome the optimization difficulties when the batch size becomes larger (e.g. 32K), where the training becomes unstable, and the training based on LARS and warm-up can’t improve the accuracy compared to the baselines.
Specific comments:
1.	In Algorithm 1, how to choose $ \eta $ and $ \beta $ in the experiment?
2.	Under the line of Equation (3), $ \nabla L(x_j, w_{t+1}) \approx L(x_j, w_{t}) $ should be $ \nabla L(x_j, w_{t+1}) \approx \nabla L(x_j, w_{t}) $.
3.	How can the training algorithm based on LARS improve the generalization for the large batch?
4.	In the experiments, what is the parameter iter_size? How to choose it?
5.	In the experiments, no descriptions and discussions are given for Table 3, Figure 4, Table 4, Figure 5, Table 5 and Table 6. The authors should give more discussions on these tables and figures. Furthermore, the captions of these tables and figures confusing.
6.	On page 4, there is a statement “The ratio is high during the initial phase, and it is rapidly decreasing after few epochs (see Figure 2).” This is quite confusing, since Figure 2 is showing the change of learning rates w.r.t. training epochs.


DISTRIBUTED FINE-TUNING OF LANGUAGE MODELS ON PRIVATE DATA
One of the big challenges in machine learning applications is that training data can be different from the real-world data faced by the algorithm. In language modeling, users' language (e.g. in private messaging) could change in a year and be completely different from what we observe in publicly available data. At the same time, public data can be used for obtaining general knowledge (i.e. general model of English). We study approaches to distributed fine-tuning of a general model on user private data with the additional requirements of maintaining the quality on the general data and minimization of communication costs. We propose a novel technique that significantly improves prediction quality on users' language compared to a general model and outperforms gradient compression methods in terms of communication efficiency. The proposed procedure is fast and leads to an almost 70% perplexity reduction and 8.7 percentage point improvement in keystroke saving rate on informal English texts. Finally, we propose an experimental framework for evaluating differential privacy of distributed training of language models and show that our approach has good privacy guarantees.

This paper deals with improving language models on mobile equipments
based on small portion of text that the user has ever input. For this
purpose, authors employed a linearly interpolated objectives between user
specific text and general English, and investigated which method (learning
without forgetting and random reheasal) and which interepolation works better.
Moreover, authors also look into privacy analysis to guarantee some level of
differential privacy is preserved.
Basically the motivation and method is good, the drawback of this paper is
its narrow scope and lack of necessary explanations. Reading the paper,
many questions arise in mind:
- The paper implicitly assumes that the statistics from all the users must
  be collected to improve "general English". Why is this necessary? Why not
  just using better enough basic English and the text of the target user?
- To achieve the goal above, huge data (not the "portion of the general English") should be communicated over the network. Is this really worth doing? If only
  "the portion of" general English must be communicated, why is it validated?
- For measuring performance, authors employ keystroke saving rate. For the
  purpose of mobile input, this is ok: but the use of language models will
  cover much different situation where keystrokes are not necessarily
  available, such as speech recognition or machine translation. Since this
  paper is concerned with a general methodology of language modeling,
  perplexity improvement (or other criteria generally applicable) is also
  important.
- There are huge number of previous work on context dependent language models,
  let alone a mixture of general English and specific models. Are there any
  comparison with these previous efforts?
Finally, this research only relates to ICLR in that the language model employed
is LSTM: in other aspects, it easily and better fit to ordinary NLP conferences, such as EMNLP, NAACL or so. I would like to advise the authors to submit
this work to such conferences where it will be reviewed by more NLP experts.
Minor:
- t of $G_t$ in page 2 is not defined so far.
- What is "gr" in Section 2.2?


PARAMETER SPACE NOISE FOR EXPLORATION
Deep reinforcement learning (RL) methods generally engage in exploratory behavior through noise injection in the action space. An alternative is to add noise directly to the agent's parameters, which can lead to more consistent exploration and a richer set of behaviors. Methods such as evolutionary strategies use parameter perturbations, but discard all temporal structure in the process and require significantly more samples. Combining parameter noise with traditional RL methods allows to combine the best of both worlds. We demonstrate that both off-and on-policy methods benefit from this approach through experimental comparison of DQN, DDPG, and TRPO on high-dimensional discrete action environments as well as continuous control tasks.

This paper proposes a method for parameter space noise in exploration.
Rather than the "baseline" epsilon-greedy (that sometimes takes a single action at random)... this paper presents an method for perturbations to the policy.
In some domains this can be a much better approach and this is supported by experimentation.
There are several things to like about the paper:
- Efficient exploration is a big problem for deep reinforcement learning (epsilon-greedy or Boltzmann is the de-facto baseline) and there are clearly some examples where this approach does much better.
- The noise-scaling approach is (to my knowledge) novel, good and in my view the most valuable part of the paper.
- This is clearly a very practical and extensible idea... the authors present good results on a whole suite of tasks.
- The paper is clear and well written, it has a narrative and the plots/experiments tend to back this up.
- I like the algorithm, it's pretty simple/clean and there's something obviously *right* about it (in SOME circumstances).
However, there are also a few things to be cautious of... and some of them serious:
- At many points in the paper the claims are quite overstated. Parameter noise on the policy won't necessarily get you efficient exploration... and in some cases it can even be *worse* than epsilon-greedy... if you just read this paper you might think that this was a truly general "statistically efficient" method for exploration (in the style of UCRL or even E^3/Rmax etc).
- For instance, the example in 4.2 only works because the optimal solution is to go "right" in every timestep... if you had the network parameterized in a different way (or the actions left/right were relabelled) then this parameter noise approach would *not* work... By contrast, methods such as UCRL/PSRL and RLSVI https://arxiv.org/abs/1402.0635 *are* able to learn polynomially in this type of environment. I think the claim/motivation for this example in the bootstrapped DQN paper is more along the lines of "deep exploration" and you should be clear that your parameter noise does *not* address this issue.
- That said I think that the example in 4.2 is *great* to include... you just need to be more upfront about how/why it works and  what you are banking on with the parameter-space exploration. Essentially you perform a local exploration rule in parameter space... and sometimes this is great - but you should be careful to distinguish this type of method from other approaches. This must be mentioned in section 4.2 "does parameter space noise explore efficiently" because the answer you seem to imply is "yes" ... when the answer is clearly NOT IN GENERAL... but it can still be good sometimes ;D
- The demarcation of "RL" and "evolutionary strategies" suggests a pretty poor understanding of the literature and associated concepts. I can't really support the conclusion "RL with parameter noise exploration learns more efficiently than both RL and evolutionary strategies individually". This sort of sentence is clearly wrong and for many separate reasons:
    - Parameter noise exploration is not a separate/new thing from RL... it's even been around for ages! It feels like you are talking about DQN/A3C/(whatever algorithm got good scores in Atari last year) as "RL" and that's just really not a good way to think about it.
    - Parameter noise exploration can be *extremely* bad relative to efficient exploration methods (see section 2.4.3 https://searchworks.stanford.edu/view/11891201)
Overall, I like the paper, I like the algorithm and I think it is a valuable contribution.
I think the value in this paper comes from a practical/simple way to do policy randomization in deep RL.
In some (maybe even many of the ones you actually care about) settings this can be a really great approach, especially when compared to epsilon-greedy.
However, I hope that you address some of the concerns I have raised in this review.
You shouldn't claim such a universal revolution to exploration / RL / evolution because I don't think that it's correct.
Further, I don't think that clarifying that this method is *not* universal/general really hurts the paper... you could just add a section in 4.2 pointing out that the "chain" example wouldn't work if you needed to do different actions at each timestep (this algorithm does *not* perform "deep exploration").
I vote accept.


VARIANCE REDUCTION FOR POLICY GRADIENT WITH ACTION-DEPENDENT FACTORIZED BASELINES
Policy gradient methods have enjoyed great success in deep reinforcement learning but suffer from high variance of gradient estimates. The high variance problem is particularly exasperated in problems with long horizons or high-dimensional action spaces. To mitigate this issue, we derive a bias-free action-dependent baseline for variance reduction which fully exploits the structural form of the stochastic policy itself and does not make any additional assumptions about the MDP. We demonstrate and quantify the benefit of the action-dependent baseline through both theoretical analysis as well as numerical results, including an analysis of the suboptimality of the optimal state-dependent baseline. The result is a computationally efficient policy gradient algorithm, which scales to high-dimensional control problems, as demonstrated by a synthetic 2000-dimensional target matching task. Our experimental results indicate that action-dependent baselines allow for faster learning on standard reinforcement learning benchmarks and highdimensional hand manipulation and synthetic tasks. Finally, we show that the general idea of including additional information in baselines for improved variance reduction can be extended to partially observed and multi-agent tasks.

The paper proposes a variance reduction technique for policy gradient methods. The proposed approach justifies the utilization of action-dependent baselines, and quantifies the gains achieved by it over more general state-dependent or static baselines.
The writing and organization of the paper is very well done. It is easy to follow, and succinct while being comprehensive. The baseline definition is well-motivated, and the benefits offered by it are quantified intuitively. There is only one mostly minor issues with the algorithm development and the experiments need to be more polished.
For the algorithm development, there is an relatively strong assumption that z_i^T z_j = 0. This assumption is not completely unrealistic (for example, it is satisfied if completely separate parts of a feature vector are used for actions). However, it should be highlighted as an assumption, and it should be explicitly stated as z_i^T z_j = 0 rather than z_i^T z_j approx 0. Further, because it is relatively strong of an assumption, it should be discussed more thoroughly, with some explicit examples of when it is satisfied.
Otherwise, the idea is simple and yet effective, which is exactly what we would like for our algorithms. The paper would be a much stronger contribution, if the experiments could be improved.
- More details regarding the experiments are desirable - how many runs were done, the initialization of the policy network and action-value function, the deep architecture used etc.
- The experiment in Figure 3 seems to reinforce the influence of \lambda as concluded by the Schulman et. al. paper. While that is interesting, it seems unnecessary/non-relevant here, unless performance with action-dependent baselines with each value of \lambda is contrasted to the state-dependent baseline. What was the goal here?
- In general, the graphs are difficult to read; fonts should be improved and the graphs polished.
- The multi-agent task needs to be explained better - specifically how is the information from the other agent incorporated in an agent's baseline?
- It'd be great if Plot (a) and (b) in Figure 5 are swapped.
Overall I think the idea proposed in the paper is beneficial. Better discussing the strong theoretical assumption should be incorporated. Adding the listed suggestions to the experiments section would really help highlight the advantage of the proposed baseline in a more clear manner. Particularly with some clarity on the experiments, I would be willing to increase the score.
Minor comments:
1. In Equation (28) how is the optimal-state dependent baseline obtained? This should be explicitly shown, at least in the appendix.
2. The listed site for videos and additional results is not active.
3. Some typos
- Section 2 - 1st para - last line: "These methods are therefore usually more sample efficient, but can be less stable than critic-based methods.".
- Section 4.1 - Equation (7) - missing subscript i for b(s_t,a_t^{-i})
- Section 4.2 - \hat{Q} is just Q in many places