Published as a conference paper at ICLR 2018 ON THE EXPRESSIVE POWER OF OVERLAPPING ARCHITECTURES OF DEEP LEARNING
Expressive efficiency refers to the relation between two architectures A and B, whereby any function realized by B could be replicated by A, but there exists functions realized by A, which cannot be replicated by B unless its size grows significantly larger. For example, it is known that deep networks are exponentially efficient with respect to shallow networks, in the sense that a shallow network must grow exponentially large in order to approximate the functions represented by a deep network of polynomial size. In this work, we extend the study of expressive efficiency to the attribute of network connectivity and in particular to the effect of "overlaps" in the convolutional process, i.e., when the stride of the convolution is smaller than its filter size (receptive field). To theoretically analyze this aspect of network's design, we focus on a well-established surrogate for ConvNets called Convolutional Arithmetic Circuits (ConvACs), and then demonstrate empirically that our results hold for standard ConvNets as well. Specifically, our analysis shows that having overlapping local receptive fields, and more broadly denser connectivity, results in an exponential increase in the expressive capacity of neural networks. Moreover, while denser connectivity can increase the expressive capacity, we show that the most common types of modern architectures already exhibit exponential increase in expressivity, without relying on fully-connected layers.

The paper studies convolutional neural networks where the stride is smaller than the convolutional filter size; the so called overlapping convolutional architectures. The main object of study is to quantify the benefits of overlap in convolutional architectures.
The main claim of the paper is Theorem 1, which is that overlapping convolutional architectures are efficient with respect to non-overlapping architectures, i.e., there exists functions in the overlapping architecture which require an exponential increase in size to be represented in the non-overlapping architecture; whereas overlapping architecture can capture within a linear size the functions represented by the non-overlapping architectures. The main workhorse behind the paper is the notion of rank of matricized grid tensors following a paper of Cohen and Shashua which capture the relationship between the inputs and the outputs, the function implemented by the neural network.
(1) The results of the paper hold only for product pooling and linear activation function except for the representation layer, which allows general functions. It is unclear why the generalized convolutional networks are stated with such generality when the results apply only to this special case. That this is the case should be made clear in the title and abstract. The paper makes a point that generalized tensor decompositions can be potentially applied to solve the more general case, but since it is left as future work, the paper should make it clear throughout.
(2) The experiment is minimal and even the given experiment is not described well. What data augmentation was used for the CIFAR-10 dataset? It is only mentioned that the data is augmented with translations and horizontal flips. What is the factor of augmentation? How much translation? These are important because there maybe a much simpler explanation to the benefit of overlap: it is able to detect these translated patterns easily. Indeed, this simple intuition seems to be why the authors chose to make the problem by introducing translations and flips.
(3) It is unclear if the paper resolves the mystery that they set out to solve, which is a reconciliation of the following two observations (a) why are non-overlapping architectures so common? (b) why only slight overlap is used in practice?  The paper seems to claim that since overlapping architectures have higher expressivity that answers (a). It appears that the paper does not answer (b) well: it points out that since there is exponential increase, there is no reason to increase it beyond a particular point. It seems the right resolution will be to show that after the overlap is set to a certain small value, there will be *only* linear increase with increasing overlap; i.e., the paper should show that small overlap networks are efficient with respect to *large* overlap networks; a comparison that does not seem to be made in the paper.
(4) Small typo: the dimensions seem to be wrong in the line below the equation in page 3.
The paper makes important progress on a highly relevant problem using a new methodology (borrowed from a previous paper). However, the writing is hurried and the high-level conclusions are not fully supported by theory and experiments.


State of the art computer vision models have been shown to be vulnerable to small adversarial perturbations of the input. In other words, most images in the data distribution are both correctly classified by the model and are very close to a visually similar misclassified image. Despite substantial research interest, the cause of the phenomenon is still poorly understood and remains unsolved. We hypothesize that this counter intuitive behavior is a naturally occurring result of the high dimensional geometry of the data manifold. As a first step towards exploring this hypothesis, we study a simple synthetic dataset of classifying between two concentric high dimensional spheres. For this dataset we show a fundamental tradeoff between the amount of test error and the average distance to nearest error. In particular, we prove that any model which misclassifies a small constant fraction of a sphere will be vulnerable to adversarial perturbations of size
. Surprisingly, when we train several different architectures on this dataset, all of their error sets naturally approach this theoretical bound. As a result of the theory, the vulnerability of neural networks to small adversarial perturbations is a logical consequence of the amount of test error observed. We hope that our theoretical analysis of this very simple case will point the way forward to explore how the geometry of complex real-world data sets leads to adversarial examples.

The idea of analyzing a simple synthetic data set to get insights into open issues about adversarial examples has merit. However, the results reported here are not sufficiently significant for ICLR.
The authors make a big deal throughout the paper about how close to training data the adversarial examples they can find on the data manifold are. E.g.: “Despite being extremely rare, these misclassifications appear close to randomly sampled points on the sphere.”  They report mean distance to nearest errors on the data manifold is 0.18 whereas mean distance between two random points on inner sphere is 1.41. However, distance between two random points on the sphere is not the right comparison. The mean distance between random nearest neighbors from the training samples would be much more appropriate.
They also stress in the Conclusions their Conjecture 5.1 that under some assumptions “the average distance to nearest error may decrease on the order of O(1 / d) as the input dimension grows large.” However, earlier they admitted that “Whether or not a similar conjecture holds for image manifolds is unclear and should be investigated in future work.” So, the practical significance of this conjecture is unclear.  Furthermore, it is well known that in high dimensions, the distances between pairs of training samples tends towards a large constant (e.g. making nearest neighbor search using triangular inequality pruning infeasible), so extreme care much be taken to not over generalize any results from these sorts of synthetic high dimensional experiments.
Authors note that for higher dimensional spheres, adversarial examples on the manifold (sphere shell) could found, but not smaller d:  “In our experiments the highest dimension we were able to train the ReLU net without adversarial examples seems to be around d = 60.”  Yet,in their later statement in that same paragraph  “We did not investigate if larger networks will work for larger d.”, it is unclear what is meant by “will work”; because, presumably, larger networks (with more weights) would be HARDER to avoid adversarial examples being found on the data manifold, so larger networks should be less likely “to work”, if “work” means avoid adversarial examples.  In any case, their apparent use of only h=1000 unit networks (for both ReLU and quadratic cases) is disappointing, because it is not clear whether the phenomena observed would be qualitatively similar for different fully-separable discriminants (e.g. different h values with different regularization costs even if all such networks had zero classification errors).
The authors repeat the following exact same phrase in both the Introduction and the Conclusion:
“Our results highlight the fact that the epsilon norm ball adversarial examples often studied in defence papers are not the real problem but are rather a tractable research problem. “
But it is not clear exactly what the authors meant by this. Also, the term “epsilon norm ball” is not commonly used in adversarial literature, and the only reference to such papers is Madry et al, (2017), which is only on ArXiv and not widely known — if these types of adversarial examples are “often studied” as claimed, there should be other / more established references to cite here.
In short, this work addresses the important problem of better understanding adversarial examples, but the simple setup has a higher burden to establish significance, which this paper as written has not met.


THE IMPLICIT BIAS OF GRADIENT DESCENT ON SEPA- RABLE DATA
We show that gradient descent on an unregularized logistic regression problem, for almost all separable datasets, converges to the same direction as the max-margin solution. The result generalizes also to other monotone decreasing loss functions with an infimum at infinity, and we also discuss a multi-class generalizations to the cross entropy loss. Furthermore, we show this convergence is very slow, and only logarithmic in the convergence of the loss itself. This can help explain the benefit of continuing to optimize the logistic or cross-entropy loss even after the training error is zero and the training loss is extremely small, and, as we show, even if the validation loss increases. Our methodology can also aid in understanding implicit regularization in more complex models and with other optimization methods.

The paper offers a formal proof that gradient descent on the logistic
loss converges very slowly to the hard SVM solution in the case where
the data are linearly separable. This result should be viewed in the
context of recent attempts at trying to understand the generalization
ability of neural networks, which have turned to trying to understand
the implicit regularization bias that comes from the choice of
optimizer. Since we do not even understand the regularization bias of
optimizers for the simpler case of linear models, I consider the paper's
topic very interesting and timely.
The overall discussion of the paper is well written, but on a more
detailed level the paper gives an unpolished impression, and has many
technical issues. Although I suspect that most (or even all) of these
issues can be resolved, they interfere with checking the correctness of
the results. Unfortunately, in its current state I therefore do not
consider the paper ready for publication.
Technical Issues:
The statement of Lemma 5 has a trivial part and for the other part the
proof is incorrect: Let x_u = ||nabla L(w(u))||^2.
  - Then the statement sum_{u=0}^t x_u < infinity is trivial, because
    it follows directly from ||nabla L(w(u))||^2 < infinity for all u. I
    would expect the intended statement to be sum_{u=0}^infinity x_u <
    infinity, which actually follows from the proof of the lemma.
  - The proof of the claim that t*x_t -> 0 is incorrect: sum_{u=0}^t x_u
    < infinity does not in itself imply that t*x_t -> 0, as claimed. For
    instance, we might have x_t = 1/i^2 when t=2^i for i = 1,2,... and
    x_t = 0 for all other t.
Definition of tilde{w} in Theorem 4:
  - Why would tilde{w} be unique? In particular, if the support vectors
    do not span the space, because all data lie in the same
    lower-dimensional hyperplane, then this is not the case.
  - The KKT conditions do not rule out the case that \hat{w}^top x_n =
    1, but alpha_n = 0 (i.e. a support vector that touches the margin,
    but does not exert force against it). Such n are then included in
    cal{S}, but lead to problems in (2.7), because they would require
    tilde{w}^top x_n = infinity, which is not possible.
In the proof of Lemma 6, case 2. at the bottom of p.14:
  - After the first inequality, C_0^2 t^{-1.5 epsilon_+} should be
    C_0^2 t^{-epsilon_+}
  - After the second inequality the part between brackets is missing an
    additional term C_0^2 t^{-\epsilon_+}.
  - In addition, the label (1) should be on the previous inequality and
    it should be mentioned that e^{-x} <= 1-x+x^2 is applied for x >= 0
    (otherwise it might be false).
In the proof of Lemma 6, case 2 in the middle of p.15:
  - In the line of inequality (1) there is a t^{-epsilon_-} missing. In
    the next line there is a factor t^{-epsilon_-} too much.
  - In addition, the inequality e^x >= 1 + x holds for all x, so no need
    to mention that x > 0.
In Lemma 1:
  - claim (3) should be lim_{t \to \infty} w(t)^\top x_n = infinity
  - In the proof: w(t)^top x_n > 0 only holds for large enough t.
Remarks:
p.4 The claim that "we can expect the population (or test)
misclassification error of w(t) to improve" because "the margin of w(t)
keeps improving" is worded a little too strongly, because it presumes
that the maximum margin solution will always have the best
generalization error.
In the proof sketch (p.3):
  - Why does the fact that the limit is dominated by gradients that are
    a linear combination of support vectors imply that w_infinity will
    also be a non-negative linear combination of support vectors?
  - "converges to some limit". Mention that you call this limit
    w_infinity
Minor Issues:
In (2.4): add "for all n".
p.10, footnote: Shouldn't "P_1 = X_s X_s^+" be something like "P_1 =
(X_s^top X_s)^+"?
A.9: ell should be ell'
The paper needs a round of copy editing. For instance:
  - top of p.4: "where tilde{w} A is the unique"
  - p.10: "the solution tilde{w} to TO eq. A.2"
  - p.10: "might BOT be unique"
  - p.10: "penrose-moorse pseudo inverse" -> "Moore-Penrose
    pseudoinverse"

In the bibliography, Kingma and Ba is cited twice, with different years.


In representation learning (RL), how to make the learned representations easy to interpret and less overfitted to training data are two important but challenging issues. To address these problems, we study a new type of regularization approach that encourages the supports of weight vectors in RL models to have small overlap, by simultaneously promoting near-orthogonality among vectors and sparsity of each vector. We apply the proposed regularizer to two models: neural networks (NNs) and sparse coding (SC), and develop an efficient ADMM-based algorithm for regularized SC. Experiments on various datasets demonstrate that weight vectors learned under our regularizer are more interpretable and have better generalization performance.

The paper proposed a new regularization approach that simultaneously encourages the weight vectors (W) to be sparse and orthogonal to each other. The argument is that the sparsity helps to eliminate the irrelevant feature vectors by making the corresponding weights zero. Nearly orthogonal sparse vectors will have zeros at different indexes and hence, encourages the weight vectors to have small overlap in terms of indices of nonzero entries (called support). Small overlap in support of weight vectors, aids interpretability as each weight vector is associated with a unique subset of feature vectors. For example, in the topic model, small overlap encourages, each topic to have unique set of representation words.
The proposed approach used L1 regularizer for enforcing sparsity in W. For enforcing orthogonality between different weight vectors (wi, wj), the log-determinant divergence (LDD) regularization term encourages the Gram Matrix G (Gij = wiTwj) to be close to an identity matrix I. The authors applied and tested the performance of proposed approach on Neural Network and Sparse Coding (SC) machine learning models. The authors validated the need for their proposed regularizer through experiments on 4 datasets (3 text and 1 images).
Major
* The novelty of the paper is not clear. Neither L1 no logdet() are novel regularizers (see the literature of Determinatal Point Process). With the presence of the auto-differentiator, one cannot claim the making derivative a novelty.
* L1 is also encourages diversity although as explicit as logdet. This is also obvious from Fig 2.  Perhaps the advantage of diversity is in interpretability but that is hard to quantify and the authors did not put enough effort to do that; we only have small anecdotal results in section 4.3.
* The Table 1 is not convincing because one can argue, for example, gun (vec 1) and weapon (vec 4) are colinear.
* In section 4.2, the authors experimented with SC on text dataset.  The overlap score decreases as the strength of regularization increases. The authors didn’t show the effect of increasing the regularization strength on the model accuracy and convergence time. This analysis is important to make sure, the decrease in overlap score is not coming at the expense of model accuracy and performance.
* In section 4.4, increase in test set accuracy and difference between test and train set accuracy is used to validate the claim, that the proposed regularizer helps reducing over fitting. In Table-2, , the test accuracy increases between SC and LDD-L1 SC while the train accuracy remains almost the same. Also, the authors didn’t do any cross validation to support their claim. The difference is numbers is too small to support the claim.
* In section on LSTM for Language Modeling, the perplexity score of LDD-L1 regularization on PytorchLM received perplexity score of 1.2 lower than without regularization. Although, the author mentions it as a significant reduction, the lowest perplexity score in Table 3 is significantly lower than this result. It’s not clear how 1.2 reduction in perplexity is significant and why the method should be preferred while much better models already exists.
* Results of the best perplexity model, Neural Architecture Search + WT V2, with proposed regularization would also help, validating the generalizability claims of the new approach.
* In CNN for Image Classification section, details of increase interpretability of the model, in terms of classification decision, is missing.
* In Table-4, the proposed LDD-L1 WideResNet is not the best results. Results of adding the proposed regularization, to the best know method (Pyramid Sep Drop) would be interesting.
* The proposed regularization claims to provide more interpretable representation and less overfit model. The given experiments are inadequate to validate the claims.
* A more extensive experimentation is required to validate the applicability of the method.
* In SC, aj are the linear coefficients or the coefficient vector of the j-th sample. If A ∈ Rm×n then aj ∈ Rm×1 and j ranges between [1,n] as in equation 6. The notation in section 2.2, Sparse Coding section is misleading as j ranges between [1,m].
* In Related works, the authors mention previous work done on interpreting the results of the machine learning models. Related works on enhancing interpretability and reducing overfitting by using regularization is missing.


Published as a conference paper at ICLR 2018 NEURAL SKETCH LEARNING FOR CONDITIONAL PROGRAM GENERATION
We study the problem of generating source code in a strongly typed, Java-like programming language, given a label (for example a set of API calls or types) carrying a small amount of information about the code that is desired. The generated programs are expected to respect a "realistic" relationship between programs and labels, as exemplified by a corpus of labeled programs available during training. Two challenges in such conditional program generation are that the generated programs must satisfy a rich set of syntactic and semantic constraints, and that source code contains many low-level features that impede learning. We address these problems by training a neural generator not on code but on program sketches, or models of program syntax that abstract out names and operations that do not generalize across programs. During generation, we infer a posterior distribution over sketches, then concretize samples from this distribution into type-safe programs using combinatorial techniques. We implement our ideas in a system for generating API-heavy Java code, and show that it can often predict the entire body of a method given just a few API calls or data types that appear in the method.

The authors introduce an algorithm in the subfield of conditional program generation that is able to create programs in a rich java like programming language. In this setting, they propose an algorithm based on sketches- abstractions of programs that capture the structure but discard program specific information that is not generalizable such as variable names. Conditioned on information such as type specification or keywords of a method they generate the method's body from the trained sketches.

Positives:

	•	Novel algorithm and addition of rich java like language in subfield of 'conditional program generation' proposed
	•	Very good abstract: It explains high level overview of topic and sets it into context plus gives a sketch of the algorithm and presents the positive results.
	•	Excellently structured and presented paper

	•	Motivation given in form of relevant applications and mention that it is relatively unstudied
	•	The hypothesis/ the papers goal is clearly stated. It is introduced with 'We ask' followed by two well formulated lines that make up the hypothesis. It is repeated multiple times throughout the paper. Every mention introduces either a new argument on why this is necessary or sets it in contrast to other learners, clearly stating discrepancies.
	•	Explanations are exceptionally well done: terms that might not be familiar to the reader are explained. This is true for mathematical aspects as well as program generating specific terms. Examples are given where appropriate in a clear and coherent manner
	•	Problem statement well defined mathematically and understandable for a broad audience
	•	Mentioning of failures and limitations demonstrates a realistic  view on the project
	•	Complexity and time analysis provided
	•	Paper written so that it's easy for a reader to implement the methods
	•	Detailed descriptions of all instantiations even parameters and comparison methods
	•	System specified
	•	Validation method specified
	•	Data and repository, as well as cleaning process provided
	•	Every figure and plot is well explained and interpreted
	•	Large successful evaluation section provided
	•	Many different evaluation measures defined to measure different properties of the project
	•	Different observability modes
	•	Evaluation against most compatible methods from other sources
	•	 Results are in line with hypothesis
	•	Thorough appendix clearing any open questions

It would have been good to have a summary/conclusion/future work section

SUMMARY: ACCEPT.  The authors present a very intriguing novel approach that  in a clear and coherent way. The approach is thoroughly explained for a large audience. The task itself is interesting and novel. The large evaluation section that discusses many different properties is a further indication that this approach is not only novel but also very promising. Even though no conclusive section is provided, the paper is not missing any information.


Published as a conference paper at ICLR 2018 DEEP COMPLEX NETWORKS
At present, the vast majority of building blocks, techniques, and architectures for deep learning are based on real-valued operations and representations. However, recent work on recurrent neural networks and older fundamental theoretical analysis suggests that complex numbers could have a richer representational capacity and could also facilitate noise-robust memory retrieval mechanisms. Despite their attractive properties and potential for opening up entirely new neural architectures, complex-valued deep neural networks have been marginalized due to the absence of the building blocks required to design such models. In this work, we provide the key atomic components for complex-valued deep neural networks and apply them to convolutional feed-forward networks and convolutional LSTMs. More precisely, we rely on complex convolutions and present algorithms for complex batch-normalization, complex weight initialization strategies for complex-valued neural nets and we use them in experiments with end-to-end training schemes. We demonstrate that such complex-valued models are competitive with their realvalued counterparts. We test deep complex models on several computer vision tasks, on music transcription using the MusicNet dataset and on Speech Spectrum Prediction using the TIMIT dataset. We achieve state-of-the-art performance on these audio-related tasks.

This paper defines building blocks for complex-valued convolutional neural networks: complex convolutions, complex batch normalisation, several variants of the ReLU nonlinearity for complex inputs, and an initialisation strategy. The writing is clear, concise and easy to follow.
An important argument in favour of using complex-valued networks is said to be the propagation of phase information. However, I feel that the observation that CReLU works best out of the 3 proposed alternatives contradicts this somewhat. CReLU simply applies ReLU component-wise to the real and imaginary parts, which has an effect on the phase information that is hard to conceptualise. It definitely does not preserve phase, like modReLU would.
This makes me wonder whether the "complex numbers" paradigm is applied meaningfully here, or whether this is just an arbitrary way of doing some parameter sharing in convnets that happens to work reasonably well (note that even completely random parameter tying can work well, as shown in "Compressing neural networks with the hashing trick" by Chen et al.). Some more insight into how phase information is used, what it represents and how it is propagated through the network would help to make sense of this.
The image recognition results are mostly inconclusive, which makes it hard to assess the benefit of this approach. The improved performance on the audio tasks seems significant, but how the complex nature of the networks helps achieve this is not really demonstrated. It is unclear how the phase information in the input waveform is transformed into the phase of the complex activations in the network (because I think it is implied that this is what happens). This connection is a bit vague. Once again, a more in-depth analysis of this phase behavior would be very welcome.
I'm on the fence about this work: I like the ideas and they are explained well, but I'm missing some insight into why and how all of this is actually helping to improve performance (especially w.r.t. how phase information is used).
Comments:
- The related work section is comprehensive but a bit unstructured, with each new paragraph seemingly describing a completely different type of work. Maybe some subsection titles would help make it feel a bit more cohesive.
- page 3: "(cite a couple of them)" should be replaced by some actual references :)
- Although care is taken to ensure that the complex and real-valued networks that are compared in the experiments have roughly the same number of parameters, doesn't the complex version always require more computation on account of there being more filters in each layer? It would be nice to discuss computational cost as well.
REVISION: I have decided to raise my rating from 5 to 7 as I feel that the authors have adequately addressed many of my comments. In particular, I really appreciated the additional appendix sections to clarify what actually happens as the phase information is propagated through the network.
Regarding the CIFAR results, I may have read over it, but I think it would be good to state even more clearly that these experiments constitute a sanity check, as both reviewer 1 and myself were seemingly unaware of this. With this in mind, it is of course completely fine that the results are not better than for real-valued networks.


The goal of imitation learning (IL) is to enable a learner to imitate an expert's behavior given the expert's demonstrations. Recently, generative adversarial imitation learning (GAIL) has successfully achieved it even on complex continuous control tasks. However, GAIL requires a huge number of interactions with environment during training. We believe that IL algorithm could be more applicable to the real-world environments if the number of interactions could be reduced. To this end, we propose a model free, off-policy IL algorithm for continuous control. The keys of our algorithm are two folds: 1) adopting deterministic policy that allows us to derive a novel type of policy gradient which we call deterministic policy imitation gradient (DPIG), 2) introducing a function which we call state screening function (SSF) to avoid noisy policy updates with states that are not typical of those appeared on the expert's demonstrations. Experimental results show that our algorithm can achieve the goal of IL with at least tens of times less interactions than GAIL on a variety of continuous control tasks.

This paper considers the problem of model-free imitation learning. The problem is formulated in the framework of generative adversarial imitation learning (GAIL), wherein we alternate between optimizing reward parameters and learner policy's parameters. The reward parameters are optimized so that the margin between the cost of the learner's policy and the expert's policy is maximized. The learner's policy is optimized (using any model-free RL method) so that the same cost margin is minimized. Previous formulation of GAIL uses a stochastic behavior policy and the RIENFORCE-like algorithms. The authors of this paper propose to use a deterministic policy instead, and apply the deterministic policy gradient DPG (Silver et al., 2014) for optimizing the behavior policy.
The authors also briefly discuss the problem of the little overlap between the teacher's covered state space and the learner's. A state screening function (SSF) method is proposed to drive the learner to remain in areas of the state space that have been covered by the teacher. Although, a more detailed discussion and a clearer explanation is needed to clarify what SSF is actually doing, based on the provided formulation.
Except from a few typos here and there, the paper is overall well-written. The proposed idea seems new. However, the reviewer finds the main contribution rather incremental in its nature. Replacing a stochastic policy with a deterministic one does not change much the original GAIL algorithm, since the adoption of stochastic policies is often used just to have differentiable parameterized policies, and if the action space is continuous, then there is not much need for it (except for exploration, which is done here through re-initializations anyway). My guess is that if someone would use the GAIL algorithm for real problems (e.g, robotic task), they would significantly reduce the stochasticity of the behavior policy, which would make it virtually similar in term of data efficiency to the proposed method.
Pros:
- A new GAIL formulation for saving on interaction data.
Cons:
- Incremental improvement over GAIL
- Experiments only on simulated toy problems
- No theoretical guarantees for the state screening function (SSF) method


LEARNING ONE-HIDDEN-LAYER NEURAL NETWORKS WITH LANDSCAPE DESIGN
We consider the problem of learning a one-hidden-layer neural network: we assume the input x ∈ R d is from Gaussian distribution and the label y = a σ(Bx) + ξ, where a is a nonnegative vector in R m with m ≤ d, B ∈ R m×d is a full-rank weight matrix, and ξ is a noise vector. We first give an analytic formula for the population risk of the standard squared loss and demonstrate that it implicitly attempts to decompose a sequence of low-rank tensors simultaneously.
Inspired by the formula, we design a non-convex objective function G(·) whose landscape is guaranteed to have the following properties:
1. All local minima of G are also global minima. 2. All global minima of G correspond to the ground truth parameters. 3. The value and gradient of G can be estimated using samples.
With these properties, stochastic gradient descent on G provably converges to the global minimum and learn the ground-truth parameters. We also prove finite sample complexity results and validate the results by simulations.

[ =========================== REVISION ===============================================================]
I am satisfied with the answers to my questions. The paper still needs some work on clarity, and authors defer the changes to the next version (but as I understood, they did no changes for this paper as of now), which is a bit frustrating. However I am fine accepting it.
[ ============================== END OF REVISION =====================================================]
This paper concerns with addressing the issue of SGD not converging to the optimal parameters on one hidden layer network for a particular type of data and label (gaussian features, label generated using a particular function that should be learnable with neural net). Authors demonstrate empirically that this particular learning problem is hard for SGD with l2 loss (due to apparently bad local optima) and suggest two ways of addressing it, on top of the known way of dealing with this problem (which is overparameterization). First is to use a new activation function, the second is by designing a new objective function that has only global optima and which can be efficiently learnt with SGD
Overall the paper is well written. The authors first introduce their suggested loss function and then go into details about what inspired its creation. I do find interesting the formulation of population risk in terms of tensor decomposition, this is insightful
My issues with the paper are as follows:
- The loss function designed seems overly complicated. On top of that authors notice that to learn with this loss efficiently, much larger batches had to be used. I wonder how applicable this in practice - I frankly didn't see insights here that I can apply to other problems that don't fit into this particular narrowly defined framework
- I do find it somewhat strange that no insight to the actual problem is provided (e.g. it is known empirically but there is no explanation of what actually happens and there is a idea that it is due to local optima), but authors are concerned with developing new loss function that has provable properties about global optima. Since it is all empirical, the first fix (activation function) seems sufficient to me and new loss is very far-fetched.
- It seems that changing activation function from relu to their proposed one fixes the problem without their new loss, so i wonder whether it is a problem with relu itself and may be other activations funcs, like sigmoids will not suffer from the same problem
- No comparison with overparameterization in experiments results is given, which makes me wonder why their method is better.
Minor: fix margins in formula 2.7.


Many tasks in natural language processing involve comparing two sentences to compute some notion of relevance, entailment, or similarity. Typically this comparison is done either at the word level or at the sentence level, with no attempt to leverage the inherent structure of the sentence. When sentence structure is used for comparison, it is obtained during a non-differentiable pre-processing step, leading to propagation of errors. We introduce a model of structured alignments between sentences, showing how to compare two sentences by matching their latent structures. Using a structured attention mechanism, our model matches possible spans in the first sentence to possible spans in the second sentence, simultaneously discovering the tree structure of each sentence and performing a comparison, in a model that is fully differentiable and is trained only on the comparison objective. We evaluate this model on two sentence comparison tasks: the Stanford natural language inference dataset and the TREC-QA dataset. We find that comparing spans results in superior performance to comparing words individually, and that the learned trees are consistent with actual linguistic structures.

This paper describes the use of latent context-free derivations, using
a CRF-style neural model, as a latent level of representation in neural
attention models that consider pairs of sentences. The model implicitly
learns a distribution over derivations, and uses marginals under this
distribution to bias attention distributions over spans in one sentence
given a span in another sentence.
This is an intriguing idea. I had a couple of reservations however:
* The empirical improvements from the method seem pretty marginal, to the
point that it's difficult to know what is really helping the model. I would
liked to have seen more explanation of what the model has learned, and
more comparisons to other baselines that make use of attention over spans.
For example, what happens if every span is considered as an independent random
variable, with no use of a tree structure or the CKY chart?
* The use of the \alpha^0 vs. \alpha^1 variables is not entirely clear. Once they
have been calculated in Algorithm 1, how are they used? Do the \rho values
somewhere treat these two quantities differently?
* I'm skeptical of the type of qualitative analysis in section 4.3, unfortunately.
I think something much more extensive would be interesting here. As one
example, the PP attachment example with "at a large venue" is highly suspect;
there's a 50/50 chance that any attachment like this will be correct, there's
absolutely no way of knowing if the model is doing something interesting/correct
or performing at a chance level, given a single example.


The growing interest to implement Deep Neural Networks (DNNs) on resourcebound hardware has motivated the innovation of compression algorithms. Using these algorithms, DNN model sizes can be substantially reduced, with little-to-no accuracy degradation. This is achieved by either eliminating components from the model, or penalizing complexity during training. While both approaches demonstrate considerable compressions, the former often ignores the loss function during compression while the latter produces unpredictable compressions. In this paper, we propose a technique that directly minimizes both the model complexity and the changes in the loss function. In this technique, we formulate compression as a constrained optimization problem, and then present a solution for it. We will show that using this technique, we can achieve competitive results.

1. Summary
This paper introduced a method to learn a compressed version of a neural network such that the loss of the compressed network doesn't dramatically change.
2. High level paper
- I believe the writing is a bit sloppy. For instance equation 3 takes the minimum over all m in C but C is defined to be a set of c_1, ..., c_k, and other examples (see section 4 below). This is unfortunate because I believe this method, which takes as input a large complex network and compresses it so the loss in accuracy is small, would be really appealing to companies who are resource constrained but want to use neural network models.
3. High level technical
- I'm confused at the first and second lines of equation (19). In the first line, shouldn't the first term not contain \Delta W ? In the second line, shouldn't the first term be \tilde{\mathcal{L}}(W_0 + \Delta W) ?
- For CIFAR-10 and SVHN you're using Binarized Neural Networks and the two nice things about this method are (a) that the memory usage of the network is very small, and (b) network operations can be specialized to be fast on binary data. My worry is if you're compressing these networks with your method are the weights not treated as binary anymore? Now I know in Binarized Neural Networks they keep a copy of real-valued weights so if you're just compressing these then maybe all is alright. But if you're compressing the weights _after_ binarization then this would be very inefficient because the weights won't likely be binary anymore and (a) and (b) above no longer apply.
- Your compression ratio is much higher for MNIST but your accuracy loss is somewhat dramatic, especially for MNIST (an increase of 0.53 in error nearly doubles your error and makes the network worse than many other competing methods: http://rodrigob.github.io/are_we_there_yet/build/classification_datasets_results.html#4d4e495354). What is your compression ratio for 0 accuracy loss? I think this is a key experiment that should be run as this result would be much easier to compare with the other methods.
- Previous compression work uses a lot of tricks to compress convolutional weights. Does your method work for convolutional layers?
- The first paper to propose weight sharing was not Han et al., 2015, it was actually:
Chen W., Wilson, J. T., Tyree, S., Weinberger K. Q., Chen, Y. "Compressing Neural Networks with the Hashing Trick" ICML 2015
Although they did not learn the weight sharing function, but use random hash functions.
4. Low level technical
- The end of Section 2 has an extra 'p' character
- Section 3.1: "Here, X and y define a set of samples and ideal output distributions we use for training" this sentence is a bit confusing. Here y isn't a distribution, but also samples drawn from some distribution. Actually I don't think it makes sense to talk about distributions at all in Section 3.
- Section 3.1: "W is the learnt model...\hat{W} is the final, trained model" This is unclear: W and \hat{W} seem to describe the same thing. I would just remove "is the learnt model and"
5. Review summary
While the trust-region-like optimization of the method is nice and I believe this method could be useful for practitioners, I found the paper somewhat confusing to read. This combined with some key experimental questions I have make me think this paper still needs work before being accepted to ICLR.


Published as a conference paper at ICLR 2018 CGANS WITH PROJECTION DISCRIMINATOR
We propose a novel, projection based way to incorporate the conditional information into the discriminator of GANs that respects the role of the conditional information in the underlining probabilistic model. This approach is in contrast with most frameworks of conditional GANs used in application today, which use the conditional information by concatenating the (embedded) conditional vector to the feature vectors. With this modification, we were able to significantly improve the quality of the class conditional image generation on ILSVRC2012 (ImageNet) 1000-class image dataset from the current state-of-the-art result, and we achieved this with a single pair of a discriminator and a generator. We were also able to extend the application to super-resolution and succeeded in producing highly discriminative super-resolution images. This new structure also enabled high quality category transformation based on parametric functional transformation of conditional batch normalization layers in the generator. The code with Chainer (Tokui et al., 2015), generated images and pretrained models are available at https://github.com/pfnet-research/sngan_projection.

I thank the authors for the thoughtful response and updated manuscript. After reading through both, my review score remains unchanged.
=================
The authors describe a new variant of a generative adversarial network (GAN) for generating images. This model employs a 'projection discriminator' in order to incorporate image labels and demonstrate that the resulting model outperforms state-of-the-art GAN models.
Major comments:
1) Spatial resolution. What spatial resolution is the model generating images at? The AC-GAN work performed an analysis to assess how information is being introduced at each spatial resolution by assessing the gains in the Inception score versus naively resizing the image. It is not clear how much the gains of this model is due to generating better lower resolution images and performing simple upscaling. It would be great to see the authors address this issue in a serious manner.
2) FID in real data. The numbers in Table 1 appear favorable to the projection model. Please add error bars (based on Figure 4, I would imagine they are quite large). Additionally, would it be possible to compute this statistic for *real* images? I would be curious to know what the FID looks like as a 'gold standard'.
3) Conditional batch normalization.  I am not clear how much of the gains arose from employing conditional batch normalization versus the proposed method for incorporating the projection based discriminator. The former has been seen to be quite powerful in accomodating multi-modal tasks (e.g. https://arxiv.org/abs/1709.07871, https://arxiv.org/abs/1610.07629
). If the authors could provide some evidence highlighting the marginal gains of one technique, that would be extremely helpful.
Minor comments:
- I believe you have the incorrect reference for conditional batch normalization on Page 5.
A Learned Representation For Artistic Style
Dumoulin, Shlens and Kudlur (2017)
https://arxiv.org/abs/1610.07629
- Please enlarge images in Figure 5-8. Hard to see the detail of 128x128 images.
- Please add citations for Figures 1a-1b. Do these correspond with some known models?
Depending on how the authors respond to the reviews, I would consider upgrading the score of my review.


There have been numerous recent advancements on learning deep generative models with latent variables thanks to the reparameterization trick that allows to train deep directed models effectively. However, since reparameterization trick only works on continuous variables, deep generative models with discrete latent variables still remain hard to train and perform considerably worse than their continuous counterparts. In this paper, we attempt to shrink this gap by introducing a new architecture and its learning procedure. We develop a hybrid generative model with binary latent variables that consists of an undirected graphical model and a deep neural network. We propose an efficient two-stage pretraining and training procedure that is crucial for learning these models. Experiments on binarized digits and images of natural scenes demonstrate that our model achieves close to the state-of-the-art performance in terms of density estimation and is capable of generating coherent images of natural scenes.

Summary of the paper:
The paper proposes to augment a variational auto encoder (VAE) with an binary restricted Boltzmann machine (RBM) in the role of the prior of the generative model. To yield a good initialisation of the parameters of the RBM and the inference network a special pertaining procedure is introduced. The model produces competitive Likelihood results on MNIST and was further tested on CIFAR 10.
Clarity and quality:
1. From the description of the pertaining procedure and the appendix B I got the impression that the inference network maps into [0,1] and not into {0,1}.  Does it mean, you are not really considering binary latent variables (making the RBM model the values in [0,1] by its probability p(z|h))?
2. on page 2:
RWS...."derive a tighter lower bound": Where does the "tighter" refer to?
3. "multivariate Bernoulli modeled by an RBM": Note,  while in a multivariate Bernoulli the binary variables would be independent from each others, this is usually not the case for the visible variables of RBMs (only in the conditional distribution given the state of the hidden variables).
4. The notation could be improved, e.g.:
-x_data and x_sample are not explained
- M is not defined in equation 5.
5. "this training method has been previously used to produce the best results on MNIST" Note, that parallel tempering often leads to better results when training RBMs (see http://proceedings.mlr.press/v9/desjardins10a/desjardins10a.pdf) . Furthermore, centred RBMs are also get better results than vanilla RBMs (see: http://jmlr.org/papers/v17/14-237.html).
Originality and significance:
As already mentioned in a comment on open-review the current version of the paper misses to mention one very related work: "discrete variational auto encoders". Also "bidirectional Helmholtz machines" could be mentioned as generative model with discrete latent variables.  The results for both should also be reported in Table 1 (discrete VAEs: 81,01, BiHMs: 84,3).
From the motivation the advantages of the model did not become very clear to me. Main advantage seems to be the good likelihood result on MNIST (but likelihood does not improve compared to IWAE on CIFAR 10 for example). However, using an RBM as prior has the disadvantage that sampling from the generative model requires running a Markov chain now while having a solely directed generative model allows for fast sampling.
Experiments show good likelihood results on MNIST. Best results are obtained when using a ResNet decoder. I wondered how much a standard VAE is improved by using such a powerful decoder. Reporting this, would allow to understand, how much is gained from using a RBM for learning the prior.
Minor comments:
page 1:
"debut of variational auto encoder (VAE) and reparametrization trick" -> debut of variational auto encoders (VAE) and the reparametrization trick",
page 2:
"with respect to the parameter of  p(x,z)" -> "with respect to the parameters of  p(x,z)"
"parameters in p" -> "parameters of p"
"is multivariate Bernoulli" ->  "is a multivariate Bernoulli"
"we compute them" -> "we compute it"
page 3:
"help find a good" ->  "help to find a good"
page 7:
"possible apply" -> "possible to apply"


Common recurrent neural network architectures scale poorly due to the intrinsic difficulty in parallelizing their state computations. In this work, we propose the Simple Recurrent Unit (SRU) architecture, a recurrent unit that simplifies the computation and exposes more parallelism. In SRU, the majority of computation for each step is independent of the recurrence and can be easily parallelized. SRU is as fast as a convolutional layer and 5-10x faster than an optimized LSTM implementation. We study SRUs on a wide range of applications, including classification, question answering, language modeling, translation and speech recognition. Our experiments demonstrate the effectiveness of SRU and the tradeoff it enables between speed and performance. We open source our implementation in PyTorch and CNTK.

The authors introduce SRU, the Simple Recurrent Unit that can be used as a substitute for LSTM or GRU cells in RNNs. SRU is much more parallel than the standard LSTM or GRU, so it trains much faster: almost as fast as a convolutional layer with properly optimized CUDA code. Authors perform experiments on numerous tasks showing that SRU performs on par with LSTMs, but the baselines for these tasks are a little problematic (see below).
On the positive side, the paper is very clear and well-written, the SRU is a superbly elegant architecture with a fair bit of originality in its structure, and the results show that it could be a significant contribution to the field as it can probably replace LSTMs in most cases but yield fast training. On the negative side, the authors present the results without fully referencing and acknowledging state-of-the-art. Some of this has been pointed out in the comments below already. As another example: Table 5 that presents results for English-German WMT translation only compares to OpenNMT setups with maximum BLEU about 21. But already a long time ago Wu et. al. presented LSTMs reaching 25 BLEU and current SOTA is above 28 with training time much faster than those early models (https://arxiv.org/abs/1706.03762). While the latest are non-RNN architectures, a table like Table 5 should include them too, for a fair presentation. In conclusion: the authors seem to avoid discussing the problem that current non-RNN architectures  could be both faster and yield better results on some of the studied problems. That's bad presentation of related work and should be improved in the next versions (at which point this reviewer is willing to revise the score). But in all cases, this is a significant contribution to deep learning and deserves acceptance.
Update: the revised version of the paper addresses all my concerns and the comments show new evidence of potential applications, so I'm increasing my score.


We propose a method to learn stochastic activation functions for use in probabilistic neural networks. First, we develop a framework to embed stochastic activation functions based on Gaussian processes in probabilistic neural networks. Second, we analytically derive expressions for the propagation of means and covariances in such a network, thus allowing for an efficient implementation and training without the need for sampling. Third, we show how to apply variational Bayesian inference to regularize and efficiently train this model. The resulting model can deal with uncertain inputs and implicitly provides an estimate of the confidence of its predictions. Like a conventional neural network it can scale to datasets of arbitrary size and be extended with convolutional and recurrent connections, if desired.

In Bayesian neural networks, a deterministic or parametric activation is typically used. In this work, activation functions are considered random functions with a GP prior and are inferred from data.
- Unnecessary complexity
The presentation of the paper is unnecessarily complex. It seems that authors spend extra space creating problems and then solving them. Although some of the derivations in Section 3.2.2 are a bit involved, most of the derivations up to that point (which is already in page 6) follow preexisting literature.
For instance, eq. (3) proposes one model for p(F|X). Eq. (8) proposes a different model for p(F|X), which is an approximation to the previous one. Instead, the second model could have been proposed directly, with the appropriate citation from the literature, since it isn't new. Eq. (13) is introduced as a "solution" to a non-existent problem, because the virtual observations are drawn from the same prior as the real ones, so it is not that we are "coming up" with a convenient GP prior that turns out to produce a computationally tractable solution, we are just using the prior on the observations consistently.
In general, the authors seem to use "approximately equal" and "equal" interchangeably, which is incorrect. There should be a single definition for p(F|X). And there should be a single definition for L_pred. The expression for L_pred given in eq. (20) (exact) and eq. (41) (approximate) do not match and yet both are connected with an equality (or proportionality), which they shouldn't.
Q(A) is sometimes taken to mean the true posterior (i.e., eq. (31)), sometimes a Gaussian approximation (i.e., eq (32) inside the integral), and both are used interchangeably.
- Incorrect references to the literature
Page 3: "using virtual observations (originally proposed by Quiñonero-Candela & Rasmussen (2005) for sparse approximations of GPs)"
The authors are citing as the origin of virtual observations a survey paper on the topic. Of course, that survey paper correctly attributes the origin to [1].
Page 4: "we apply the technique of variational inference Wainwright et al. (2008)".
How can variational inference be attributed to (again) a survey paper on the topic from 2008, when for instance [2] appeared in 2003?
- Correctness of the approach
Can the authors guarantee that the variational bound that they are introducing (as defined in eqs. (19) and (41)) is actually a variational bound? It seems to me that the approximations made to Q(A) to propagate the uncertainty are breaking the bounding guarantee. If it is no longer a lower bound, what is the rationale behind maximizing it?
The mathematical basis for this paper is actually introduced in [3] and a single-layer version of the current model is developed in [4]. However, in [4] the authors manage to avoid the additional Q(A) approximation that breaks the variational bound. The authors should contrast their approach with [4] and discuss if and why that additional central limit theorem application is necessary.
- No experiments
The use of a non-parametric definition for the activation function should be contrasted with the use of a parametric one. With enough data, both might produce similar results. And the parameter sharing in the parametric one might actually be beneficial. With no experiments at all showing the benefit of this proposal, this paper cannot be considered complete.
- Minor errors:
Eq. (4), for consistency, should use the identity matrix for the covariance matrix definition.
Eq. (10) uses subscript d where it should be using subscript n
Eq. (17) includes p(X^L|F^L) in the definition of Q(...), but it shouldn't. That was particularly misleading, since if we take eq. (17) to be correct (which I did at first), then p(X^L|F^L) cancels out and should not appear in eq. (20).
Eq. (23) uses Q(F|A) to mean the same as P(F|A) as far as I understand. Then why use Q?
- References
[1] Edward Snelson and Zoubin Ghahramani. Sparse Gaussian processes using pseudo-inputs.
[2] Beal, M.J. Variational Algorithms for Approximate Bayesian Inference.
[3] M.K. Titsias and N.D. Lawrence. Bayesian Gaussian process latent variable model.
[4] M. Lázaro-Gredilla. Bayesian warped Gaussian processes.


We propose a novel architecture for k-shot classification on the Omniglot dataset. Building on prototypical networks, we extend their architecture to what we call Gaussian prototypical networks. Prototypical networks learn a map between images and embedding vectors, and use their clustering for classification. In our model, a part of the encoder output is interpreted as a confidence region estimate about the embedding point, and expressed as a Gaussian covariance matrix. Our network then constructs a direction and class dependent distance metric on the embedding space, using uncertainties of individual data points as weights. We show that Gaussian prototypical networks are a preferred architecture over vanilla prototypical networks with an equivalent number of parameters. We report results consistent with state-of-the-art performance in 1-shot and 5-shot classification both in 5-way and 20-way regime on the Omniglot dataset. We explore artificially down-sampling a fraction of images in the training set, which improves our performance. Our experiments therefore lead us to hypothesize that Gaussian prototypical networks might perform better in less homogeneous, noisier datasets, which are commonplace in real world applications.

The paper extends the prototypical networks of Snell et al, NIPS 2017 for one shot learning. Snell et al use a soft kNN classification rule, typically used in standard metric learning work (e.g. NCA, MCML), over learned instance projections, i.e. distances are computed over the learned projections. Each class is represented by a class prototype which is given by the average of the projections of the class instances. Classification is done with soft k-NN on the class prototypes. The distance that is used is the Euclidean distance over the learned representations, i.e. (z-c)^T(z-c), where z is the projection of the x instance to be classified and c is a class prototype, computed as the average of the projections of the support instances of a given class.
The present paper extends the above work to include the learning of a Mahalanobis matrix, S, for each instance, in addition to learning its projection. Thus now the classification is based on the Mahalanobis distance: (z-c)^T S_c (z-c). On a conceptual level since S_c should be a PSD matrix it can be written as the square of some matrix, i.e. S_c = A_c^TA_c, then the Mahanalobis distance becomes (A_c z - A_c c)^T ( A_c z-A_c c), i.e. in addition to learning a projection as it is done in Snell et al, the authors now learn also a linear transformation matrix which is a function of the support points (i.e. the ones which give rise to the class prototypes). The interesting part here is that the linear projection is a function of the support points. I wonder though if such a transformation could not be learned by the vanilla prototypical networks simply by learning now a projection matrix A_z as a function of the query point z. I am not sure I see any reason why the vanilla prototypical networks cannot learn to project x directly to A_z z and why one would need to do this indirectly through the use of the Mahalanobis distance as proposed in this paper.
On a more technical level the properties of the learned Mahalanobis matrix, i.e. the fact that it should be PSD, are not really discussed neither how this can be enforced especially in the case where S is a full matrix (even though the authors state that this method was not further explored). If S is diagonal then the S generation methods a) b) c) in the end of section 3.1 will make sure that S is PSD, I do not think that this is the case with d) though.
In the definition of the prototypes the component wise weigthing (eq. 5) works when the Mahalanobis matrix is diagonal (even though the weighting should be done by the \sqrt of it), how would it work if it was a full matrix is not clear.
On the experiments side the authors could have also experimented with miniImageNet and not only omniglot as is the standard practice in one shot learning papers.
I am not sure I understand figure 3 in which the authors try to see what happens if instead of learning the Mahalanobis matrix one would learn a projection that would have as many additional dimensions as free elements in the Mahalanobis matrix. I would expect to see a comparison of the vanilla prototypical nets against their method for each one of the different scenarios of the free parameters of the S matrix, something like a ratio of accuracies of the two methods in order to establish whether learning the Mahalanobis matrix brings an improvement over the prototypical nets with an equal number of output parameters.


Learning to Optimize (Li & Malik, 2016) is a recently proposed framework for learning optimization algorithms using reinforcement learning. In this paper, we explore learning an optimization algorithm for training shallow neural nets. Such high-dimensional stochastic optimization problems present interesting challenges for existing reinforcement learning algorithms. We develop an extension that is suited to learning optimization algorithms in this setting and demonstrate that the learned optimization algorithm consistently outperforms other known optimization algorithms even on unseen tasks and is robust to changes in stochasticity of gradients and the neural net architecture. More specifically, we show that an optimization algorithm trained with the proposed method on the problem of training a neural net on MNIST generalizes to the problems of training neural nets on the Toronto Faces Dataset, CIFAR-10 and CIFAR-100.

Summary of the paper
---------------------------
The paper derives a scheme for learning optimization algorithm for high-dimensional stochastic problems as the one involved in shallow neural nets training. The main motivation is to learn to optimize with the goal to design a meta-learner able to generalize across optimization problems (related to machine learning applications as learning a neural network) sharing the same properties. For this sake, the paper casts the problem into reinforcement learning framework and relies on guided policy search (GPS) to explore the space of states and actions. The states are represented by the iterates, the gradients, the objective function values, derived statistics and features, the actions are the update directions of parameters to be learned. To make the formulated problem tractable, some simplifications are introduced (the policies are restricted to gaussian distributions family, block diagonal structure is imposed on the involved parameters). The mean of the stationary non-linear policy of GPS is modeled as a recurrent network with parameters to be learned. A hatch of how to learn the overall process is presented. Finally experimental evaluations on synthetic or real datasets are conducted to show the effectiveness of the approach.
Comments
-------------
- The overall idea of the paper, learning how to optimize, is very seducing and the experimental evaluations (comparison to normal optimizers and other meta-learners) tend to conclude the proposed method is able to learn the behavior of an optimizer and to generalize to unseen problems.
- Materials of the paper sometimes appear tedious to follow, mainly in sub-sections 3.4 and 3.5. It would be desirable to sum up the overall procedure in an algorithm. Page 5, the term $\omega$ intervening in the definition of the policy $\pi$ is not defined.
- The definitions of the statistics and features (state and observation features) look highly elaborated. Can authors provide more intuition on these precise definitions? How do they impact for instance changing the time range in the definition of $\Phi$) in the performance of the meta-learner?
- Figures 3 and 4 illustrate some oscillations of the proposed approach. Which guarantees do we have that the algorithm will not diverge as L2LBGDBGD does? How long should be the training to ensure a good and stable convergence of the method?
- An interesting experience to be conducted and shown is to train the meta-learner on another dataset (CIFAR for example) and to evaluate its generalization ability on the other sets to emphasize the effectiveness of the method.


Jointly Learning to Construct and Control Agents using Deep Reinforcement Learning
The physical design of a robot and the policy that controls its motion are inherently coupled. However, existing approaches largely ignore this coupling, instead choosing to alternate between separate design and control phases, which requires expert intuition throughout and risks convergence to suboptimal designs. In this work, we propose a method that jointly optimizes over the physical design of a robot and the corresponding control policy in a model-free fashion, without any need for expert supervision. Given an arbitrary robot morphology, our method maintains a distribution over the design parameters and uses reinforcement learning to train a neural network controller. Throughout training, we refine the robot distribution to maximize the expected reward. This results in an assignment to the robot parameters and neural network policy that are jointly optimal. We evaluate our approach in the context of legged locomotion, and demonstrate that it discovers novel robot designs and walking gaits for several different morphologies, achieving performance comparable to or better than that of hand-crafted designs.

This is a well written paper, very nice work.
It makes progress on the problem of co-optimization of the physical parameters of a design
and its control system.  While it is not the first to explore this kind of direction,
the method is efficient for what it does; it shows that at least for some systems,
the physical parameters can be optimized without optimizing the controller for each
individual configuration. Instead, they require that the same controller works over an evolving
distribution of the agents.  This is a simple-but-solid insight that makes it possible
to make real progress on a difficult problem.
Pros:  simple idea with impact;  the problem being tackled is a difficult one
Cons:  not many;  real systems have constraints between physical dimensions and the forces/torques they can exert
       Some additional related work to consider citing.  The resulting solutions are not necessarily natural configurations,
   given the use of torques instead of musculotendon-modeling.  But the current system is a great start.
The introduction could also promote that over an evolutionary time-frame, the body and
control system (reflexes, muscle capabilities, etc.) presumably co-evolved.
The following papers all optimize over both the motion control and the physical configuration of the agents.
They all use derivative free optimization, and thus do not require detailed supervision or precise models
of the dynamics.
- Geijtenbeek, T., van de Panne, M., & van der Stappen, A. F. (2013). Flexible muscle-based locomotion
  for bipedal creatures. ACM Transactions on Graphics (TOG), 32(6), 206.
  (muscle routing parameters, including insertion and attachment points) are optimized along with the control).
- Sims, K. (1994, July). Evolving virtual creatures. In Proceedings of the 21st annual conference on
  Computer graphics and interactive techniques (pp. 15-22). ACM.
  (a combination of morphology, and control are co-optimized)
- Agrawal, S., Shen, S., & van de Panne, M. (2014). Diverse Motions and Character Shapes for Simulated
  Skills. IEEE transactions on visualization and computer graphics, 20(10), 1345-1355.
  (diversity in control and diversity in body morphology are explored for fixed tasks)
re: heavier feet requiring stronger ankles
This commment is worth revisiting.  Stronger ankles are more generally correlated with
a heavier body rather than heavy feet, given that a key role of the ankle is to be able
to provide a "push" to the body at the end of a stride, and perhaps less for "lifting the foot".
I am surprised that the optimization does not converge to more degenerate solutions
given that the capability to generate forces and torques is independent of the actual
link masses, whereas in nature, larger muscles (and therefore larger masses) would correlate
with the ability to generate larger forces and torques.  The work of Sims takes these kinds of
constraints loosely into account (see end of sec 3.3).
It would be interesting to compare to a baseline where the control systems are allowed to adapt to the individual design parameters.
I suspect that the reward function that penalizes torques in a uniform fashion across all joints would
favor body configurations that more evenly distribute the motion effort across all joints, in an effort
to avoid large torques.
Are the four mixture components over the robot parameters updated independently of each other
when the parameter-exploring policy gradients updates are applied?  It would be interesting
to know a bit more about how the mean and variances of these modes behave over time during
the optimization, i.e., do multiple modes end up converging to the same mean? What does the
evolution of the variances look like for the various modes?


Deep neural networks have been tremendously successful in a number of tasks. One of the main reasons for this is their capability to automatically learn representations of data in levels of abstraction, increasingly disentangling the data as the internal transformations are applied. In this paper we propose a novel regularization method that penalize covariance between dimensions of the hidden layers in a network, something that benefits the disentanglement. This makes the network learn nonlinear representations that are linearly uncorrelated, yet allows the model to obtain good results on a number of tasks, as demonstrated by our experimental evaluation. The proposed technique can be used to find the dimensionality of the underlying data, because it effectively disables dimensions that aren't needed. Our approach is simple and computationally cheap, as it can be applied as a regularizer to any gradient-based learning model.

This paper presents a regularization mechanism which penalizes covariance between all dimensions in the latent representation of a neural network. This penalty is meant to disentangle the latent representation by removing shared covariance between each dimension.
While the proposed penalty is described as a novel contribution, there are multiple instances of previous work which use the same type of penalty (Cheung et. al. 2014, Cogswell et. al. 2016). Like this work, Cheung et. al. 2014 propose the XCov penalty which penalizes cross-covariance to disentangle subsets of dimensions in the latent representation of autoencoder models. Cogswell et. al. 2016 also proposes a similar penalty (DeCov) to this work for reducing overfitting in supervised learning.
The novel contribution of the regularizer proposed in this work is that it also penalizes the variance of individual dimensions along with the cross-covariance. Intuitively, this should lead to dimensionality reduction as the model will discard variance in dimensions which are unnecessary for reconstruction. But given the similarity to previous work, the authors need to quantitatively evaluate the value in additionally penalizing variance of each dimension as compared with earlier work. Cogswell et. al. 2016 explicitly remove these terms from their regularizer to prevent the dynamic range of the activations from being unnecessarily rescaled.  It would be helpful to understand how this approach avoids this issues - i.e.,  if you penalize all the variance terms then you could just be arbitrarily rescaling the activities, so what prevents this trivial solution?
There doesn't appear to be a definition of the L1 penalty this paper compares against and it's unclear why this is a reasonable baseline. The evaluation metrics this work uses (MAPC, CVR, TdV, UD) need to be justified more in the absence of their use in previous work. While they evaluate their method on non-toy dataset such as CIFAR, they do not show what the actual utility of their proposed regularizer serves for such a dataset beyond having no-regularization at all. Again, the utility of the evaluation metrics proposed in this work is unclear.
The toy examples are kind of interesting but it would be more compelling if the dimensionality reduction aspect extended to real datasets.
> Our method has no penalty on the performance on tasks evaluated in the experiments, while it does disentangle the data
This needs to be expanded in the results as all the results presented appear to show Mean Squared Error increasing when increasing the weight of the regularization penalty.


Workshop track -ICLR 2018 GEOSEQ2SEQ: INFORMATION GEOMETRIC SEQUENCE-TO-SEQUENCE NETWORKS
The Fisher information metric is an important foundation of information geometry, wherein it allows us to approximate the local geometry of a probability distribution. Recurrent neural networks such as the Sequence-to-Sequence (Seq2Seq) networks that have lately been used to yield state-of-the-art performance on speech translation or image captioning have so far ignored the geometry of the latent embedding, that they iteratively learn. We propose the information geometric Seq2Seq (GeoSeq2Seq) network which abridges the gap between deep recurrent neural networks and information geometry. Specifically, the latent embedding offered by a recurrent network is encoded as a Fisher kernel of a parametric Gaussian Mixture Model, a formalism common in computer vision. We utilise such a network to predict the shortest routes between two nodes of a graph by learning the adjacency matrix using the GeoSeq2Seq formalism; our results show that for such a problem the probabilistic representation of the latent embedding supersedes the non-probabilistic embedding by 10-15%.

==== UPDATE AFTER REVIEWER RESPONSE
I apologize to the authors for my late response.
I appreciate the reviewer responses, and they are helpful on a number of
fronts. Still, there are several problematic points.
First, as the authors anticipated, I question whether the geometric encoding
operations can be included in an end-to-end learning setting. I can imagine
several arguments why an end-to-end algorithm may not be preferred, but the
authors do not offer any such arguments.
Second, I am still interested in more discussion of the empirical investigation
into the behavior of the algorithm. For example, "Shortest" and "Successful"
in Table 1 still do not really capture how close "successful but not shortest"
paths are to optimal.
The authors have addressed a number of my concerns, but there
are a few outstanding concerns. Also, other reviewers are much more familiar
with the work than myself. I defer to their judgement after the updates.
==== Original review
In this work, the authors propose an approach to adapt latent representations to account for local geometry in the embedding space. They show modest improvement compared to reasonable baselines.
While I find the idea of incorporating information geometry into embeddings very promising, the current work omits a number of key details that would allow the reader to draw deeper connections between the two (specific comments below). Additionally, the experiments are not particularly insightful.
I believe a substantially revised version of the paper could address most of my concerns; still, I find the current version too preliminary for publication.
=== Major comments / questions
The transformation from context vectors into Fisher vectors is not clear. Presumably, shortest paths in the training data have different lengths, and thus produce different numbers of context vectors. Does the GMM treat all of these independently (regardless of sample)? or is a separate GMM somehow trained for each training sequence? The same question applies to the VLAD-based approach.
In a related vein, it is not clear to what extent this method depends on the sequential nature of the considered networks. In particular, could a similar approach be applied to latent space embeddings from non-sequential models?
It is not clear if the geometric encoding operations are differentiable, or more generally, the entire training algorithm is not clear.
The choice to limit the road network graph feels quite arbitrary. Why was this done?
Deep models are known to be sensitive to the choice of hyperparameters. How were these chosen? was a validation set used in addtion to the training and testing sets?
The target for training is very unclear. Throughout Sections 1 and 2, the aim of the paper appears to be to learn shortest paths; however, Section 3 states that the “network is capable of learning the adjacency matrix”, and the caption for Figure 2 suggests that “[t]he adjacency matrix is iteratively learnt (sic)....” However, calculating such training error for back-propagation/optimization would seem to rely on *already knowing* the adjacency matrix.
The performed experiments are minimal and offer very little insight into what is learned. For example, does the model predict “short” shortest paths better than longer ones? what do the “valid but not optimal” paths look like? are they close to optimal? what do the invalid paths look like? does it seem to learn parts of the road network better than others? sparse parts of the network? dense parts?
=== Minor comments / questions
The term “context vector” is not explicitly defined or described. Based on the second paragraph in the “Fisher encoding” section, I assume these are the latent states for each element in the shortest path sequences.
Is the graph directed? weighted? by Euclidean distance? (Roads are not necessarily straight, so the Euclidean distance from intersection to intersection may not accurately reflect the distance in some cases.)
Are the nodes sampled uniformly at random for creating the training data?
Is the choice to use a diagonal covariance matrix (as opposed to some more flexible one) a computational choice? or does the theory justify this choice?
Roughly, what are the computational resources required for training?
The discussion should explain “condition number” in more detail.
Do the “more precise” results for the Fisher encoding somehow rely on an infinite mixture? or, how much does using only a single component in the GMM affect the results?
It is not clear what “features” and “dictionary elements” are in the context of VLAD.
What value of k was used for K-means clustering for VLAD?
It is not possible to assess the statistical significance of the presented experimental results. More datasets (or different parts of the road network) or cross-validation should be used to provide an indication of the variance of each method.
=== Typos, etc.
The paper includes a number of runon sentences and other small grammatical mistakes. I have included some below.
The first paragraph in Section 2.2 in particular needs to be edited.
The references are inconsistently and improperly (e.g., “Turing” should be capitalized) formatted.
It seems like that $q_{ik} \in \{0,1\}$ for the hard assignments in clustering.


Published as a conference paper at ICLR 2018 MAXIMUM A POSTERIORI POLICY OPTIMISATION
We introduce a new algorithm for reinforcement learning called Maximum aposteriori Policy Optimisation (MPO) based on coordinate ascent on a relativeentropy objective. We show that several existing methods can directly be related to our derivation. We develop two off-policy algorithms and demonstrate that they are competitive with the state-of-the-art in deep reinforcement learning. In particular, for continuous control, our method outperforms existing methods with respect to sample efficiency, premature convergence and robustness to hyperparameter settings.

The paper presents a new algorithm for inference-based reinforcement learning for deep RL. The algorithm decomposes the policy update in two steps, an E and an M-step. In the E-step, the algorithm estimates a variational distribution q which is subsequentially used for the M-step to obtain a new policy. Two versions of the algorithm are presented, using a parametric or a non-parametric (sample-based) distribution for q. The algorithm is used in combination with the retrace algorithm to estimate the q-function, which is also needed in the policy update.
This is a well written paper presenting an interesting algorithm. The algorithm is similar to other inference-based RL algorithm, but is the first application of inference based RL to deep reinforcement learning. The results look very promising and define a new state of the art or deep reinforcement learning in continuous control, which is a very active topic right now. Hence, I think the paper should be accepted.
I do have a few comments / corrections / questions about the paper:
- There are several approaches that already use the a combination of the KL-constraint with reverse KL on a non-parametric distribution and subsequently an M-projection to obtain again a parametric distribution, see HiREPS, non-parametric REPS [Hoof2017, JMLR] or AC-REPS [Wirth2016, AAAI]. These algorithms do not use the inference-based view but the trust region justification. As in the non-parametric case, the asymptotic performance guarantees from the EM framework are gone, why is it beneficial to formulate it with EM instead of directly with a trust region of the expected reward?
- It is not clear to me whether the algorithm really optimizes the original maximum a posteriori objective defined in Equation 1. First, alpha changes every iteration of the algorithm while the objective assumes that alpha is constant. This means that we change the objective all the time which is theoretically a bit weird. Moreover, the presented algorithm also changes the prior all the time (in order to introduce the 2nd trust region) in the M-step. Again, this changes the objective, so it is unclear to me what exactly is maximised in the end. Would it not be cleaner to start with the average reward objective (no prior or alpha) and then introduce both trust regions just out of the motivation that we need trust regions in policy search? Then the objective is clearly defined.
- I did not get whether the additional "one-step KL regularisation" is obtained from the lower bound or just added as additional regularisation? Could you explain?
- The algorithm has now 2 KL constraints, for E and M step. Is the epsilon for both the same or can we achieve better performance by using different epsilons?
- I think the following experiments would be very informative:
   - MPO without trust region in M-step

   - MPO without retrace algorithm for getting the Q-value
   - test different epsilons for E and M step


A DEEP REINFORCED MODEL FOR ABSTRACTIVE SUMMARIZATION
Attentional, RNN-based encoder-decoder models for abstractive summarization have achieved good performance on short input and output sequences. For longer documents and summaries however these models often include repetitive and incoherent phrases. We introduce a neural network model with a novel intraattention that attends over the input and continuously generated output separately, and a new training method that combines standard supervised word prediction and reinforcement learning (RL). Models trained only with supervised learning often exhibit "exposure bias" -they assume ground truth is provided at each step during training. However, when standard word prediction is combined with the global sequence prediction training of RL the resulting summaries become more readable. We evaluate this model on the CNN/Daily Mail and New York Times datasets. Our model obtains a 41.16 ROUGE-1 score on the CNN/Daily Mail dataset, an improvement over previous state-of-the-art models. Human evaluation also shows that our model produces higher quality summaries.

The paper proposes a model for abstractive document summarization using a self-critical policy gradient training algorithm, which is mixed with maximum likelihood objective. The Seq2seq architecture incorporates both intra-temporal and intra-decoder attention, and a pointer copying mechanism. A hard constraint is imposed during decoding to avoid trigram repetition. Most of the modelling ideas already exists, but this paper show how they can be applied as a strong summarization model.
The approach obtains strong results on the CNN/Daily Mail and NYT datasets. Results show that intra-attention improves performance for only one of the datasets. RL results are reported with only the best-performing attention setup for each dataset. My concern with that is that the authors might be using the test set for model selection; It is not a priori clear that the setup that works better for ML should also be better for RL, especially as it is not the same across datasets. So I suggest that results for RL should be reported with and without intra-attention on both datasets, at least on the validation set.
It is shown that intra-decoder attention decoder improves performance on longer sentences. It would be interesting to see more analysis on this, especially analyzing what the mechanism is attending to, as it is less clear what its interpretation should be than for intra-temporal attention. Further ablations such as the effect of the trigram repetition constraint will also help to analyse the contribution of different modelling choices to the performance.
For the mixed decoding objective, how is the mixing weight chosen and what is its effect on performance? If it is purely a scaling factor, how is the scale quantified? It is claimed that readability correlates with perplexity, so it would be interesting to see perplexity results for the models. The lack of correlation between automatic and human evaluation raises interesting questions about the evaluation of abstractive summarization that should be investigated further in future work.
This is a strong paper that presents a significant improvement in document summarization.