Published as a conference paper at ICLR 2018 ATTACKING BINARIZED NEURAL NETWORKS
Neural networks with low-precision weights and activations offer compelling efficiency advantages over their full-precision equivalents. The two most frequently discussed benefits of quantization are reduced memory consumption, and a faster forward pass when implemented with efficient bitwise operations. We propose a third benefit of very low-precision neural networks: improved robustness against some adversarial attacks, and in the worst case, performance that is on par with full-precision models. We focus on the very low-precision case where weights and activations are both quantized to ±1, and note that stochastically quantizing weights in just one layer can sharply reduce the impact of iterative attacks. We observe that non-scaled binary neural networks exhibit a similar effect to the original defensive distillation procedure that led to gradient masking, and a false notion of security. We address this by conducting both black-box and white-box experiments with binary models that do not artificially mask gradients.

This paper starts by gently going over the concept of adversarial attacks on neural networks (black box vs white box, reactive vs proactive, transfer of attacks, linearity hypothesis), as well as low-precision nets and their deployement advantages.
Adversarial examples are introduced as a norm-measurable deviation from natural inputs to a system. We are reminded of adversarial training, and of the fact that binarized nets are highly non linear due to the nature of their weights and activations.
This paper then proposes to examine the robustness of binarized neural networks to adversarial attacks on MNIST and CIFAR-10.
The quantization scheme used here is v32 Conv2D -> ReLU -> BNorm -> sign -> bit Conv2D -> ReLU -> Scalar -> BNorm -> sign, but sign is really done with the stochastic quantization method of Courbariaux et al, even at test time (in order to make it more robust).
What the experimental results show:
- High capacity BNNs are usually more robust to white-box attacks than normal networks, probably because the gradient information that an adversary would use becomes very poor as training progresses.
- BNNs are harder to properly train with adversarial examples because of the polarized weight distribution that they induce
- Against black-box attacks, it seems there is little difference between NNs and BNNs.
Some comments and questions:
- In figure 1 I'm not sure what "Scalar" refers to, and it is not explained in the paper (nor could I find it explained in Papernot et al 2017a).
- Do you adopt the "Shift based Batch Normalizing Transform" of Courbariaux et al? If not, why?
- It might be worth at least _quickly_ explaining what the 'Carlini-Wagner L2 from CleverHans' is rather than simply offering a citation with no explanation. Idem for 'smooth substitute model black-box misclassificiation attack'. We often assume our readers know most of what we know, but I find this is often not the case and can discourage the many newcomers of our field.
- "Running these attacks to 1000 iterations [...], therefore we believe this targeted attack represents a fairly substantial level of effort on behalf of the adversary." while true for us researchers, computational difficulty will not be a criterion to stop for state actor or multinational tech companies, unless it can be proven that e.g. the number of iterations needs to grow exponentially (or in some other unreasonable way) in order to get reliable attacks.
- "MLP with binary units 3 better", 'as in Fig.' is missing before '3', or something of the sort.
- You say "We postpone a formal explanation of this outlier for the discussion." but really you're explaining it in the next paragraph (unless there's also another explanation I'm missing). Training BNNs with adversarial examples is hard.
- You compare stochastic BNNs with deterministic NNs, but not with stochastic NNs. What do you think would happen? Some of arguments that you make in favour of BNNs could also maybe be applied to stochastic NNs.
My opinions on this paper:
- Novelty: it certainly seems to be the first time someone has tackled BNNs and adversarial examples
- Relevance: BNNs can be a huge deal when deploying applications, it makes sense to study their vulnerabilities
- Ease of understanding: To me the paper was mostly easy to understand, yet, considering there is no page limit in this conference, I would have buffed up the appendix, e.g. to include more details about the attacks used and how various hyperparameters affect things.
- Clarity: I feel like some details are lacking that would hinder reproducing and extending the work presented here. Mostly, it isn't always clear why the chosen prodecures and hyperparameters were chosen (wrt the model being a BNN)
- Method: I'm concerned by the use of MNIST to study such a problem. MNIST is almost linearly separable, has few examples, and given the current computational landscape, much better alternatives are available (SVHN for example if you wish to stay in the digits domain). Concerning black-box attacks, it seems that BNNs less beneficial in a way; trying more types of attacks and/or delving a bit deeping into that would have been nice. The CIFAR-10 results are barely discussed.
Overall I think this paper is interesting and relevant to ICLR. It could have stronger results both in terms of the datasets used and the variety of attacks tested, as well as some more details concerning how to perform adversarial training with BNNs (or why that's not a good idea).


Published as a conference paper at ICLR 2018 UNSUPERVISED CIPHER CRACKING USING DISCRETE GANS
This work details CipherGAN, an architecture inspired by CycleGAN used for inferring the underlying cipher mapping given banks of unpaired ciphertext and plaintext. We demonstrate that CipherGAN is capable of cracking language data enciphered using shift and Vigenère ciphers to a high degree of fidelity and for vocabularies much larger than previously achieved. We present how CycleGAN can be made compatible with discrete data and train in a stable way. We then prove that the technique used in CipherGAN avoids the common problem of uninformative discrimination associated with GANs applied to discrete data.

SUMMARY
The paper considers the problem of using cycle GANs to decipher text encrypted with historical ciphers. Also it presents some theory to address the problem that discriminating between the discrete data and continuous prediction is too simple. The model proposed is a variant of the cycle GAN in which in addition embeddings helping the Generator are learned for all the values of the discrete variables.
The log loss of the GAN is replaced by a quadratic loss and a regularization of the Jacobian of the discriminator. Experiments show that the method is very effective.
REVIEW
The paper considers an interesting and fairly original problem and the overall discussion of ciphers is quite nice. Unfortunately, my understanding is that the theory proposed in section 2 does not correspond to the scheme used in the experiments (contrarily to what the conclusion suggest and contrarily to what the discussion of the end of section 3, which says that using embedding is assumed to have an equivalent effect to using the methodology considered in the theoretical part). Another important concern is with the proof: there seems to be an unmotivated additional assumption that appears in the middle of the proof of Proposition 1 + some steps need to be clarified (see comment 16 below).
The experiments do not have any simple baseline, which is somewhat unfortunate.
DETAILED COMMENTS:
1- The paper makes a few bold and debatable statements:
line 9 of section 1
"Such hand-crafted features have fallen out of favor (Goodfellow et al., 2016) as a
result of their demonstrated inferiority to features learned directly from data in end-to-end learning
frameworks such as neural networks"
This is certainly an overstatement and although it might be true for specific types of inputs it is not universally true, most deep architectures rely on a human-in-the-loop and there are number of areas where human crafted feature are arguably still relevant, if only to specify what is the input of a deep network: there are many domains where the notion of raw data does not make sense, and, when it does, it is usually associated with a sensing device that has been designed by a human and which implicitly imposes what the data is based on human expertise.
2- In the last paragraph of the introduction, the paper says that previous work has only worked on vocabularies of 26 characters while the current paper tackles word level ciphers with 200 words. But, isn't this just a matter of scalability and only possible with very large amounts of text? Is it really because of an intrinsic limitation or lack of scalability of previous approaches or just because the authors of the corresponding papers did not care to present larger scale experiments?
3- The discussion at the top of page 5 is difficult to follow. What do you mean when you say "this motivates the benefits of having strong curvature globally, as opposed to linearly between etc"
Which curvature are we talking about? and what how does the "as opposed to linearly" mean? Should we understand "as opposed to having curvature linearly interpolated between etc" or "as opposed to having a linear function"? Please clarify.
4- In the same paragraph: what does "a region that has not seen the Jacobian norm applied to it" mean? How is a norm applied to a region? I guess that what you mean is that the generator G might creates samples in a part of the space where the function F has not yet been learned and is essentially close to 0. Is this what you mean?
5- I do not understand why the paper introduces WGAN since in the end it does not use them but uses a quadratic loss, introduced in the first display of section 4.3.
6- The paper makes a theoretical contribution which supports replacing the sample y by a sample drawn from a region around y. But it seems that this is not used in the experiment and that the authors consider that the introduction of the embedding is a substitution for this. Indeed, in the last paragraph of section 3.1, the paper says "we make the assumption that the training of the embedding vectors approximates random sampling similar to what is described in Proposition 1". This does not make any sense to me because the embedding vectors map each y deterministically to a single point, and so the distribution on the corresponding vectors is still a fixed discrete distribution. This gives me this impression that the proposed theory does not match what is used in the experiments.
(The last sentence of section 3.1, which is commenting on this and could perhaps clarify the situation is ill formed with two verbs.)
7- In the definitions: "A discriminator is said to perform uninformative discrimination" etc. -> It seems that the choice of the word uninformative would be misleading: an uninformative discrimination would be a discrimination that completely fails, while what the condition is saying it that it cannot perform perfect discrimination. I would thus suggest to call this "imperfect discrimination".
8- It seems that the same embedding is used in X space and in Y space (from equations 6 and 7). Is there any reason for that? I would seem more natural to me to introduce two different embeddings since the objects are a priori different...
Actually I don't understand how the embeddings can be the same in the Vignere code case since time taken into account one one side.
9- On the 5th line after equation (7), the paper says "the embeddings... are trained to minimize L_GAN and L_cyc, meaning... and are easy to discriminate" -> This last part of the sentence seems wrong to me. The discriminator is trying to maximize L_GAN and so minimizing w.r.t. to the embedding is precisely trying to prevent to the discriminator to tell apart too easily the true elements from the estimated ones.
In fact the regularization of the Jacobian that will be preventing the discriminator to vary too quickly in space is more likely to explain the fact that the discrimination is not too easy to do between the true and mapped embeddings. This might be connected to the discussion at the top of page 5. Since there are no experiments with alpha different than the default value = 10, this is difficult to assess.
10-The Vigenere cipher is explained again at the end of section 4.2 when it has already been presented in section 1.1
11- Concerning results in Table 2: I do not see why it would not be possible to compare the performance of the method with classical frequency analysis, at least for the character case.
12- At the beginning of section 4.3, the text says that the log loss was replaced with the quadratic loss, but without giving any reason. Could you explain why.
13- The only comparison of results with and without embeddings is presented in the curves of figure 3, for Brown-W with a vocabulary of 200 words. In that case it helps. Could the authors report systematically results about all cases? (I guess this might however be the only hard case...)
14- It would be useful to have a brief reminder of the architecture of the neural network (right now the reader is just refered to Zhu et al., 2017): how many layers, how many convolution layers etc.
The same comment applies for the way the position of the letter/word in the text appear is in encoded in a feature that is provided as input to the neural network: it would be nice if the paper could provide a few details here and be more self contained. (The fact that the engineering of the time feature can "dramatically" improve the performance of the network should be an argument to convince the authors that hand-crafted feature have not fallen out of favor completely yet...)
15- I disagree with the statement made in the conclusion that the proposed work "empirically confirms [...] that the use of continuous relaxation of discrete variable facilitates [...] and prevents [...]" because for me the proposed implementation does not use at all the theoretical idea of continuous relaxation proposed in the paper, unless there is a major point that I am missing.
16- I have two issues with the proof in the appendix
a) after the first display of the last page the paper makes an additional assumption which is not announced in the statement of the theorem, which is that two specific inequality hold...
Unless I am mistaken this assumption is never proven (later or earlier). Given that this inequality is just "the right inequality to get the proof go through" and given that there are no explanation for why this assumption is reasonable, to me this invalidates the proof. The step of going from G(S_y) to S_(G(y)) seems delicate...
b) If we accept these inequalities, the determinant of the Jacobian (the notation is not defined) of F at (x_bar) disappears from the equations, as if it could be assumed to be greater than one. If this is indeed the case, please provide a justification of this step.
17- A way to address the issue of trivial discrimination in GANs with discrete data has been proposed in

Luc, P., Couprie, C., Chintala, S., & Verbeek, J. (2016). Semantic segmentation using adversarial networks. arXiv preprint arXiv:1611.08408.
The authors should probably reference this paper.
18- Clarification of the Jacobian regularization: in equation (3), the Jacobian computed seems to be w.r.t D composed with F while in equation (8) it is only the Jacobian of D. Which equation is the correct one?
TYPOS:
Proposition 1: the if-then statement is broken into two sentences separated by a full point and a carriage return.
sec. 4.3 line 10 we use a cycle loss *with a regularization coefficient* lambda=1 (a piece of the sentence is missing)
sec. 4.3 lines 12-13 the learning rates given are the same at startup and after "warming up"...
In the appendix:
3rd line of proof of prop 1: I don' understand "countably infinite finite sequences of vectors lying in the vertices of the simplex" -> what is countable infinite here? The vertices?


Recurrent auto-encoder model can summarise sequential data through an encoder structure into a fixed-length vector and then reconstruct into its original sequential form through the decoder structure. The summarised information can be used to represent time series features. In this paper, we propose relaxing the dimensionality of the decoder output so that it performs partial reconstruction. The fixedlength vector can therefore represent features only in the selected dimensions. In addition, we propose using rolling fixed window approach to generate samples. The change of time series features over time can be summarised as a smooth trajectory path. The fixed-length vectors are further analysed through additional visualisation and unsupervised clustering techniques. This proposed method can be applied in large-scale industrial processes for sensors signal analysis purpose where clusters of the vector representations can be used to reflect the operating states of selected aspects of the industrial system.

The paper describes a sequence to sequence auto-encoder model which is used to learn sequence representations. The authors show that for their application, better performance is obtained when the network is only trained to reconstruct a subset of the data measurements. The paper also presents some visualizations the similarity structure of the learned representations and proposes a window-based method for processing the data.
According to the paper, the experiments are done using a data set which is obtained from measurements of an industrial production process. Figure 2 indicates that reconstructing fewer dimensions of this dataset leads to lower MSE scores. I don’t see how this is showing anything besides the obvious fact that reconstructing fewer dimensions is an easier task than reconstructing all of them.  The only conclusions I can draw from the visual analysis is that the context vectors are more similar to each other when they are obtained from time steps in the data stream which are close to each other. Since the paper doesn’t describe much about the privately owned data at all, there is no possibility to replicate the work. The paper doesn’t frame the work in prior research at all and the six papers it cites are only referred to in the context of describing the architecture.
I found it very hard to distil what the main contribution of this work was according to the paper. There were also not many details about the precise architecture used. It is implied that GRU networks and were used but the text doesn’t actually state this explicitly. By saying so little about the data that was used, it was also not clear what the temporal correlations of the context vectors are supposed to tell us.
The paper describes how existing methods are applied to a specific data set. The benefit of only reconstructing a subset of the input dimensions seems very data specific to me and I find it hard to consider this a novel idea by itself. Presenting sequential data in a windowed format is a standard procedure and not a new idea either. All in all I don't think that the paper presents any new ideas or interesting results.
Pros:
* The visualizations look nice.
Cons:
* It is not clear what the main contribution is.
* Very little information about the data.
* No clear experiments from which conclusions can be drawn.
* No new ideas.
* Not well rooted in prior work.


Generative Adversarial Networks (GANs) learn a generative model by playing an adversarial game between a generator and an auxiliary discriminator, which classifies data samples vs. generated ones. However, it does not explicitly model feature co-occurrences in samples. In this paper, we propose a novel Autoregressive Generative Adversarial Network (ARGAN), that models the latent distribution of data using an autoregressive model, rather than relying on binary classification of samples into data/generated categories. In this way, feature co-occurrences in samples can be more efficiently captured. Our model was evaluated on two widely used datasets: CIFAR-10 and STL-10. Its performance is competitive with respect to other GAN models both quantitatively and qualitatively.

This paper proposes an alternative GAN formulation that replaces the standard binary classification task in the discriminator with a autoregressive model that attempts to capture discriminative feature dependencies on the true data samples.
Summary assessment:
The paper presents a novel perspective on GANs and an interesting conjecture regarding the failure of GANs to capture global consistency. However the experiments do not directly support this conjecture. In addition, both qualitative and quantitative results to not provide significant evidence of the value of this technique over and above the establish methods in the literature.
The central motivation of the method proposed in the paper, is a conjecture that the lack of global consistency in GAN-generated samples is due to the binary classification formulation of the discriminator. While this is an interesting conjecture, I am somewhat unconvinced that this is indeed the cause of the problem. First, I would argue that other high-performing auto-regressive models such as PixelRNN and PixelCNN also seem to lack global consistency. This observation would seem to violate this conjecture. More importantly, the paper does not show any direct empirical evidence in support of this conjecture.
The authors make a very interesting observation in their description of the proposed approach. In discussing an initial variant of the model (Eqns. (5) and (6) and text immediately below), the authors state that attempting to maximize the negative log likelihood of the auto-regressive modelling the generated samples results in unstable training. I would like to see more discussion of this point as it could have some bearing on the difficulty of GAN to model sequential data in general. Does the failure occurs because the auto-regressive discriminator is able to "overfit" the generated samples?
As a result of the observed failure of the formulation given in Eqns. (5) and (6), the authors propose an alternative formulation that explicitly removes the negative likelihood maximization for generated samples. As a result the only objective for the auto-regressive model is an attempt to maximize the log-likelihood of the true data. The authors suggest that this should be sufficient to provide a reliable training signal for the generator. It would be useful if the authors showed a representation of these features (perhaps via T-SNE) for both true data and generated samples.
Empirical results:
The authors experiments show samples (qualitative comparison) and inception scores (quantitative comparison) for 3 variants of the proposed model and compare these to methods in the literature. The comparisons show the proposed model preforms well, but does not exceed the performance of many of the existing methods in the literature.
Also, I fail to observe significantly more global consistency for these samples compared to samples of other SOTA GAN models in the literature. Again, there is no attempt made by the authors to make this direct comparison of global consistency either qualitatively or quantitatively.
Minor comment:
I did not see where PARGAN was defined. Was this the combination of the Auto-regressive GAN with Patch-GAN?


Generative networks are known to be difficult to assess. Recent works on generative models, especially on generative adversarial networks, produce nice samples of varied categories of images. But the validation of their quality is highly dependent on the method used. A good generator should generate data which contain meaningful and varied information and that fit the distribution of a dataset. This paper presents a new method to assess a generator. Our approach is based on training a classifier with a mixture of real and generated samples. We train a generative model over a labeled training set, then we use this generative model to sample new data points that we mix with the original training data. This mixture of real and generated data is thus used to train a classifier which is afterwards tested on a given labeled test dataset. We compare this result with the score of the same classifier trained on the real training data mixed with noise. By computing the classifier's accuracy with different ratios of samples from both distributions (real and generated) we are able to estimate if the generator successfully fits and is able to generalize the distribution of the dataset. Our experiments compare different generators from the VAE and GAN framework on MNIST and fashion MNIST dataset.

The authors propose to evaluate how well generative models fit the training set by analysing their data augmentation capacity, namely the benefit brought by training classifiers on mixtures of real/generated data, compared to training on real data only. Despite the the idea of exploiting generative models to perform data augmentation is interesting, using it as an evaluation metric does not constitute an innovative enough contribution.
In addition, there is a fundamental matter which the paper does not address: when evaluating a generative model, one should always ask himself what purpose the data is generated for. If the aim is to have realistic samples, a visual turing test is probably the best metric. If instead the purpose is to exploit the generated data for classification, well, in this case an evaluation of the impact of artificial data over training is a good option.
PROS:
The idea is interesting.
CONS:
1. The authors did not relate the proposed evaluation metric to other metrics cited (e.g., the inception score, or a visual turing test, as discussed in the introduction). It would be interesting to understand how the different metrics relate. Moreover, the new metric is introduced with the following motivation “[visual Turing test and Inception Score] do not indicate if the generator collapses to a particular mode of the data distribution”. The mode collapse issue is never discussed elsewhere in the paper.
2. Only two datasets were considered, both extremely simple: generating MNIST digits is nearly a toy task nowadays. Different works on GANs make use of CIFAR-10 and SVHN, since they entail more variability: those two could be a good start.
3. The authors should clarify if the method is specifically designed for GANs and VAEs. If not, section 2.1 should contain several other works (as in Table 1).
4. One of the main statements of the paper “Our approach imposes a high entropy on P(Y) and gives unbiased indicator about entropy of both P(Y|X) and P(X|Y)” is never proved, nor discussed.
5. Equation 2 (the proposed metric) is not convincing: taking the maximum over tau implies training many models with different fractions of generated data, which is expensive. Further, how many tau’s one should evaluate? In order to evaluate a generative model one should test on the generated data only (tau=1) I believe. In the worst case, the generator experiences mode collapse and performs badly. Differently, it can memorize the training data and performs as good as the baseline model. If it does actual data augmentation, it should perform better.
6. The protocol of section 3 looks inconsistent with the aim of the work, which is to evaluate data augmentation capability of generative models. In fact, the limit of training with a fixed dataset is that the model ‘sees’ the data multiple times across epochs with the risk of memorizing. In the proposed protocol, the model ‘sees’ the generated data D_gen (which is fixed before training) multiple time across epochs. This clearly does not allow to fully evaluate the capability of the generative model to generate newer and newer samples with significant variability.
Minor:
Section 2.2 might be more readable it divided in two (exploitation and evaluation).


Lifelong multitask learning poses considerable challenges in terms of effectiveness (minimizing prediction errors for all tasks) and overall computational tractability for real-time performance. This paper addresses continuous lifelong multitask learning by jointly re-estimating the inter-task relations (output kernel) and the per-task model parameters at each round, assuming data arrives in a streaming fashion. A new algorithm called Online Output Kernel Learning Algorithm (OOKLA) for lifelong learning setting is proposed. To avoid the memory explosion, a robust budget-limited versions of the proposed algorithm is introduced, which efficiently utilizes the relationships between the tasks to bound the total number of representative examples in the support set. In addition, a two-stage budgeted scheme for efficiently tackling the task-specific budget constraints in lifelong learning is proposed. Empirical results over three datasets indicate superior AUC performance for OOKLA and its budget-limited cousins over strong baselines.

CONTRIBUTION
The main contribution of the paper is not clearly stated.  To the reviewer, It seems “life-long learning” is the same as “online learning”.  However, the whole paper does not define what “life-long learning” is.
The limited budget scheme is well established in the literature.
1. J. Hu, H. Yang, I. King, M. R. Lyu, and A. M.-C. So. Kernelized online imbalanced learning with fixed budgets. In AAAI, Austin Texas, USA, Jan. 25-30 2015.

2. Y. Engel, S. Mannor, and R. Meir. The kernel recursive least-squares algorithm. IEEE Transactions on Signal Processing, 52(8):2275–2285, 2004.
It is not clear what the new proposal in the paper.
WRITING QUALITY
The paper is not well written in a good shape. Many meanings of the equations are not stated clearly, e.g., $phi$ in eq. (7). Furthermore, the equation in algorithm 2 is not well formatted.
DETAILED COMMENTS
1. The mapping function $phi$ appears in Eq. (1) without definition.
2. The last equation in pp. 3 defines the decision function f by an inner product. In the equation, the notation x_t and i_t is not clearly defined.  More seriously, a comma is missed in the definition of the inner product.
3. Some equations are labeled but never referenced, e.g., Eq. (4).
4. The physical meaning of Eq.(7) is unclear.  However, this equation is the key proposal of the paper.   For example, what is the output of the Eq. (7)? What is the main objective of Eq. (7)?  Moreover, what support vectors should be removed by optimizing Eq. (7)?  One main issue is that the notation $phi$ is not clearly defined.   The computation of f-y_r\phi(s_r) makes it hard to understand.  Especially,  the dimension of $phi$ in Eq.(7) is unknown.
ABOUT EXPERIMENTS
1.	It is unclear how to tune the hyperparameters.
2.	In Table 1, the results only report the standard deviation of AUC. No standard deviations of nSV and Time are reported.


One important aspect of generalization in machine learning involves reasoning about previously seen data in new settings. Such reasoning requires learning disentangled representations of data which are interpretable in isolation, but can also be combined in a new, unseen scenario. To this end, we introduce the contextaware learner, a model based on the variational autoencoding framework, which can learn such representations across data sets exhibiting a number of distinct contexts. Moreover, it is successfully able to combine these representations to generate data not seen at training time. The model enjoys an exponential increase in representational ability for a linear increase in context count. We demonstrate that the theory readily extends to a meta-learning setting such as this, and describe a fully unsupervised model in complete generality. Finally, we validate our approach using an adaptation with weak supervision.

The authors propose an extension to the Neural Statistician which can model contexts with multiple partially overlapping features. This model can explain datasets by taking into account covariate structure needed to explain away factors of variation and it can also share this structure partially between datasets.
A particularly interesting aspect of this model is the fact that it can learn these context c as features conditioned on meta-context a, which leads to a disentangled representation.
This is also not dissimilar to ideas used in 'Bayesian Representation Learning With Oracle Constraints' Karaletsos et al 2016 where similar contextual features c are learned to disentangle representations over observations and implicit supervision.
The authors provide a clean variational inference algorithm to learn their model. However, a key problem is the following: the nature of the discrete variables being used makes them hard to be inferred with variational inference. The authors mention categorical reparametrization as their trick of choice, but do not go into empirical details int heir experiments regarding the success of this approach. In fact, it would be interesting to study which level of these variables could be analytically collapsed (such as done in the Semi-Supervised learning work by Kingma et al 2014) and which ones can be sampled effectively using a form of reparametrization.
This also touches on the main criticism of the paper: While the model technically makes sense and is cleanly described and derived,  the empirical evaluation is on the weak side and the rich properties of the model are not really shown off. It would be interesting if the authors could consider adding a more illustrative experiment and some more empirical results regarding inference in this model and the marginal structures that can be learned with this model in controlled toy settings.
Can the model recover richer structure that was imposed during data generation? How limiting is the learning of a?
How does the likelihood of the model behave under the circumstances?
The experiments do not really convey how well this all will work in practice.


A key challenge in model-based reinforcement learning (RL) is to synthesize computationally efficient and accurate environment models. We show that carefully designed models that learn predictive and compact state representations, also called state-space models, substantially reduce the computational costs for predicting outcomes of sequences of actions. Extensive experiments establish that state-space models accurately capture the dynamics of Atari games from the Arcade Learning Environment (ALE) from raw pixels. Furthermore, RL agents that use Monte-Carlo rollouts of these models as features for decision making outperform strong model-free baselines on the game MS PACMAN, demonstrating the benefits of planning using learned dynamic state abstractions.

Summary:
This paper studies how to learn (hidden)-state-space models of environment dynamics, and integrate them with Imagination-Augmented Agents (I2A). The paper considers single-agent problems and tests on Ms Pacman etc.
There are several variations of the hidden-state space [ds]SSM model: using det/stochastic latent variables + using det/stochastic decoders. In the stochastic case, learning is done using variational methods.
[ds]SSM is integrated with I2A, which generates rollouts of future states, based on the inferred hidden states from the d/sSSM-VAE model. The rollouts are then fed into the agent's policy / value function.
Main results seem to be:
1. Experiments on learning the forward model, show that latent forward models work better and faster than naive AR models on several Atari games, and better than fully model-free baselines.
2. I2A agents with latent codes work better than model-free models or I2A from pixels. Deterministic latent models seem to work better than stochastic ones.
Pro:
- Relatively straightforward idea: learn the forward model on hidden states, rather than raw states.
- Writing is clear, although a bit dense in places.
Con:
- Paper only shows training curves for MS Pacman. What about the other games from Table 1?
- The paper lacks any visualization of the latent codes. What do they represent? Can we e.g. learn a raw-state predictor from the latent codes?
- Are the latent codes relevant in the stochastic model? See e.g. the discussion in "Variational Lossy Autoencoder" (Chen et al. 2016)
- Experiments are not complete (e.g. for AR, as noted in the paper).
- The games used are fairly reactive (i.e. do not require significant long-term planning), and so the sequential hidden-state-space model does not have to capture long-term dependencies. It would be nice to see how this technique fares on Montezuma's revenge, for instance.
Overall:
The paper proposes a simple idea that seems to work well on reactive 1-agent games. However, the paper could give more insights into *how* this works: e.g. a better qualitative inspection of the learned latent model, and how existing questions surrounding sequential stochastic model affect the proposed method. Also, not all baseline experiments are done, and the impact on training is only evaluated on 1 game.
Detailed:
-


Determining the optimal order in which data examples are presented to Deep Neural Networks during training is a non-trivial problem. However, choosing a non-trivial scheduling method may drastically improve convergence. In this paper, we propose a Self-Paced Learning (SPL)-fused Deep Metric Learning (DML) framework, which we call Learning Embeddings for Adaptive Pace (LEAP). Our method parameterizes mini-batches dynamically based on the easiness and true diverseness of the sample within a salient feature representation space. In LEAP, we train an embedding Convolutional Neural Network (CNN) to learn an expressive representation space by adaptive density discrimination using the Magnet Loss. The student CNN classifier dynamically selects samples to form a minibatch based on the easiness from cross-entropy losses and true diverseness of examples from the representation space sculpted by the embedding CNN. We evaluate LEAP using deep CNN architectures for the task of supervised image classification on MNIST, FashionMNIST, CIFAR-10, CIFAR-100, and SVHN. We show that the LEAP framework converges faster with respect to the number of mini-batch updates required to achieve a comparable or better test performance on each of the datasets.

The authors purpose a method for creating mini batches for a student network by using a second learned representation space to dynamically selecting  examples by their 'easiness and true diverseness'. The framework is detailed and results on MNIST, cifar10 and fashion-MNIST are presented. The work presented is novel but there are some notable omissions:
 - there are no specific numbers presented to back up the improvement claims; graphs are presented but not specific numeric results
- there is limited discussion of the computational cost of the framework presented
- there is no comparison to a baseline in which the additional learning cycles used for learning the embedding are used for training the student model.
- only small data sets are evaluated. This is unfortunate because if there are to be large gains from this approach, it seems that they are more likely to be found in the domain of large scale problems, than toy data sets like mnist.
**edit
In light of the changes made, and in particular the performance gains achieved on CIFAR-100, i have increased my ratting from a 4 to a 6


Published as a conference paper at ICLR 2018 REINFORCEMENT LEARNING ALGORITHM SELECTION
This paper formalises the problem of online algorithm selection in the context of Reinforcement Learning (RL). The setup is as follows: given an episodic task and a finite number of off-policy RL algorithms, a meta-algorithm has to decide which RL algorithm is in control during the next episode so as to maximize the expected return. The article presents a novel meta-algorithm, called Epochal Stochastic Bandit Algorithm Selection (ESBAS). Its principle is to freeze the policy updates at each epoch, and to leave a rebooted stochastic bandit in charge of the algorithm selection. Under some assumptions, a thorough theoretical analysis demonstrates its near-optimality considering the structural sampling budget limitations. ESBAS is first empirically evaluated on a dialogue task where it is shown to outperform each individual algorithm in most configurations. ESBAS is then adapted to a true online setting where algorithms update their policies after each transition, which we call SSBAS. SSBAS is evaluated on a fruit collection task where it is shown to adapt the stepsize parameter more efficiently than the classical hyperbolic decay, and on an Atari game, where it improves the performance by a wide margin.

The paper considers the problem of online selection of RL algorithms. An algorithm selection (AS) strategy called ESBAS is proposed. It works in a sequence of epochs of doubling length, in the following way: the algorithm selection is based on a UCB strategy, and the parameters of the algorithms are not updated within each epoch (in order that the returns obtained by following an algorithm be iid). This selection allows ESBAS to select a sequence of algorithms within an epoch to generate a return almost as high as the return of the best algorithm, if no learning were made. This weak notion of regret is captured by the short-sighted pseudo regret.
Now a bound on the global regret is much harder to obtain because there is no way of comparing, without additional assumption, the performance of a sequence of algorithms to the best algorithm had this one been used to generate all trajectories from the beginning. Here it is assumed that all algorithms learn off-policy. However this is not sufficient, since learning off-policy does not mean that an algorithm is indifferent to the behavior policy that has generated the data. Indeed even for the most basic off-policy algorithms, such as Q-learning, the way data have been collected is extremely important, and collecting transitions using that algorithm (such as epsilon-greedy) is certainly better than following an arbitrary policy (such as uniformly randomly, or following an even poorer policy which would not explore at all). However the authors seem to state an equivalence between learning off-policy and fairness of learning (defined in Assumption 3). For example in their conclusion they mention “Fairness of algorithm evaluation is granted by the fact that the RL algorithms learn off-policy”. This is not correct. I believe the main assumption made in this paper is the Assumption 3 (and not that the algorithms learn off-policy) and this should be dissociated from the off-policy learning.
This fairness assumption is very a strong assumption that does not seem to be satisfied by any algorithm that I can think of. Indeed, consider D being the data generated by following algorithm alpha, and let D’ be the data generated by following algorithm alpha’. Then it makes sense that the performance of algorithm alpha is better when trained on D rather than D’, and alpha’ is better when trained on D’ than on D. This contradicts the fairness assumption.
This being said, I believe the merit of this paper is to make explicit the actual assumptions required to be able to derive a bound on the global regret. So although the fairness assumption is unrealistic, it has the benefit of existing…
So in the end I liked the paper because the authors tried to address this difficult problem of algorithmic selection for RL the best way they could. Maybe future work will do better but at least this a first step in an interesting direction.
Now, I would have liked a comparison with other algorithms for algorithm selection, like:
- explore-then-exploit, where a fraction of T is used to try each algorithm uniformly, then the best one is selected and played for the rest of the rounds.
- Algorithms that have been designed for curriculum learning, such as some described in [Graves et al., Automated Curriculum Learning for Neural Networks, 2017], where a proxy for learning progress is used to estimate how much an algorithm can learn from data.
Other comments:
- Table 1 is really incomprehensible. Even after reading the Appendix B, I had a hard time understanding these results.
- I would suggest adding the fairness assumption in the main text and discussing it, as I believe this is crucial component for the understanding of how the global regret can be controlled.
- you may want to include references on restless bandits in Section 2.4, as this is very related to AS of RL algorithms (arms follow Markov processes).
- The reference [Best arm identification in multi-armed bandits] is missing an author.


As deep learning-based classifiers are increasingly adopted in real-world applications, the importance of understanding how a particular label is chosen grows. Single decision trees are an example of a simple, interpretable classifier, but are unsuitable for use with complex, high-dimensional data. On the other hand, the variational autoencoder (VAE) is designed to learn a factored, low-dimensional representation of data, but typically encodes high-likelihood data in an intrinsically non-separable way. We introduce the differentiable decision tree (DDT) as a modular component of deep networks and a simple, differentiable loss function that allows for end-to-end optimization of a deep network to compress high-dimensional data for classification by a single decision tree. We also explore the power of labeled data in a supervised VAE (SVAE) with a Gaussian mixture prior, which leverages label information to produce a high-quality generative model with improved bounds on log-likelihood. We combine the SVAE with the DDT to get our classifier+VAE (C+VAE), which is competitive in both classification error and log-likelihood, despite optimizing both simultaneously and using a very simple encoder/decoder architecture.

Summary
This paper proposes a hybrid model (C+VAE)---a variational autoencoder (VAE) composed with a differentiable decision tree (DDT)---and an accompanying training scheme.  Firstly, the prior is specified as a mixture distribution with one component per class (SVAE).  During training, the ELBO’s KL term uses the component that corresponds to the known label.  Secondly, the DDT’s leaves are parametrized with the encoder distribution q(z|x), and thus gradient information flows back through the DDT into the posterior approximations in order to make them more discriminative.  Lastly, the VAE and DDT are trained together by alternating optimization of each component (plus a ridge penalty on the decoder means).  Experiments are performed on MNIST, demonstrating tree classification performance, (supervised) neg. log likelihood performance, and latent space interpretability via the DDT.
Evaluation
Pros:  Giving the VAE discriminative capabilities is an interesting line of research, and this paper provides another take on tree-based VAEs, which are challenging to define given the discrete nature of the former and continuous nature of the latter.  Thus, I applaud the authors for combining the two in a way that admits efficient training.  Moreover, I like the qualitative experiment (Figure 2) in which the tree is used to vary a latent dimension to change the digit’s class.  I can see this being used for dataset augmentation or adversarial example generation, for instance.
Cons:  An indefensible flaw in the work is that the model is evaluated on only MNIST.  As there is no strong theory in the paper, this limited experimental evaluation is reason enough for rejection.  Yet, moreover, the negative log likelihood comparison (Table 2) is not an informative comparison, as it speaks only to the power of adding supervision.  Lastly, I do not think the interpretability provided by the decision tree is as great as the authors seem to claim.  Decision trees provide rich and interpretable structure only when each input feature has clear semantics.  However, in this case, the latent space is being used as input to the tree.  As the decision tree, then, is merely learning hard, class-based partitioning rules for the latent space, I do not see how the tree is representing anything especially revealing.  Taking Figure 2 as an example (which I do like the end result of), I could generate similar results with a black-box classifier by using gradients to perturb the latent ‘4’ mean into a latent ‘7’ mean (a la DeepDream).  I could then identify the influential dimension(s) by taking the largest absolute values in the gradient vector.  Maybe there is another use case in which a decision tree is superior; I’m just saying Section 4.3 doesn’t convince me to the extent that was promised earlier in the paper (and by the title).
Comment:  It's easier to make a latent variable model interpretable when the latent variables are given clear semantics in the model definition, in my opinion.  Otherwise, the semantics of the latent space become too entangled.  Could you, somehow, force the tree to encode an identifiable attribute at each node, which would then force that attribute to be encoded in a certain dimension of latent space?


MODEL COMPRESSION VIA DISTILLATION AND QUANTIZATION
Deep neural networks (DNNs) continue to make significant advances, solving tasks from image classification to translation or reinforcement learning. One aspect of the field receiving considerable attention is efficiently executing deep models in resource-constrained environments, such as mobile or embedded devices. This paper focuses on this problem, and proposes two new compression methods, which jointly leverage weight quantization and distillation of larger networks, called "teachers," into compressed "student" networks. The first method we propose is called quantized distillation and leverages distillation during the training process, by incorporating distillation loss, expressed with respect to the teacher network, into the training of a smaller student network whose weights are quantized to a limited set of levels. The second method, differentiable quantization, optimizes the location of quantization points through stochastic gradient descent, to better fit the behavior of the teacher model. We validate both methods through experiments on convolutional and recurrent architectures. We show that quantized shallow students can reach similar accuracy levels to state-of-the-art full-precision teacher models, while providing up to order of magnitude compression, and inference speedup that is almost linear in the depth reduction. In sum, our results enable DNNs for resource-constrained environments to leverage architecture and accuracy advances developed on more powerful devices.

The paper proposes to combine two approaches to compress deep neural networks - distillation and quantization. The authors proposed two methods, one largely relying on the distillation loss idea then followed by a quantization step, and another one that also learns the location of the quantization points. Somewhat surprisingly, nobody has combined the two approaches before, which makes this paper interesting. Experiments show that both methods work well in compressing large deep neural network models for applications where resources are limited, like on mobile devices.
Overall I am mostly OK with this paper but not impressed by it.  Detailed comments below.
1. Quantizing with respect to the distillation loss seems to do better than with the normal loss - this needs more discussion.
2. The idea of using the gradient with respect to the quantization points to learn them is interesting but not entirely new (see, e.g., "Matrix Recovery from Quantized and Corrupted Measurements", ICASSP 2014 and "OrdRec: An Ordinal Model for Predicting Personalized Item Rating Distributions", RecSys 2011, although in a different context). I also wonder if it would work better if you can also allow the weights to move a little bit (it seems to me from Algorithm 2 that you only update the quantization points). How about learning them altogether? Also this differentiable quantization method does not really depend on distillation, which is kind of confusing given the title.
3. I am a little bit confused by how the bits are redistributed in the second method, as in the end it seems to use more than the proposed number of bits shown in the table (as recognized in section 4.2). This makes the comparison a little bit unfair (especially for the CIFAR 100 case, where the "2 bits" differentiable quantization is actually using 3.23 bits). This needs more clarification.
4. The writing can be improved. For example, the concepts of "teacher" and "student" is not clear at all in the abstract - consider putting the first sentence of Section 3 in there instead. Also, the first sentence of the paper reads as "... have showed tremendous performance", which is not proper English. At the top of page 3 I found "we will restrict our attention to uniform and non-uniform quantization". What are you not restricting to, then?
Slightly increased my rating after reading the rebuttal and the revision.


Published as a conference paper at ICLR 2018 REGULARIZING AND OPTIMIZING LSTM LANGUAGE MODELS
In this paper, we consider the specific problem of word-level language modeling and investigate strategies for regularizing and optimizing LSTM-based models. We propose the weight-dropped LSTM, which uses DropConnect on hidden-tohidden weights, as a form of recurrent regularization. Further, we introduce NTAvSGD, a non-monotonically triggered (NT) variant of the averaged stochastic gradient method (AvSGD), wherein the averaging trigger is determined using a NT condition as opposed to being tuned by the user. Using these and other regularization strategies, our AvSGD Weight-Dropped LSTM (AWD-LSTM) achieves state-of-the-art word level perplexities on two data sets: 57.3 on Penn Treebank and 65.8 on WikiText-2. In exploring the effectiveness of a neural cache in conjunction with our proposed model, we achieve an even lower state-of-the-art perplexity of 52.8 on Penn Treebank and 52.0 on WikiText-2. We also explore the viability of the proposed regularization and optimization strategies in the context of the quasi-recurrent neural network (QRNN) and demonstrate comparable performance to the AWD-LSTM counterpart. The code for reproducing the results is open sourced and is available at https://github.com/salesforce/ awd-lstm-lm.

This is a well-written paper that proposes regularization and optimization strategies for word-based language modeling tasks.   The authors propose the use of DropConnect  on the hidden-hidden connections as a regularization method, in order to take advantage of high-speed LSTM implementations via the cuDNN LSTM libraries from NVIDIA.  The  focus of this work is on the prevention of overfitting on the recurrent connections of the LSTM.  The authors explore a variant of Average-SGD (NT-ASGD) as an optimization strategy which eliminates the need for tuning the average trigger and uses a constant learning rate.  Averaging is triggered when the validation loss worsens or stagnates for a few cycles, leading to two new hyper parameters: logging interval and non-monotone interval.  Other forms of well-know regularization methods were applied to the non-recurrent connections, input, output and embedding matrices.
As the authors point out, all the methods used in this paper have been proposed before and theoretical convergence explained. The novelty of this work lies in its successful application to the language modeling task achieving state-of-the-art results.
On the PTB task, the proposed AWD-LSTM achieves a perplexity of 57.3 vs 58.3 (Melis et al 2017) and almost the same perplexity as Melis et el. on the Wiki-Text2 task (65.8 vs 65.9).  The addition of a cache model provides significant gains on both tasks.
It would be useful, if authors had explored the behavior of the  AWD-LSTM algorithm with respect to various hyper parameters  and provided a few insights towards their choices for other large vocabulary language modeling tasks (1 million vocabulary sizes).
Similarly, the choice of the average trigger and number of cycles seem arbitrary -  it would have been good to see a graph over a range of values, showing their impact on the model's performance.
A 3-layer LSTM has been used for the experiments  - how was this choice made?  What is the impact of this algorithm if the net was a 2-layer net as is typical in most large-scale LMs?
Table 3 is interesting to see how the cache model helps with rare words  and as such has applications in key word spotting tasks. Were the hyper parameters of the cache tuned to perform better on rare words?  More details on the design of the cache model would have been useful.
You state that the gains obtained using the cache model were far less than what was obtained in Graves et al 2016 - what do you attribute this to?
Ablation analysis in Table 4 is very useful - in particular it shows how lack of regularization of the recurrent connections can lead to maximum degradation in performance.
Most of the results in this paper have been based on one choice of various model parameters. Given the emperical nature of this work, it would have made the paper even clearer if an analysis of their choices were presented.  Overall, it would be beneficial to the MLP community to see this paper accepted in the conference.


We consider the problem of training generative models with deep neural networks as generators, i.e. to map latent codes to data points. Whereas the dominant paradigm combines simple priors over codes with complex deterministic models, we argue that it might be advantageous to use more flexible code distributions. We demonstrate how these distributions can be induced directly from the data. The benefits include: more powerful generative models, better modeling of latent structure and explicit control of the degree of generalization.

The paper proposes, under the GAN setting, mapping real data points back to the latent space via the "generator reversal" procedure on a sample-by-sample basis (hence without the need of a shared recognition network) and then using this induced empirical distribution as the "ideal" prior targeting which yet another GAN network might be trained to produce a better prior for the original GAN.
I find this idea potentially interesting but am more concerned with the poorly explained motivation as well as some technical issues in how this idea is implemented, as detailed below.
1. Actually I find the entire notion of an "ideal" prior under the GAN setting a bit strange. To start with, GAN is already training the generator G to match the induced P_G(x) (from P(z)) with P_d(x), and hence by definition, under the generator G, there should be no better prior than P(z) itself (because any change of P(z) would then induce a different P_G(x) and hence only move away from the learning target).
I get it that maybe under different P(z) the difficulty of learning a good generator G can be different, and therefore one may wish to iterate between updating G (under the current P(z)) and updating P(z) (under the current G), and hopefully this process might converge to a better solution. But I feel this sounds like a new angle and not the one that is adopted by the authors in this paper.
2. I think the discussions around Eq. (1) are not well grounded. Just as you said right before presenting Eq. (1), typically the goal of learning a DGM is just to match Q_x with the true data distrubution P_x. It is **not** however to match Q(x,z) with P(x,z). And btw, don't you need to put E_z[ ... ] around the 2nd term on the r.h.s. ?
3. I find the paper mingles notions from GAN and VAE sometimes and misrepresents some of the key differences between the two.
E.g. in the beginning of the 2nd paragraph in Introduction, the authors write "Generative models like GANs, VAEs and others typically define a generative model via a deterministic generative mechanism or generator ...". While I think the use of a **deterministic** generator is probably one of the unique features of GAN, and that is certainly not the case with VAE, where typically people still need to specify an explicit probabilistic generative model.
And for this same reason, I find the multiple references of "a generative model P(x|z)" in this paper inaccurate and a bit misleading.
4. I'm not sure whether it makes good sense to apply an SVD decomposition to the \hat{z} vectors. It seems to me the variances \nu^2_i shall be directly estimated from \hat{z} as is. Otherwise, the reference "ideal" distribution would be modeling a **rotated** version of the \hat{z} samples, which imo only introduces unnecessary discrepancies.
5. I don't quite agree with the asserted "multi-modal structure" in Figure 2. Let's assume a 2d latent space, where each quadrant represents one MNIST digit (e.g. 1,2,3,4). You may observe a similar structure in this latent space yet still learn a good generator under even a standard 2d Gaussian prior. I guess my point is, a seemingly well-partitioned latent space doesn't bear an obvious correlation with a multi-modal distribution in it.
6. The generator reversal procedure needs to be carried out once for each data point separately, and also when the generator has been updated, which seems to be introducing a potentially significant bottleneck into the training process.


Improved generative adversarial network (Improved GAN) is a successful method of using generative adversarial models to solve the problem of semi-supervised learning. However, it suffers from the problem of unstable training. In this paper, we found that the instability is mostly due to the vanishing gradients on the generator. To remedy this issue, we propose a new method to use collaborative training to improve the stability of semi-supervised GAN with the combination of Wasserstein GAN. The experiments have shown that our proposed method is more stable than the original Improved GAN and achieves comparable classification accuracy on different data sets.

Summary of paper and review:
The paper presents the instability issue of training GANs for semi-supervised learning. Then, they propose to essentially utilize a wgan for semi-supervised learning.
The novelty of the paper is minor, since similar approaches have been done before. The analysis is poor, the text seems to contain mistakes, and the results don't seem to indicate any advantage or promise of the proposed algorithm.
Detailed comments:
- Unless I'm grossly mistaken the loss function (2) is clearly wrong. There is a cross-entropy term used by Salimans et al. clearly missing.
- As well, if equation (4) is referring to feature matching, the expectation should be inside the norm and not outside (this amounts to matching random specific random fake examples to specific random real examples, an imbalanced form of MMD).
- Theorem 2.1 is an almost literal rewrite of Theorem 2.4 of [1], without proper attribution. Furthermore, Theorem 2.1 is not sufficient to demonstrate existence of this issues. This is why [1] provides an extensive batch of targeted experiments to verify this assumptions. Analogous experiments are clearly missing. A detailed analysis of these assumptions and its implications are missing.
- In section 3, the authors propose a minor variation of the Improved GAN approach by using a wgan on the unsupervised part of the loss. Remarkably similar algorithms (where the two discriminators are two separate heads) to this have been done before (see for example, [2], but other approaches exist after that, see for examples papers citing [2]).
- Theorem 3.1 is a trivial consequence of Theorem 3 from WGAN.
- The experiments leave much to be desired. It is widely known that MNIST is a bad benchmark at this point, and that no signal can be established from a minor success in this dataset. Furthermore, the results in CIFAR don't seem to bring any advantage, considering the .1% difference in accuracy is 1/100 of chance in this dataset.
[1]: Arjovsky & Bottou, Towards Principled Methods for Training Generative Adversarial Networks, ICLR 2017
[2]: Mroueh & Sercu, Goel, McGan: Mean and Covariance Feature Matching GAN, ICML 2017


Published as a conference paper at ICLR 2018 ON UNIFYING DEEP GENERATIVE MODELS
Deep generative models have achieved impressive success in recent years. Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs), as powerful frameworks for deep generative model learning, have largely been considered as two distinct paradigms and received extensive independent studies respectively. This paper aims to establish formal connections between GANs and VAEs through a new formulation of them. We interpret sample generation in GANs as performing posterior inference, and show that GANs and VAEs involve minimizing KL divergences of respective posterior and inference distributions with opposite directions, extending the two learning phases of classic wake-sleep algorithm, respectively. The unified view provides a powerful tool to analyze a diverse set of existing model variants, and enables to transfer techniques across research lines in a principled way. For example, we apply the importance weighting method in VAE literatures for improved GAN learning, and enhance VAEs with an adversarial mechanism that leverages generated samples. Experiments show generality and effectiveness of the transfered techniques.

The authors develops a framework interpreting GAN algorithms as performing a form of variational inference on a generative model reconstructing an indicator variable of whether a sample is from the true of generative data distributions. Starting from the ‘non-saturated’ GAN loss the key result (lemma 1) shows that GANs minimizes the KL divergence between the generator(inference) distribution and a posterior distribution implicitly defined by the discriminator. I found the paper IWGAN and especially the AAVAE experiments quite interesting.  However the paper is also very dense and quite hard to follow at times - In general I think the paper would benefit from moving some content (like the wake-sleep part of the paper) to the appendix and concentrating more on the key results and a few more experiments as detailed in the comments / questions below.
Q1) What would happen if the KL-divergence minimizing loss proposed by Huszar (see e.g http://www.inference.vc/an-alternative-update-rule-for-generative-adversarial-networks/) was used instead of the “non-saturated” GAN loss - would the residial JSD terms in Lemma 1 cancel out then?
Q2) In Lemma 1 the negative JSD term looks a bit nasty to me e.g. in addition to KL divergence the GAN loss also maximises the JSD between the data and generative distributions. This JSD term acts in a somewhat opposite direction of the KL-divergence that we are interested in minimizing. Can the authors provide some more detailed comments / analysis on these two somewhat opposed terms - I find this quite important to include given the opposed direction of the JSD versus the KL term and that the JSD is ignored in e.g. section 4.1? secondly did the authors do any experiments on the the relative sizes of these two terms? I imagine it would be possible to perform some low-dimensional toy experiments where both terms were tractable to compute numerically?
Q3) I think the paper could benefit from some intuition / discussion of the posterior term q^r(x|y) in lemma 1 composed on the prior p_theta0(x) and discriminator q^r(y|x). The terms drops out nicely in math however i had a bit of a hard time wrapping my head around what minimizing the KL-divergence between this term and the inference distribution p(xIy). I know this is a kind of open ended question but i think it would greatly aid the reader in understanding the paper if more ‘guidance’ is provided instead of just writing “..by definition this is the posterior.’
Q4) In a similar vein to the above. It would be nice with some more discussion / definitions of the terms in Lemma 2. e.g what does “Here most of the components have exact correspondences (and the same definitions) in GANs and InfoGAN (see Table 1)” mean?
Q5) The authors state that there is ‘strong connections’ between VAEs and GANs. I agree that both (after some assumptions) both minimize a KL-divergence (table 1) however to me it is not obvious how strong this relation is. Could the authors provide some discussion / thoughts on this topic?
Overall i like this work but also feel that some aspects could be improved: My main concern is that a lot of the analysis hinges on the JSD term being insignificant, but the authors to my knowledge does but provide any prof / indications that this is actually true. Secondly I think the paper would greatly benefit from concentration on fewer topics (e.g. maybe drop the RW topic as it feels a bit like an appendix) and instead provide a more throughout discussion of the theory (lemma 1 + lemma 2) as well as some more experiments wrt JSD term.


Published as a conference paper at ICLR 2018 DON'T DECAY THE LEARNING RATE, INCREASE THE BATCH SIZE
It is common practice to decay the learning rate. Here we show one can usually obtain the same learning curve on both training and test sets by instead increasing the batch size during training. This procedure is successful for stochastic gradient descent (SGD), SGD with momentum, Nesterov momentum, and Adam. It reaches equivalent test accuracies after the same number of training epochs, but with fewer parameter updates, leading to greater parallelism and shorter training times. We can further reduce the number of parameter updates by increasing the learning rate and scaling the batch size B ∝ . Finally, one can increase the momentum coefficient m and scale B ∝ 1/(1 − m), although this tends to slightly reduce the test accuracy. Crucially, our techniques allow us to repurpose existing training schedules for large batch training with no hyper-parameter tuning. We train ResNet-50 on ImageNet to 76.1% validation accuracy in under 30 minutes.

## Review Summary
Overall, the paper's paper core claim, that increasing batch sizes at a linear
rate during training is as effective as decaying learning rates, is
interesting but doesn't seem to be too surprising given other recent work in
this space. The most useful part of the paper is the empirical evidence to
backup this claim, which I can't easily find in previous literature. I wish
the paper had explored a wider variety of dataset tasks and models to better
show how well this claim generalizes, better situated the practical benefits
of the approach (how much wallclock time is actually saved? how well can it be
integrated into a distributed workflow?), and included some comparisons with
other recent recommended ways to increase batch size over time.
## Pros / Strengths
+ effort to assess momentum / Adam / other modern methods
+ effort to compare to previous experimental setups
## Cons / Limitations
- lack of wallclock measurements in experiments
- only ~2 models / datasets examined, so difficult to assess generalization
- lack of discussion about distributed/asynchronous SGD
## Significance
Many recent previous efforts have looked at the importance of batch sizes
during training, so topic is relevant to the community. Smith and Le (2017)
present a differential equation model for the scale of gradients in SGD,
finding a linear scaling rule proportional to eps N/B, where eps = learning
rate, N = training set size, and B = batch size. Goyal et al (2017) show how
to train deep models on ImageNet effectively with large (but fixed) batch
sizes by using a linear scaling rule.
A few recent works have directly tested increasing batch sizes during
training. De et al (AISTATS 2017) have a method for gradually increasing batch
sizes, as do Friedlander and Schmidt (2012). Thus, it is already reasonable to
practitioners that the proposed linear scaling of batch sizes during training
would be effective.
While increasing batch size at the proposed linear scale is simple and seems
to be effective, a careful reader will be curious how much more could be
gained from the backtracking line search method proposed in De et al.
## Quality
Overall, only single training runs from a random initialization are used. It
would be better to take the best of many runs or to somehow show error bars,
to avoid the reader wondering whether gains are due to changes in algorithm or
to poor exploration due to bad initialization. This happens a lot in Sec. 5.2.
Some of the experimental setting seem a bit haphazard and not very systematic.
In Sec. 5.2, only two learning rate scales are tested (0.1 and 0.5). Why not
examine a more thorough range of values?
Why not report actual wallclock times? Of course having reduced number of
parameter updates is useful, but it's difficult to tell how big of a win this
could be.
What about distributed SGD or asyncronous SGD (hogwild)? Small batch sizes
sometimes make it easier for many machines to be working simultaneously. If we
scale up to batch sizes of ~ N/10, we can only get 10x speedups in
parallelization (in terms of number of parameter updates). I think there is
some subtle but important discussion needed on how this framework fits into
modern distributed systems for SGD.
## Clarity
Overall the paper reads reasonably well.
Offering a related work "feature matrix" that helps readers keep track of how
previous efforts scale learning rates or minibatch sizes for specific
experiments could be valueable. Right now, lots of this information is just
provided in text, so it's not easy to make head-to-head comparisons.
Several figure captions should be updated to clarify which model and dataset
are studied. For example, when skimming Fig. 3's caption there is no such
information.
## Paper Summary
The paper examines the influence of batch size on the behavior of stochastic
gradient descent to minimize cost functions. The central thesis is that
instead of the "conventional wisdom" to fix the batch size during training and
decay the learning rate, it is equally effective (in terms of training/test
error reached) to gradually increase batch size during training while fixing
the learning rate. These two strategies are thus "equivalent". Furthermore,
using larger batches means fewer parameter updates per epoch, so training is
potentially much faster.
Section 2 motivates the suggested linear scaling using previous SGD analysis
from Smith and Le (2017). Section 3 makes connections to previous work on
finding optimal batch sizes to close the generaization gap. Section 4 extends
analysis to include SGD methods with momentum.
In Section 5.1, experiments training a 16-4 ResNet on CIFAR-10 compare three
possible SGD schedules: * increasing batch size * decaying learning rate *
hybrid (increasing batch size and decaying learning rate) Fig. 2, 3 and 4 show
that across a range of SGD variants (+/- momentum, etc) these three schedules
have similar error vs. epoch curves. This is the core claimed contribution:
empirical evidence that these strategies are "equivalent".
In Section 5.3, experiments look at Inception-ResNet-V2 on ImageNet, showing
the proposed approach can reach comparable accuracies to previous work at even
fewer parameter updates (2500 here, vs. ∼14000 for Goyal et al 2007)


Published as a conference paper at ICLR 2018 LEVERAGING GRAMMAR AND REINFORCEMENT LEARNING FOR NEURAL PROGRAM SYNTHESIS
Program synthesis is the task of automatically generating a program consistent with a specification. Recent years have seen proposal of a number of neural approaches for program synthesis, many of which adopt a sequence generation paradigm similar to neural machine translation, in which sequence-to-sequence models are trained to maximize the likelihood of known reference programs. While achieving impressive results, this strategy has two key limitations. First, it ignores Program Aliasing: the fact that many different programs may satisfy a given specification (especially with incomplete specifications such as a few input-output examples). By maximizing the likelihood of only a single reference program, it penalizes many semantically correct programs, which can adversely affect the synthesizer performance. Second, this strategy overlooks the fact that programs have a strict syntax that can be efficiently checked. To address the first limitation, we perform reinforcement learning on top of a supervised model with an objective that explicitly maximizes the likelihood of generating semantically correct programs. For addressing the second limitation, we introduce a training procedure that directly maximizes the probability of generating syntactically correct programs that fulfill the specification. We show that our contributions lead to improved accuracy of the models, especially in cases where the training data is limited.

The authors consider the task of program synthesis in the Karel DSL. Their innovations are to use reinforcement learning to guide sequential generation of tokes towards a high reward output, incorporate syntax checking into the synthesis procedure to prune syntactically invalid programs. Finally they learn a model that predicts correctness of syntax in absence of a syntax checker.
While the results in this paper look good, I found many aspects of the exposition difficult to follow. In section 4, the authors define objectives, but do not clearly describe how these objectives are optimized, instead relying on the read to infer from context how REINFORCE and beam search are applied. I was not able to understand whether syntactic corrected is enforce by way of the reward introduced in section 4, or by way of the conditioning introduced in section 5.1. Discussion of the experimental results coould similarly be clearer. The best method very clearly depends on the taks and the amount of available data, but I found it difficult to extract an intuition for which method works best in which setting and why.
On the whole this seems like a promising paper. That said, I think the authors would need to convincingly address issues of clarity in order for this to appear.
Specific comments
- Figure 2 is too small
- Equation 8 is confusing in that it defines a Monte Carlo estimate of the expected reward, rather than an estimator of the gradient of the expected reward (which is what REINFORCE is).
- It is not clear the how beam search is carried out. In equation (10) there appear to be two problems. The first is that the index i appears twice (once in i=1..N and once in i \in 1..C), the second is that λ_r refers to an index that does not appear. More generally, beam search is normally an algorithm where at each search depth, the set of candidate paths is pruned according to some heuristic. What is the heuristic here? Is syntax checking used at each step of token generation, or something along these lines?

- What is the value of the learned syntax in section 5.2? Presumaly we need a large corpus of syntax-checked training examples to learn this model, which means that, in practice, we still need to have a syntax-checker available, do we not?


Published as a conference paper at ICLR 2018 QANET: COMBINING LOCAL CONVOLUTION WITH GLOBAL SELF-ATTENTION FOR READING COMPRE- HENSION
Current end-to-end machine reading and question answering (Q&A) models are primarily based on recurrent neural networks (RNNs) with attention. Despite their success, these models are often slow for both training and inference due to the sequential nature of RNNs. We propose a new Q&A architecture called QANet, which does not require recurrent networks: Its encoder consists exclusively of convolution and self-attention, where convolution models local interactions and self-attention models global interactions. On the SQuAD dataset, our model is 3x to 13x faster in training and 4x to 9x faster in inference, while achieving equivalent accuracy to recurrent models. The speed-up gain allows us to train the model with much more data. We hence combine our model with data generated by backtranslation from a neural machine translation model. On the SQuAD dataset, our single model, trained with augmented data, achieves 84.6 F1 score 1 on the test set, which is significantly better than the best published F1 score of 81.8.

This paper proposes two contributions: first, applying CNNs+self-attention modules instead of LSTMs, which could result in significant speedup and good RC performance; second, enhancing the RC model training with passage paraphrases generated by a neural paraphrasing model, which could improve the RC performance marginally.
Firstly, I suggest the authors rewrite the end of the introduction. The current version tends to mix everything together and makes the misleading claim. When I read the paper, I thought the speeding up mechanism could give both speed up and performance boost, and lead to the 82.2 F1. But it turns out that the above improvements are achieved with at least three different ideas: (1) the CNN+self-attention module; (2) the entire model architecture design; and (3) the data augmentation method.
Secondly, none of the above three ideas are well evaluated in terms of both speedup and RC performance, and I will comment in details as follows:
(1) The CNN+self-attention was mainly borrowing the idea from (Vaswani et al., 2017a) from NMT to RC. The novelty is limited but it is a good idea to speed up the RC models. However, as the authors hoped to claim that this module could contribute to both speedup and RC performance, it will be necessary to show the RC performance of the same model architecture, but replacing the CNNs with LSTMs. Only if the proposed architecture still gives better results, the claims in the introduction can be considered correct.
(2) I feel that the model design is the main reason for the good overall RC performance. However, in the paper there is no motivation about why the architecture was designed like this. Moreover, the whole model architecture is only evaluated on the SQuAD dataset. As a result, it is not convincing that the system design has good generalization. If in (1) it is observed that using LSTMs in the model instead of CNNs could give on par or better results, it will be necessary to test the proposed model architecture on multiple datasets, as well as conducting more ablation tests about the model architecture itself.
(3) I like the idea of data augmentation with paraphrasing. Currently, the improvement is only marginal, but there seems many other things to play with. For example, training NMT models with larger parallel corpora; training NMT models with different language pairs with English as the pivot; and better strategies to select the generated passages for data augmentation.
I am looking forward to the test performance of this work on SQuAD.


We describe an approach to understand the peculiar and counterintuitive generalization properties of deep neural networks. The approach involves going beyond worst-case theoretical capacity control frameworks that have been popular in machine learning in recent years to revisit old ideas in the statistical mechanics of neural networks. Within this approach, we present a prototypical Very Simple Deep Learning (VSDL) model, whose behavior is controlled by two control parameters, one describing an effective amount of data, or load, on the network (that decreases when noise is added to the input), and one with an effective temperature interpretation (that increases when algorithms are early stopped). Using this model, we describe how a very simple application of ideas from the statistical mechanics theory of generalization provides a strong qualitative description of recently-observed empirical results regarding the inability of deep neural networks not to overfit training data, discontinuous learning and sharp transitions in the generalization properties of learning algorithms, etc.

The authors suggest that ideas from statistical mechanics will help to understand the "peculiar and counterintuitive generalization properties of deep neural networks." The paper's key claim (from the abstract) is that their approach "provides a strong qualitative description of recently-observed empirical results regarding the inability of deep neural networks not to overfit training data, discontinuous learning and sharp transitions in the generalization properties of learning algorithms, etc." This claim is restated on p. 2, third full paragraph.
I am sympathetic to the idea that ideas from statistical mechanics are relevant to modern learning theory. However, I do not find this paper at all convincing. I find the paper incoherent: I am unable to understand the argument for the central claims. On the one hand, the paper seems to be written as a "response" to Zhang et al.'s "Understanding Deep Learning Requires Rethinking Generalization", (henceforth Z): the introduction mentions Z multiple times, and the title of this work refers to Z. On the other hand, none of the issues raised by Z are (as far as I can tell) addressed in any substantial way by this paper. In somewhat more detail, this work discusses two major observations:
1. Neural nets can easily overtrain, even to random data.
2. Popular ways to regularize may or may not help.
Z certainly observes 1 and arguably observes 2. (I'd argue against, see below, but it's at least arguable.) I do not see how this paper addresses either observation. Instead, what the statistical mechanics (SM) approach seems to do is explain (or predict) the existence of phase transitions, where we suddenly go from a regime of poor generalization to good generalization or vice versa. However, neither Z nor, as far as I can tell, any other reference given here, suggests that these phase transitions are frequently observed in modern deep learning. The most relevant bit from Z is Figure 1c, which suggests that as the noise level is increased (corresponding to alpha decreasing in this paper), the generalization error increases smoothly. This seems to be in direct contradiction to the predictions made by the theories presented here.
If the authors wish to hold to the claim that their work "can provide a qualitative explanation of recently-observed empirical properties that are not easily-understandable from within PAC/VC theory of generalization, as it is commonly-used in ML" (p. 2), it is absolutely critical that they be more specific about which specific observations from which papers they think they are explaining. As written, I simply do not see which actual observations they think they explain.
In observation 2, the authors suggest that many popular ways to implement regularization "do not substantially improve the situation". A careful reading of Z (and this was corroborated by discussion with the authors) is that Z observed that regularization with parameters commonly used in practice (or, put differently, regularization parameters that led to the highest holdout accuracy in other papers) still led to substantial overtraining on noisy data. I think it is almost certainly true (see below for more discussion) that much larger values of regularization can prevent overfitting, at the cost of underfitting. It's also worth noting that Z agrees with basically all practitioners that various regularization techniques can make an important difference to practitioners who want to minimize test error; what they don't do (at least at moderate values) is *qualitatively* destroy a network's ability to overfit to noise. It is unclear to me how this paper explains observation 2 (see below for extensive discussion).
I don't actually understand the first full paragraph on p. 2 well. It is true that we can always avoid overtraining by tuning regularization parameters to get better generalization *error* (difference beween train and test) on the test data set (but possibly worse generalization accuracy); the rest of the paper seems to take the opposite side on this. A Gaussian kernel SVM with a small enough bandwidth and small enough regularization parameter can also overfit to noise. The argument needs to be sharpened here.
I find the discussion of noise at the bottom of p. 2 confusing. The authors describe tau "having to do with noise in the learning process", but then suggest that "adding noise decreases the effective load." This is the first time noise is really talked about, and it seems like maybe noise in the data is about alpha, but noise in the "learning process" is about tau? This should be clarified.
On p. 3, the authors refer to "the two parameters used by Z and many others." I am honestly not sure what's being referred to here. I just reread Z and I don't get it.  What two parameters are used by Z?
p. 3, figure. The authors should be clear about what recent (ideally widely-discussed) experimental results look anything like this figure. I found nothing in Z et al. In Appendix A.4, there is a mention of Figure 3 of Chromanska et al. 2014; that figure also seems to be totally consistent with smooth transitions and does not (to me) present any obvious evidence of a sharp phase transition. (In any case, the main paper should not rely heavily on the appendix for its main empirical evidence.)
p. 3, figure 1a. What is essential in this figure? A single phase transition? That the error be very low on the r.h.s. of the phase transition (probably not that, judging from the related models in the
Appendix).
p. 3, figure 1b/c. What does SG stand for? As far as I can tell it's never discussed.
p. 4. "Thus, an important more general insight from our approach is that --- depending strongly on details of the model, the specific details of the learning algorithm, the detailed properties of the data and their noise etc. --- going beyond worst-case bounds can lead to a rich and complex array of manners in which generalization can depend on the control parameters of the ML process." This is well-known to all practitioners. This paper does not seem to offer any specific testable explanations or predictions of any sort. I certainly agree that the study of SM models is "interesting", but what would
make this valuable would be a more direct analogy, a direct explanation of some empirical phenomenon.
Section 2 in general. The authors discuss a couple different types of observations: (1) "strong discontinuities in generalization performance as a function of control parameters" aka phase transitions, and (2) generalization performance can depend sensitively on details of the model, details of algorithms, implicit regularization properties, detailed properties of data and noise, etc." (1) shows up in the SM literature from the 90's discussed in Appendix A. I don't think it shows up in modern practice, and I don't think it shows up in Z. (2) is absolutely relevant to modern practitioners, but I don't see what this paper has to say about it beyond "SM literature from the 90's exhibits similar phenomena." The model introduced in Section 3 abstracts all such concerns away.
Section 3. I am not super comfortable with the idea of "Claims", especially since the 3 Claims seem to be different sorts of things. I would normally think of a "Claim" as something that could be true or false, possibly with some argument for its truth.
Claim 1 introduces a model (VSDL), but I wouldn't call this a claim, since nothing is actually "claimed." The subpoints of Claim 1 are arguably claims, but they're not introduced as such. I address these
in turn:
"Adding noise decreases an effective load alpha." The paper states "N is the effective capacity of the model trained on these data", but "effective capacity" is never defined. Certainly, if we *define* alpha = m_eff / N and *define* m_eff = m - m_rand, the (sub)claim follows, but why are those definitions good?  I *think* what's going on here is hidden in the sentence "empirical results indicate that for realistic DNNs it is close to 1. Thus, the model capacity N of realistic DNNs scales with m and not m_eff.", where "it" refers to the Rademacher complexity. Well, OK, but if we agree with that, then aren't we just *assuming* the main result of Z rather than explaining it? We're basically just stating that the models can memorize the data?
I don't really understand the point the last part of the paragraph is trying to make (everything after what I quoted above).
"Early stopping increases an effective temperature tau." I find this plausible but don't understand the argument at all. To this reader, it's just "stuff from SM I don't understand." I think the typical ML reader of this paper won't necessarily be familiar with any of "the weights evolve according to a relaxation Langevin equation", "from the fluctuation-dissipation theorem", or the reference to annealing rate schedules. Consider either explaining this more or just appealing to SM and relegating this to an appendix.
After the claim, the paper mentions that the VSDL model ignores other "knobs". This is fine for a model, but I think it's totally disingenuous to then suggest that this model explains anything about other popular ways to regularize (Observation 2 in the intro, see also my comment on Section 2). In the intro, the claim is "Other regularizations sometimes help and sometimes don't and we don't understand why" (the claim is about overfitting but it's also true for improving performance in general), which is basically true. But introducing a model which completely abstracts these things away cannot possibly explain anything about the behavior.
Claim 2 is that we should consider a thermodynamic limit where model complexity grows with data (the paper says grows with the number of parameters, I assume this is a typo). I would probably call this one an "Assumption", with some arguments for the justification. I think this is one of the most interesting and important ideas in the paper, and I don't fully understand it, even after reading the appendix. I have questions. How should / could this apply to practitioners, who cannot in general hope to obtain arbitrary amounts of data? Are we assuming that any (or all) modern DNN experiments are in the asymptotic regime? Are we assuming the experiments in Z are in this regime? Is there any relevance to the fact that in an ML problem (unlike in say a spin glass, at least as far as I know) the "complexity" of the *task* is *not* increasing with the data size, so eventually one will have seen "enough" data to "saturate" the task?  I'd love to know more.
Claim 3 is more of an "Informal Theorem" that under the model of Claim 1 and the assumption of Claim 2, the phase diagrams of Figure 1 hold. The "proof" is a reference to SM papers. This should be clarified.
Yet again, I point out that I do not know any modern large-scale NN experiments that correspond to any of the pictures in Figure 1.
There's a mention of "tau = 0 or t > 0." What is the significance of tau = 0? How should an ML reader think about this?
Section 3.2 suggests that Claim 3 (the existence of the 1 and 2d phase diagrams) "explain" Observations 1 and 2 from the Appendix. I simply do not see this.
For Observation 1, that NNs can easily overtrain, the "argument" seems to boil down to "the system is in a phase where it cannot help but overtrain." This is hardly an explanation at all. How do we know what phase these experiments were in? How do we know these experiments were in the thermodynamic limit?
For Observation 2, the authors point out that in VSDL, "the only way to prevent overfitting is to decrease the number of iterations." This seems true but vacuous: the authors introduced a model where regularization doesn't correspond to any knobs, so of course to the extent that that model explains reality, the knobs don't stop overfitting. But this feels like begging the question. If we accept the VSDL model, we'd also accept that various regularizations can't improve generalization, which goes directly against basically all practice. I guess I technically have to concede that "Given the three
claims", Observation 2 follows, but Claim 1 by itself seems to be already assuming the conclusion.
Minor writing issues:
The authors mention at least four times that reproducing others' results is not easy (p. 1 observation 1, p. 4 first paragraph, p. 4 footnote 6, last sentence of the main text). While I think this statement is true, it is quite well-known, and I suggest that the authors may simply alienate readers by harping on it here.
p. 1. "may always overtrain" is unclear. I don't know what it means. Is the claim that SOTA DNNs wll always overtrain when presented with enough data? I don't think so from the rest of the paper, but I'm not sure.
I'm a little unclear what the authors mean by "generalization error" (or "generalization accuracy", which seems to only be used on p. 2). Z use "generalization error = training error - test error". Check the appendix for consistency here too.
Replace "phenomenon" with "phenomena", at least twice where appropriate.
p. 3, first paragraph. I think the reference to the Hopfield model should be relegated to a footnote. The text "two or more such parameter holds more generally" is confusing; is it two, or is it two or more? What will I understand differently if I use more than two parameters? The next paragraph, starting with "Given these two identifications, which are novel to this work," seems odd, since we've
just seen 7+ references and a claim that they have similar parameterizations, so it's unclear what's novel.
Appendix A.5. "For non-linear dynamical systems... NNs from the 80s/90s or our VSDL model or realistic DNNs today .. there is simply no reason to expect this to be true." where "this" refers to "one can always choose a value of lambda to prevent overfitting, potentially at the expense of underfitting." I don't understand, and I also think this disagrees with the first full paragraph on p. 2. Is there some thermodynamic limit argument required here? The very next bullet states that x = argmin_x f(x) + lambda g(x) can prevent overfitting with large lambda. What's different? I'm overall not clear what's being implied here. Consider a modern DNN for classification. A network with all zero weights will have some empirical loss L(0). If I minimize, for the weights of a network w, L(w) + lambda ||w||^2, I have that L(w) + lambda ||w||^2 <= L(0) (assuming I can solve the optimization), and assuming L is non-negative, lambda ||w||^2 <= L(0), or ||w||^2 <= L(0) / lambda. So for very large lambda, I can drive ||w||^2 arbitrarily close to zero. How is this importantly different from the linear case?  What am I missing?
p. 3. "inability not to overfit." Avoid the double negative.
Intro, last paragraph. Weird section order description, with ref to Section A coming before section 4.
Footnote 2. "but it can be quite limiting." More detail needed. Limiting how?
Footnotes 3 and 4. The text says there are "technical" and "non-technical" reasons, but 3 and 4 both seem technical to me.
Appendix A.2. "on a randomly chosen subset of X." Is it really subset? Are we picking subsets uniformly at random?