ON COMPUTATION AND GENERALIZATION OF GENER- ATIVE ADVERSARIAL NETWORKS UNDER SPECTRUM CONTROL
Generative Adversarial Networks (GANs), though powerful, is hard to train. Several recent works (Brock et al., 2016; suggest that controlling the spectra of weight matrices in the discriminator can significantly improve the training of GANs. Motivated by their discovery, we propose a new framework for training GANs, which allows more flexible spectrum control (e.g., making the weight matrices of the discriminator have slow singular value decays). Specifically, we propose a new reparameterization approach for the weight matrices of the discriminator in GANs, which allows us to directly manipulate the spectra of the weight matrices through various regularizers and constraints, without intensively computing singular value decompositions. Theoretically, we further show that the spectrum control improves the generalization ability of GANs. Our experiments on CIFAR-10, STL-10, and ImgaeNet datasets confirm that compared to other methods, our proposed method is capable of generating images with competitive quality by utilizing spectral normalization and encouraging the slow singular value decay.

This paper proposes to parameterize the weight matrices of neural nets using the SVD, with approximate orthogonality enforced on the singular vectors using Orthogonal Regularization (as opposed to e.g. the Cayley transform or optimizing on the Stiefel manifold), allowing for direct, efficient control over the spectra. The method is applied to GAN discriminators to stabilize training as a natural extension of Spectral Normalization. This method incurs a slight memory and compute cost and achieves a minor performance improvement over Spectral Normalization on two benchmark image generation tasks.
I'm a bit back and forth on this paper. On the one hand, I think the ideas this paper proposes are very interesting and could provide a strong basis off which future work can be built--the extension of spectral normalization to further study and manipulation of the spectra is natural and very promising. However, the results obtained are not particularly strong, and as they stand do not, in my opinion, justify the increased compute and memory cost of the proposed methods. The paper's presentation also wavers between being strong (there were some sections I read and immediately understood) and impenetrable (there were other sections which I had to read 5-10 times just to try and grip what was going on).
Ultimately, my vote is for acceptance. I think that we should not throw out a work with interesting and potentially useful ideas just because it does not set a new SOTA, especially when the current trend with GANs seems to suggest that top performance comes at a compute cost that all but a few groups do not have access to. With another editing pass to improve language and presentation this would be a strong, relevant paper worthy of the attention of the ICLR community.
My notes:
-The key idea of parameterizing matrices as the SVD by construction, but using a regularizer to properly constrain U and V (instead of the expensive Cayley transform, or trying to pin the matrices to the Stifel manifold) is very intriguing, and I think there is a lot of potential here.
-This paper suffers from a high degree of mathiness, substituting dense notation in places where verbal explanation would be more appropriate. There are several spots where explaining the intuition behind a given idea (particularly when proposing the various spectrum regularizers) would be far more effective than the huge amount of notation. In the author's defense, the notation is generally used as effectively as it could be. My issue is that it often is just insufficient, and communication would be better served with more illustrative figures and/or language.
-I found the way the paper references Figure 1 confusing. The decays are substantially different for each layer--are these *all* supposed to be examples of slow decay? Layer 6 appears to have 90% of its singular values below 0.5, while layer 0 has more than 50%. If this is slow decay, what does an undesirable fast decay look like? Isn't the fast decay as shown in figure 2 almost exactly what we see for Layer 6 in figure 1? What is the significance of the sharp drop that occurs after some set number of singular values? The figure itself is easy to understand, but the way the authors repeatedly refer to it as an example of smooth singular decays is confusing.
-what is D-optimal design? This is not something commonly known in the ML literature. The authors should explain what exactly that D-optimal regularizer does, and elucidate its backward dynamics (in an appendix if space does not permit it in the main body). Does it encourage all singular values to have similar values? Does it push them all towards 1? I found the brief explanation ("encourages a slow singular value decay") to be too brief--consider adding  a plot of the D-optimal spectrum to Figure 1, so that the reader can easily see how it would compare to the observed spectra. Ideally, the authors would show an example of the target spectra for each of the proposed regularizers in Figure 1. This might also help elucidate what the authors consider a desirable singular value decay, and mollify some of the issues I take with the way the paper references figure 1.
-The explanation of the Divergence Regularizer is similarly confusing and suffers from mathiness, a fact which I believe is further exacerbated by its somewhat odd motivation. Why, if the end result is a reference curve toward which the spectra will be regularized, do the authors propose (1) a random variable which is a transformation of a gaussian (2) to take the PDF of that random variable (3) discretize the PDF  (4) take the KL between a uniform discrete distribution and the discretized PMF and (5) ignore the normalization term? If the authors were actually working with random variables and proposing a divergence this might make sense, but the items under consideration are singular values which are non-stochastic parameters of a model, so treating them this way seems very odd. Based on figure 2 it looks like the resulting reference curves are fine, but the explanation of how to arrive there is quite convoluted--I would honestly have been more satisfied if the authors had simply designed a function (a polynomial logarithmic function perhaps) with a hyperparameter or two to control the curvature.
-"Our experimental results show that both combinations achieve an impressive results on CIFAR10 and STL-10 datasets"
Please do not use subjective adjectives like "impressive." A 6.5% improvement is okay, but not very impressive, and when you use subjective language you run the risk of readers and reviewers subjectively disagreeing with you, as is the case with this reviewer. Please also fix the typo in this sentence, it should at least be "...achieve [impressive] results" or "achieve an [impressive] improvement on..."
Section 3:
-What is generalization supposed to mean in this context? It's unclear to me why this is at all relevant--is this supposed to be indicating the bounds for which the Discriminator will correctly distinguish real vs generated images? Or is there some other definition of generalization which is relevant? Does it actually matter for what we care about (training implicit generative models)?
-What exactly is the use of this generalization bound? What does it tell us? What are the actual situations in which it holds? Is it possible that it will ever be relevant to training GANs or to developing new methods for training GANs?
Experiments:
-I appreciate that results are taken over 10 different random seeds.
-If the choice of gamma is unimportant then why is it different for one experiment? I found footnote 4 confusing and contradictory.
-For figure 3, I do not think that the margin is "significant"--it constitutes a relative 6.5% improvement, which I do not believe really justifies the increased complexity and compute cost of the method.
-I appreciate Table 1 and Figure 4 for elucidating (a) how orthogonal the U and V matrices end up and (b) the observed decay of the spectra.
Appendix:
-Please change table 7 to be more readable, with captions underneath each figure rather than listed at the top and forcing readers to count the rows and match them to the caption. What is the difference between SN-GAN and Spectral Norm in this table? Or is that a typo, and it should be spectral-constraint?
-I Would like to see a discussion of table 7 / interpretation of why the spectra look that way (and why they evolve that way over training) for each regularizer.
Minor:
-Typos and grammatical mistakes throughout.
-As per the CIFAR-10/100 website (https://www.cs.toronto.edu/~kriz/cifar.html) the Torralba citation is not the proper one for the CIFAR datasets, despite several recent papers which have used it.
-Intro, last paragraph, "Generation bound" should be generalization bound?
-Page 4, paragraph 2, last sentence, problem is misspelled.


Past works have shown that, somewhat surprisingly, over-parametrization can help generalization in neural networks. Towards explaining this phenomenon, we adopt a margin-based perspective. We establish: 1) for multi-layer feedforward relu networks, the global minimizer of a weakly-regularized cross-entropy loss has the maximum normalized margin among all networks, 2) as a result, increasing the over-parametrization improves the normalized margin and generalization error bounds for deep networks. In the case of two-layer networks, an infinite-width neural network enjoys the best generalization guarantees. The typical infinite feature methods are kernel methods; we compare the neural net margin with that of kernel methods and construct natural instances where kernel methods have much weaker generalization guarantees. We validate this gap between the two approaches empirically. Finally, this infinite-neuron viewpoint is also fruitful for analyzing optimization. We show that a perturbed gradient flow on infinite-size networks finds a global optimizer in polynomial time.

The authors claim to prove three things: (1) Under logistic loss (with a vanishing regularization), the normalized margin (of the solution) converges to the max normalized margin, for positive homogenous functions. This is an asymptotic result: the amount of regularization vanishes. (2) For one hidden layer NN, the max margin under l_2 norm constraint on weights in the limit, is equivalent to the l_1 constraint (total variation) on the sign measure (specified by infinite neurons) for the one hidden layer NN. (3) Show some convergence rate for the mean-field view of one hidden layer NN, i.e., the Wasserstein gradient flow on the measure (of the neurons). The author show some positive result for a perturbed version.
The problem is certainly interesting. However, my main concerns are: (1) the novelty of the main theorems given the literature, and (2) the carefulness of stating what is known in the literature review.
In summary:
1. Theorem 2.1, Theorem 3.1, and Theorem 3.3 are anticipated, or not as critical, given the literature (detailed reasons in major comments).
2. The construction in Theorem 3.5 is nice, but, it is only able to say an upper bound of the generalization of kernel is not good (comparing upper bounds is not enough). In addition, For Theorem 4.3. [Mei Montanari and Nguyen 2018] also considers similar perturbed Wasserstein gradient flow, with many convergence results. One needs to be more careful in stating what is new.
Major comments:
1. Theorem 3.3 (and Theorem 3.2) seems to be the most interesting/innovative one.
However, I would like to argue that it might be natural in one line proof, with the following alternative view:
--
l_2 norm constraint normalized margin, one hidden layer NN, with infinite neurons
gamma^star, infty :=
\max \min_i y_i int_{neuron} w || u || ReLU( x_i \bar{u}) dS^{d-1} -- integral over normalized neurons over sphere
under the constraint
int_{neuron} (w^2 + ||u||^2) dS^{d-1} \leq 1
This is equivalent to the l_1 constraint margin (variation norm), one hidden layer NN,
gamma_l_1 :=
\max \min_i y_i int_{neuron} rho(u) ReLU( x_i \bar{u}) dS^{d-1} -- integral over normalized neurons over sphere
under the constraint
int_{neuron} |rho(u)| dS^{d-1} \leq 1/2
here rho(u) is the sign measure represented by neurons. Simply because at the optimum
w || u || = 1/2 ( w^2 + || u ||^2) := rho(u)
therefore
gamma^star, infty =  gamma_l_1
So one see the factor 1/2 exactly.
--
In addition, [Bach 18, JMLR:v18:14-546] discuss more in depth the l_1 type constraint
(TV of sign measure) rather then l_2 type constraint (RKHS) for one hidden layer NN with infinite neurons. The authors should cite this work.
It is clear that l_1(neuron) < l_2(neuron) therefore
l_2 constraint margin is always smaller than l_1 constraint margin.
2. Theorem 2.1. I think the proof is almost a standard exercise given [Rosset, Zhu, and Hastie 04].
The observation for it generalizes to positive homogenous function beyond linear is a nice addition, but not crucial enough to stand out as an innovation.
Much of the difficulty in related paper lies in achieving non asymptotic convergence rate to max margin solution, for logistic loss [Soudry, Hoffer and Srebro 18], or what happens when data is not perfectly separable [Ji and Telgarsky 18].
3. Generalization result Theorem 3.1. Maybe it is better to state as a corollary, given the known results in the literature, in my opinion. This generalization is standard result from margin-based bounds available
[Koltchinskii and Panchenko 02, Bartlett and Mendelson 02].
In addition, the authors remark that the limit for (3.3) may not exist. You can change to limsup, your footnote[4]
is essentially the limsup definition.
4. Theorem 3.5. This construction of the data distribution is the part I like. However, you should remind the reader that
having a small margin for the kernel only implies the the upper bound for generalization is bad.
Comparing the upper bound doesn't mean kernel method is performing bad for the instance.
From a logic view, it is unclear the benefit of Theorem 3.5.
I do agree one can try to see in simulation if kernel/RKHS approach (l_2) is performing worse for generalization, for one hidden layer NN. But this is separate from the theory.
5. Theorem 4.3. This result should be put in the context of the literature. Specifically
[Mei Montanari and Nguyen 2018], Eqn 11-12. The perturbed wasserstein flow the authors considered
looks very close to [Mei Montanari and Nguyen 2018], Eqn 11-12, admittedly with the logistic loss instead of the square loss.
Right now, as stated in the current paper, it is very hard for the general audience to understand the contribution. A better job in comparing the literature will help.
For the technical crowd, maybe emphasize on why the "simga" can help you achieve a positive result.
Minor Comments:
6. One additional suggestion: seems to me Section 4 is a bit away from the central topic of the current paper.
I can understand that the optimization/convergence result will help complete the whole picture. However, to contribute to the "margin theme", it would be better to state with the "small vanishing regularization", how it affects the convergence of Theorem 4.3.
Even with this, it is unclear as one don't know how to connect different part of the paper: with what choice of vanishing regularization will generate a solution with a good margin, using the Wasserstein gradient flow.


NO TRAINING REQUIRED: EXPLORING RANDOM EN- CODERS FOR SENTENCE CLASSIFICATION
We explore various methods for computing sentence representations from pretrained word embeddings without any training, i.e., using nothing but random parameterizations. Our aim is to put sentence embeddings on more solid footing by 1) looking at how much modern sentence embeddings gain over random methods-as it turns out, surprisingly little; and by 2) providing the field with more appropriate baselines going forward-which are, as it turns out, quite strong. We also make important observations about proper experimental protocol for sentence classification evaluation, together with recommendations for future research.

This paper is about exploring better baselines for sentence-vector representations through randomly initialized/untrained networks. I applaud the overall message of this paper that we need to evaluate our models more thoroughly and have better baselines. The experimentation is quite thorough and I like that you
1) explored several different architectures
2) varied the dimensionality of representations
3) examine representations with probing tasks in the Analysis section.
Main Critique
- In your takeaways you say that, “For some of the benchmark datasets, differences between random and trained encoders are so small that it would probably be best not to use those tasks anymore.” I don’t think this follows from your results. Just because current trained encoders do not perform better than random encoders on these tasks doesn’t in itself mean these tasks aren’t good evaluation tasks. These tasks could be faulty for other reasons, but just because we have no better technique than random encoders currently, doesn’t make these evaluation tasks not worthwhile. Perhaps you could further examine what features (n-gram, etc.) it takes to do well on these tasks in order to argue that they shouldn’t be used.
- In your related work section you say that “We show that a lot of information may be crammed into vectors using randomly parameterized combinations of pre-trained word embeddings: that is, most of the power in modern NLP systems is derived from having high-quality word embeddings, rather than from having better encoders.” Did you run experiments with randomly initialized embeddings? This paper (https://openreview.net/forum?id=ryeNPi0qKX) finds that representations from LSTMs with randomly initialized embeddings can perform quite well on some transfer tasks. I think in order to make such a claim about the power of high-quality word embeddings you should include numbers comparing them to randomly initialized embeddings.
Questions
- Did you find that your results were sensitive to the initialization technique used for your random LSTMs / projections?
- Do you have a sense of why random non-linear features are able to perform well on these tasks? What kind of features are the skip-thought and InferSent representations learning if they do not perform much better? It’s interesting that many of the random encoder methods outperform the trained models on word content. I think you could discuss these Analysis section findings more.
Other Critiques
- In the introduction, instead of simply describing what is commonly done to obtain and evaluate sentence embeddings, it would be better to include a sentence or two about the motivation for sentence embeddings at all.
- The first sentence, “Sentence embeddings are learned non-linear recurrent combinations of pre-trained word embeddings”, doesn’t seem to be true as BOE representations are also sentence embeddings and CNNs/transformers could also work. “Non-linear” and “recurrent” are not inherent requirements for sentence embeddings, but just techniques that researchers commonly use.
- In the second paragraph of introduction instead saying “Natural language processing does not yet have a clear grasp on the relationship between word and sentence embeddings…” it might be better to say “NLP researchers” or the “NLP community” instead of “NLP” as a field doesn’t have a clear grasp.
- In the introduction: “It is unclear how much sentence-encoding architectures improve over the raw word embeddings, and what aspect of such architectures is responsible for any improvement.” It would be also good to mention that it’s unclear how much the training task / procedure also is affects improvements.
- You could describe more about applications of reservoir computing in your related work section as it’s been used in NLP before.
- I don’t think you actually ever describe the type of data that InferSent is trained on, only that it is “expensive” annotated data. It might be useful to add a sentence about natural language inference for clarity.
- In the conclusion, change “performance improvements are less than 1 and less than 2 points on average over the 10 SentEval tasks, respectively” to  “performance improvements are less than 2 percentage points on average over the 10 SentEval tasks, respectively”
- It would be nice if you bolded/underlined the best performing numbers in your results tables.


Unsupervised learning is about capturing dependencies between variables and driven by the contrast between the probable vs improbable configurations of these variables, often either via a generative model which only samples probable ones or with an energy function (unnormalized log-density) which is low for probable ones and high for improbable ones. Here we consider learning both an energy function and an efficient approximate sampling mechanism for the corresponding distribution. Whereas the critic (or discriminator) in generative adversarial networks (GANs) learns to separate data and generator samples, introducing an entropy maximization regularizer on the generator can turn the interpretation of the critic into an energy function, which separates the training distribution from everything else, and thus can be used for tasks like anomaly or novelty detection. This paper is motivated by the older idea of sampling in latent space rather than data space because running a Monte-Carlo Markov Chain (MCMC) in latent space has been found to be easier and more efficient, and because a GAN-like generator can convert latent space samples to data space samples. For this purpose, we show how a Markov chain can be run in latent space whose samples can be mapped to data space, producing better samples. These samples are also used for the negative phase gradient required to estimate the log-likelihood gradient of the data space energy function. To maximize entropy at the output of the generator, we take advantage of recently introduced neural estimators of mutual information. We find that in addition to producing a useful scoring function for anomaly detection, the resulting approach produces sharp samples (like GANs) while covering the modes well, leading to high Inception and Frchet scores.

Thank you for an interesting read.
The paper proposes an approximate training technique for energy-based models (EBMs). More specifically, the samples used negative phase gradient in EBM training is approximated by samples from another generator. This "approximate generator" is a composition of a decoder (which, with a Gaussian prior on latent variable z, is trained to approximate the data distribution) and another EBM in latent space. The authors show connections to WGAN training, thus the name EnGAN. Experiments on natural image generation and anomaly detection show promising improvements, although not very significant.
From my understanding of the paper, the main contribution of the paper comes from section 4, which proposes a latent-space MCMC scheme to improve sample quality. I have seen several papers fusing EBMs and GAN training together and to the best of my knowledge section 4 is novel (but with problems, see below). Section 3's recipe is quite standard, e.g. as seen in Kim and Bengio (2017), and in principle contrastive divergence also uses the same idea. The idea of estimating of the entropy term for the implicit distribution p_G with adversarial mutual information estimation is something new, although quite straight-forward.
Although I do agree that MCMC mixing in x space can be much harder than MCMC mixing in z space, since I don't think the proposed latent-space MCMC scheme is exact (apart from finite-time simulation, rejection...), I don't see theoretically why the method works.
1. The MCMC method essentially samples z from another EBM, where that EBM(z) has energy function -E_{\theta}(G(z)), and then generate x = G(z). Note here EBM(z) != p(z). The key issue is, even when p_G(x) = p_{\theta}(x), there is no guarantee that the proposed latent-space MCMC method would return x samples according to distribution p_{\theta}(x). You can easily work out a counter example by considering G is an invertible transformation. Therefore I don't understand why doing MCMC on this latent-space EBM can help improve sample quality in x space.
2. Continuing point 1, with Algorithm 1 that only fits p_G(x) towards p_{\theta}(x), I am confident that the negative phase gradient is still quite biased. Why not just use the latent-space MCMC sampler composited with G as the generator, and use these MCMC samples to train both the decoder G and the mutual information estimator?
3. I am not exactly sure why the gradient norm regulariser in (3) make sense here? True that it would be helpful to correct the bias of the negative phase, but why this particular form? We are not doing WGAN here and in general we don't usually put a Lipschitz constraint on the energy function. I've seem several GAN papers arguing that gradient penalty helps in cases beyond WGAN, but most of them are just empirical observations...
Also the Omega regulariser is computed on which x? On data? Do you know whether the energy is guaranteed to be minimized at data locations? In this is that appropriate to call Omega a regulariser?
The presentation is overall clear, although I think there are a few typos and confusing equations:
1. There should be a negative sign on the LHS of equation 2.
2. Equation 3 is inconsistent with the energy update equation in Algorithm 1. The latter one makes more sense.
3. Where is the ratio between the transition kernels in the acceptance ratio equation? In general for Langevin dynamics the transition kernel is not symmetric.


Deep reinforcement learning has enabled robots to complete complex tasks in simulation. However, the resulting policies do not transfer to real robots due to model errors in the simulator. One solution is to randomize the simulation environment, so that the resulting, trained policy achieves high performance in expectation over a variety of configurations that could represent the real-world. However, the distribution over simulator configurations must be carefully selected to represent the relevant dynamic modes of the system, as otherwise it can be unlikely to sample challenging configurations frequently enough. Moreover, the ideal distribution to improve the policy changes as the policy (un)learns to solve tasks in certain configurations. In this paper, we propose to use an inexpensive, kernel-based summarization method method that identifies configurations that lead to diverse behaviors. Since failure modes for the given task are naturally diverse, the policy trains on a mixture of representative and challenging configurations, which leads to more robust policies. In experiments, we show that the proposed method achieves the same performance as domain randomization in simple cases, but performs better when domain randomization does not lead to diverse dynamic modes.

This paper presents a new method for learning diverse policies that can potentially transfer better to new environments. The proposed method aims to find simulation configurations that lead to diverse behaviors using “submodular optimizations” technique; an idea stemmed from past data summarization methods.
Pros:
-The paper deals with an important problem in RL which is learning robust policies that can transfer better
- Simple idea based on prior information theory literature is proposed.
- Good incorporation of past methods for improving robustness in RL.
Cons:
>>The paper has provided weak evidence and to support the effectiveness and significance of the proposed approach. The current analysis and experimental evaluations are not convincing.
Relevant baselines are missed and limited tasks are explored:
- A relevant prior work of Pinto et al. (2017) is not considered in the baselines for comparison.
- Only two tasks are considered in experiments which are not representative of the effectiveness of the proposed approach and does not provide convincing evidence that the proposed method performs better than prior works.
-In the HalfCheetah task only the performance of random baseline is shown and comparison with EpOpt is not reported
-Results of the base DDPG policy is reported in the experiments (Fig 3 and Table 1). For a thorough comparison these missing results are necessary.
>>Correct citation to the prior work of “domain randomization” is necessary:
- The second paragraph of related work section (first paragraph of page 2) explains that the idea of domain randomization has been first used in computer vision by Tobin et al. 2017. The idea of domain randomization was first proposed and deployed on a real robot platform in Sadeghi and Levine 2016 (Sadeghi, Fereshteh, and Sergey Levine. "CAD2RL: Real single-image flight without a single real image." arXiv preprint arXiv:1611.04201 (2016).) and was later used in Tobin et al. 2017 (this is also explained in the Tobin et al. 2017). Adding proper citation to Sadeghi and Levine 2016 is necessary.
>> Relevant prior works are missed:
- The proposed idea in the paper is relevant to multi-task RL (e.g. Teh Y, Bapst V, Czarnecki WM, Quan J, Kirkpatrick J, Hadsell R, Heess N, Pascanu R. Distral: Robust multitask reinforcement learning. InAdvances in Neural Information Processing Systems 2017 (pp. 4496-4506). Adding related discussion about prior multi-task RL methods is highly recommended for improving the paper.
- Citating several relevant prior RL methods that deal with transfer learning is missed. A few examples are:
Rusu AA, Vecerik M, Rothörl T, Heess N, Pascanu R, Hadsell R. Sim-to-real robot learning from pixels with progressive nets. arXiv preprint arXiv:1610.04286. 2016 Oct 13.
Rusu AA, Rabinowitz NC, Desjardins G, Soyer H, Kirkpatrick J, Kavukcuoglu K, Pascanu R, Hadsell R. Progressive neural networks. arXiv preprint arXiv:1606.04671. 2016 Jun 15.
Rusu AA, Colmenarejo SG, Gulcehre C, Desjardins G, Kirkpatrick J, Pascanu R, Mnih V, Kavukcuoglu K, Hadsell R. Policy distillation. arXiv preprint arXiv:1511.06295. 2015 Nov 19.
>> The format of the paper needs to be improved.
- The introduction section is rather incomplete and does not properly motivate the problem and the proposed solution.
-The organization of the experimental section should be improved. It is hard to follow the experiments and find the key experimental results in the current version of the paper.


LEARNING DEEP REPRESENTATIONS BY MUTUAL IN- FORMATION ESTIMATION AND MAXIMIZATION
This work investigates unsupervised learning of representations by maximizing mutual information between an input and the output of a deep neural network encoder. Importantly, we show that structure matters: incorporating knowledge about locality in the input into the objective can significantly improve a representation's suitability for downstream tasks. We further control characteristics of the representation by matching to a prior distribution adversarially. Our method, which we call Deep InfoMax (DIM), outperforms a number of popular unsupervised learning methods and compares favorably with fully-supervised learning on several classification tasks in with some standard architectures. DIM opens new avenues for unsupervised learning of representations and is an important step towards flexible formulations of representation learning objectives for specific end-goals.

Revision 2: The new comparisons with CPC are very helpful.  Most of my other comments are addressed in the response and paper revision.  I am still uncomfortable with the sentence "Our method ... compares favorably with fully-supervised learning on several classification tasks in the settings studied."  This strongly suggests to me that you are claiming to be competitive with SOTA supervised methods.  The paper does not contain supervised results for the resnet-50 architecture.  I would recommend that this sentence should either be dropped from the abstract or have the phrase "in the settings studied" replaced by "for an alexnet architecture".  If you have supervised results for resnet-50 they should be added to table 3 and the abstract could be adjusted to that.  I apologize that this is coming after the update deadline (I have been traveling).  The authors should simply consider the reaction of the community to over-claiming.  Because of the new comparisons with CPC on resnet-50 I am upping my score.  My confidence is low only because the real significance can only be judged over time.
Revision 1: This is a revision of my earlier review.  My overly-excited earlier rating was based on tables 1 and 2 and the claim to have unsupervised features that are competitive with fully-supervised features. (I also am subject to an a-priori bias in favor of mutual information methods.)  I took the authors word for their claim and submitted the review without investigating existing results on CIFAR10.  It seems that tables 1 and 2 are presenting extremely weak fully supervised baselines.  If DIM(L) can indeed produce features that are competitive with state of the art fully supervised features, the result is extremely important.  But this claim seems misrepresented in the paper.
Original review:
There is a lot of material in this paper and I respect this groups
high research-to-publication ratio. However, it might be nice to have
the paper more focused on the subset of ideas that seem to matter.
My biggest comment is that the top level spin seems wrong.
Specifically, the paper focuses on the two bullets on page 3 ---
mutual information and statistical constraints.  Here mutual
information is interpreted as the information between the input and
output of a feature encoder.  Clearly this has a trivial solution
where the input equals the output so the second bullet --- statistical
constraints --- are required.  But the empirical content of the paper
strongly undermines these top level bullets.  Setting the training
objective to be the a balance of MI between input and output under a
statistical consrtraint leads to DIM(G) which, according the results in
the paper, is an empirical disaster.  DIM(L) is the main result and
something else seems to be going on there (more later).  Furthermore,
the empirical results suggest that the second bullet --- statistical
constraints --- is of very little value for DIM(L). The key ablation
study here seems to be missing from the paper.  Appendix A.4 states
that "a small amount of the [statistical constraint] helps improve
classification results when used with the [local information
objective].  No quantitative ablation number is given.  Other measures
of the statistical constraint seem to simply measure to what extent
the constraint has been successfully enforced.  But the results
suggest that even successfully enforcing the constraint is of little,
if any, value for the ability of the features to be effective in
prediction.  So, it seems to me, the paper to really just about the
local information objective.
The real power house of the paper --- the local information objective
--- seems related to mutual information predictive coding as
formalized in the recent paper from deep mind by van den Oord et al
and also an earlier arxiv paper by McAllester on information-theoretic
co-training.  In these other papers one assumes a signal x_1, ... x_T
and tries to extract low dimensional features F(x_t) such that F(x_1),
..., F(x_t) carries large mutual information with F(x_{t+1}).  The
local objective of this paper takes a signal x1, ..., x_k (nXn
subimages) and extracts local features F(x_1), ... F(x_k) and a global
feature Y(F(x_1), ..., F(x_k)) such that Y carries large mutual
information with each of the features F(x_i).  These seem different
but related.  The first seems more "on line" while the second seems
more "batch" but both seem to be getting at the same thing, especially
when Y is low dimensional.
Another comment about top level spin involves the Donsker-Varadhan
representation of KL divergence (equation (2) in the paper).  The
paper states that this is not used in the experiments.  This suggests
that it was tried and failed.  If so, it would be good to report this.
Another contribution of the paper seems to be that the mutual
information estimators (4) and (5) dominate (2) in practice.  This
seems important.


Learning deep neural networks with hard-threshold activation has recently become an important problem due to the proliferation of resource-constrained computing devices. In order to circumvent the inability to train with backpropagation in the present of hard-threshold activations, Friesen & Domingos (2018) introduced a discrete target propagation framework for training hard-threshold networks in a layer-by-layer fashion. Rather than using a gradient-based target heuristic, we explore the use of search methods for solving the target setting problem. Building on both traditional combinatorial optimization algorithms and gradient-based techniques, we develop a novel search algorithm Guided Random Local Search (GRLS). We demonstrate the effectiveness of our algorithm in training small networks on several datasets and evaluate our target-setting algorithm compared to simpler search methods and gradient-based techniques. Our results indicate that combinatorial optimization is a viable method for training hard-threshold networks that may have the potential to eventually surpass gradient-based methods in many settings.

Summary:
The paper presents a novel combinatorial search algorithm for the discrete target propagation framework developed in Friesen & Domingos (2018). Experiments on small datasets with small models demonstrate some potential for the proposed approach; however, scalability remains a concern.
  Pros:
-	I like the goal of the work and think that if the targeted problem were to be solved it would be an interesting contribution to the field.
-	The proposed search algorithm is reasonable and works OK.
-	The paper is mostly well written and clear.
-	The experiments are reasonably thorough.
  Cons:
-	The paper states that it is a feasibility study on search methods for learning hard-threshold networks, however, it only evaluates the feasibility of one combinatorial search method.
-	It’s not made clear whether other approaches were also investigated or what the authors learned from their exploration of this approach.
-	The actual algorithm is not very well explained, despite being the main contribution of the paper.
-	The datasets and models are small and not necessarily representative of the requirements of the field.
-	Scalability remains a serious concern with the proposed approach.
-	It’s not clear to me that the paper presents a sufficient enough contribution to warrant acceptance.
Overall, I like the direction but do not feel that the paper has contributed enough to warrant acceptance. The authors should use the experiments they’ve run and also run more experiments in order to fully analyze their method and use this analysis to improve their proposed approach.
Questions and comments:
1.	Did you try alternative local search algorithms or did you just come up with a single approach and evaluate it? What did you learn from the experiments and the development of this algorithm that will let you create a better algorithm in the next iteration?
2.	I think that it is unfair to say that “it suggests a simpler, independent justification for the performance improvements obtained by their method.” in reference to the work of Friesen & Domingos (2018), given that the straight-through estimator is not well justified to begin with and their work in fact provides a justification for it. I do agree that it is important to investigate alternative heuristics and approaches within the discrete target propagation framework, however.
3.	Sections 2 and 3 do not clearly define what L_i is and where it comes from. Since these do not normally exist in a deep network they should be clearly defined.
4.	“step 2(b)” is not well defined in section 3.1.1. I assume that this refers to lines 4-8 of Algorithm 1? The paper should explain this procedure more clearly in the text. Further, I question the locality of this method, as it seems capable of generating any possible target setting as a neighbor, with no guarantee that the generated neighbors are within any particular distance of the uniform random seed candidate. Please clarify this.
5.	I believe that a negative sign is missing in the equation for T_i in ‘Generating a seed candidate’. For example, in the case where |N| = 1, then T_i = sign(dL/dT_i) would set the targets to attain a higher loss, not lower. Further, for |N|=1, this seems to essentially reduce to the heuristic method of Friesen & Domingos (2018).
6.	In the ‘Setting the probabilities’ section:
(a) All uses of sign(h) can be rewritten as h (since h \in {-1, +1}), which would be simpler.
(b) The paper contradicts itself: it says here ‘flip entries only when sign(dL/dh) = sign(h)’ but Algorithm 1 says ‘flip entries only when sign(dL/dh) != sign(h)’. Which is it?
(c) What is the value of a_h in the pseudocode? (i.e., how is this computed in the experiments)
7.	In the experiments, the paper says that ‘[this indicates] that the higher dimensionality of the CIFAR-10 data manifold compared to MNIST may play a much larger role in inhibiting the performance of GRLS.’ How could GRLS overcome this? Also, I don’t agree with the claim made in the next sentence – there’s not enough evidence to support this claim as the extra depth of the 4-layer network may also be the contributing factor.
8.	In Table 2, why are some numbers missing? The paper should explain what this means in the caption and why it occurs. Same for the bolded numbers.
9.	The Loss Weighting, Gradient Guiding, Gradient Seeding, and Criterion Weighting conditions are not clearly defined but need to be to understand the ablation experiments. Please define these properly.
10.	The overall structure of the algorithm is not stated. Algorithm 1 shows how to compute the targets for one particular layer but how are the targets for all layers computed? What is the algorithm that uses Algorithm 1 to set the targets and then set the weights? Do you use a recursive approach as in Friesen & Domingos (2018)?
11.	In Figure 2, what dataset is this evaluation performed on? It should say in the caption. It looks like this is for MNIST, which is a dataset that GRLS performs well on. What does this figure look like for CIFAR-10? Does increasing the computation for the heuristic improve performance or is it also flat for a harder dataset? This might indicate that the initial targets computed are useful but that the local search is not helping. It would be helpful to better understand (via more experiments) why this is and use that information to develop a better heuristic.
12.	It would be interesting to see how GRLS performs on other combinatorial search tasks, to see if it is a useful approach beyond this particular problem.
13.	In the third paragraph of Section 4.2, it says ‘The results are presented in Table 3.’ I believe this should say Figure 3. Also, the ordering of Figure 3 and Table 3 should be swapped to align with the order they are discussed in the text. Finally, the caption for Table 3 is insufficiently explanatory, as are most other captions; please make these more informative.
14.	In Section 4.3:
(a), the paper refers to Friesen & Domingos (2018) indicating that zero loss is possible if the dataset is separable. However, what leads you to believe that these datasets are separable? A more accurate comparison would be the loss for a powerful non-binarized baseline network.
(b) Further, given the standard error of GRLS, it’s possible that its loss could be substantially higher than that of FTPROP as well. It would be interesting to investigate the cases where it does much better and the cases where it does much worse to see if these cases are informative for improving the method.
15.	Why is there no discussion of training time in the experiments? While it is not surprising that GRLS is significantly slower, it should not be ignored either. The existence of the Appendix should also be mentioned in the main paper with a brief mention of what information can be found in it.
16.	In Algorithm 1, line 2 is confusingly written. Also, notationally, it’s a bit odd to use h both as an element and as an index into T.
17.	There are a number of capitalization issues in the references.
18.	The Appendix refers to itself (“additional hyperparameter details can be found in the appendices”).


Currently, progressively larger deep neural networks are trained on ever growing data corpora. In result, distributed training schemes are becoming increasingly relevant. A major issue in distributed training is the limited communication bandwidth between contributing nodes or prohibitive communication cost in general. To mitigate this problem we propose Sparse Binary Compression (SBC), a compression framework that allows for a drastic reduction of communication cost for distributed training. SBC combines existing techniques of communication delay and gradient sparsification with a novel binarization method and optimal weight update encoding to push compression gains to new limits. By doing so, our method also allows us to smoothly trade-off gradient sparsity and temporal sparsity to adapt to the requirements of the learning task. Our experiments show, that SBC can reduce the upstream communication on a variety of convolutional and recurrent neural network architectures by more than four orders of magnitude without significantly harming the convergence speed in terms of forward-backward passes. For instance, we can train ResNet50 on ImageNet in the same number of iterations to the baseline accuracy, using ×3531 less bits or train it to a 1% lower accuracy using ×37208 less bits. In the latter case, the total upstream communication required is cut from 125 terabytes to 3.35 gigabytes for every participating client. Our method also achieves state-of-the-art compression rates in a Federated Learning setting with 400 clients.

The paper once again looks at the problem of reducing the communication requirement for implementing the distributed optimization techniques, in particular, SGD. This problem has been looked at from multiple angles by many authors. And although there are many unanswered questions in this area, I do not see the authors providing any compelling contribution to answering those questions. A big chunk of the paper is devoted to expressing some shallow theorems, which in some cases I do not even see their importance or connection to the main point of the paper; see my comments below. In terms of the techniques for reducing the communication burden, the authors seem to just put all the other approaches together with minimal novelty.
More detailed comments:
- I do not really understand what the authors mean by noise damping. I would appreciate it if they could clarify that point. This seems to be a very important point as the model they propose for the noise in the process is basically based on this notion. It is a great failure on the authors' part that such a crucial notion in their paper is not clearly described.
- The model that is proposed for noise is too strong and too simplistic. Do you guys have any evidence to back this up?
- Theorem 2.1 is not a theorem. The result is super shallow and relatively trivial.
- In corollary 2.1 it seems that no matter what the randomness in the system is, the algorithm is going to converge to the same solution. This is not true even for the non-strongly convex objectives, let alone the non-convex problems where there are so many stationary solutions and whatnot.
- With regards to Fig 3 (and other related figures in the appendix) and the discussion on the multiplicative nature of compression: The figure does not seem to suggest multiplicative nature in all the regimes. It seems to hold in high compression/ low-frequency communication regime. But on the other side of the spectrum, it does not seem to hold very strongly.
- The residual accumulation only applies when all the nodes update in all the iterations. I do not believe this would generalize to the federated learning, where nodes do not participate in all the updates. I do not know if the authors have noted this point in their federated learning experiments.
- Theorem 3.1 is very poorly stated. Other that than it is shallow and in my opinion irrelevant. What is the argument in favor of the authors' thought that could be built based on the result of Theorem 3.1?
- One major point that is missing in the experiments (and probably in the experiments in other papers on the same topic) is to see how much do all these compressions affect the speed of learning in different scenarios in realistic scenarios? Note that in realistic scenarios many things other than communication could affect the convergence time.


Principal Filter Analysis (PFA) is an easy to implement, yet effective method for neural network compression. PFA exploits the intrinsic correlation between filter responses within network layers to recommend a smaller network footprint. We propose two compression algorithms: the first allows a user to specify the proportion of the original spectral energy that should be preserved in each layer after compression, while the second is a heuristic that leads to a parameter-free approach that automatically selects the compression used at each layer. Both algorithms are evaluated against several architectures and datasets, and we show considerable compression rates without compromising accuracy, e.g., for VGG-16 on CIFAR-10, CIFAR-100 and ImageNet, PFA achieves a compression rate of 8x, 3x, and 1.4x with an accuracy gain of 0.4%, 1.4% points, and 2.4% respectively. In our tests we also demonstrate that networks compressed with PFA achieve an accuracy that is very close to the empirical upper bound for a given compression ratio. Finally, we show how PFA is an effective tool for simultaneous compression and domain adaptation.

This paper introduces an approach to compressing a trained neural network by looking at the correlation of the filter responses in each layer. Two strategies are proposed: one based on trying to preserve the energy of the original activations and one based on looking at the KL divergence between the normalized eigenvalues of the activation covariance matrix and the uniform distribution.
Strengths:
- The KL-divergence-based method is novel and has the advantage of not requiring to define any hyper-parameter.
- The results show the good behavior of the approach.
Weaknesses:
Method:
- One thing that bothers me is the spatial max pooling of the activations of convolutional layers. This means that is two filters have high responses on different regions of the input image, they will be treated as correlated. I do not understand the intuition behind this.
- In Section 2, the authors mention that other methods have also proposed to take the activation into account for pruning, but that they aim to minimize the reconstruction error of these activations. In fact, this is also what PFA-En does; for a given dimension, PCA gives the representation that minimizes the reconstruction error. Therefore, the connection between this method and previous works is stronger than claimed by the authors.
- While it is good that the KL-divergence-based method does not rely on any hyper-parameter, the function \psi used in Eq. 3 seems quite ad hoc. As such, there has also been some manual tuning of the method.
Experiments:
- In Table 1, there seems to be a confusion regarding how the results of FGA are reported. First, in (Peng et al., 2018), the %FLOPS is reported the other way around, i.e., the higher the better, whereas here the lower the better. Similarly, in (Peng et al., 2018), a negative \Delta in accuracy means an improved performance (as stated in the caption of their Table 2, where the numbers reported here were taken). As such, the numbers reported here, and directly taken from this work, are misinterpreted.
- Furthermore, Peng et al., 2018 report much better compression results, with %FLOP compression going up to 88.58%. Why are these results not reported here? (To avoid any misunderstanding, I would like to mention that I am NOT an author of (Peng et al., 2018)).
- Many of the entries in Table 1 are empty due to the baselines not reporting results on these datasets or with the same network. This makes an actual comparison more difficult.
- Many compression methods report results on ImageNet. This would make this paper more convincing.
- While I appreciate the domain adaptation experiments, it would be nice to see a comparison with Masana et al., 2017, which also considers the problem of domain adaptation with network compression and, as mentioned in Section 2, also makes use of the activations to achieve compression.
Related work:
- It is not entirely clear to me why tensor factorization methods are considered being so different from the proposed approach. In essence, they also perform structured network pruning.
- The authors argue that performing compression after having trained the model is beneficial. This is in contrast with what was shown by Alvarez & Salzmann, NIPS 2017, where incorporating a low-rank prior during training led to higher compression rates.
- The authors list (Dai et al., 2018) as one of the methods that aim to minimize the reconstruction error of the activations. Dai et al., 2018 rely on the mutual information between the activations in different layers to perform compression. It is not entirely clear to me how this relates to reconstruction error.
Summary:
I do appreciate the idea of aiming for a hyper-parameter-free compression method. However, I feel that there are too many points to be corrected or clarified and too many missing experiments for this paper to be accepted to ICLR.
After Response:
I appreciate the authors' response, which clarified several of my concerns. I would rate this paper as borderline. My main concern now is that the current comparison with existing method still seems too incomplete, especially with ResNet architectures, to really draw conclusions. I would therefore encourage the authors to revise their paper and re-submit it to an upcoming venue.


Generative modeling of natural images has been extensively studied in recent years, yielding remarkable progress. Current state-of-the-art methods are either based on maximum likelihood estimation or adversarial training. Both methods have their own drawbacks, which are complementary in nature. The first leads to over-generalization as the maximum likelihood criterion encourages models to cover the support of the training data by heavily penalizing small masses assigned to training data. Simplifying assumptions in such models limits their capacity and makes them spill mass on unrealistic samples. The second leads to mode-dropping since adversarial training encourages high quality samples from the model, but only indirectly enforces diversity among the samples. To overcome these drawbacks we make two contributions. First, we propose a novel extension to the variational autoencoders model by using deterministic invertible transformation layers to map samples from the decoder to the image space. This induces correlations among the pixels given the latent variables, improving over commonly used factorial decoders. Second, we propose a training approach that leverages coverage and quality based criteria. Our models obtain likelihood scores competitive with state-of-the-art likelihood-based models, while achieving sample quality typical of adversarially trained networks.

The paper proposes to alleviate two issues with VAEs and GANs
Mode covering behaviour of the MLE loss used in VAEs causing blurry image samples despite high likelihood scores (Coverage driven training in the paper)
Poor likelihood scores in GAN based models despite great looking amples (Quality driven training in the paper)
Both of these issues are well known and have been previously described in the literature (e.g. MLE models: bishop 2006, GANs: arjovsky et. al. 2017 or sonderby et. al. 2017)
The main contribution of the paper are described in the Eq (8) augmenting the VAE ELBO (L_C term) with a GAN-discriminator based loss (L_Q term). Combining the fact that the L_Q loss (essentially MLE) minimises KL[data|model] and L_Q discriminator loss used minimises KL[model|data] the authors show that this in theory minimizes a lower bound on the forward and reverse KL divergence i.e. somewhere between mode-covering and mode-seeking behavior
As a secondary contribution the authors show that adding a flow based module to the generative model p(x|z) increases both the likelihood and the fidelity of the samples.
The experimental section seems sound with extensive results on CIFAR10 and some additional results on STL, LSUN, CelebA and ImageNet.

Comments
Overall i find the paper well written and easy to understand and the experiments seems sound. My main criticism of the paper is the novelty of the proposed model. Adding a VAE and a GAN has been done before (e.g. Larsen 2016 as the authors cite as well). Outside of generative models the combination of MLE and GAN based losses have been studied in e.g. Super-Resolution (Shi et. al. 2016). In both papers of these papers the GAN based losses are added for exactly the same reasons as provided in this work i.e. to increase the sample fidelity.
I’m unsure if adding a flow to the output of the VAE generative model have been done before, however in spirit the approach is quite similar in the ideas in “Variational Lossy Autoencoder” (Chen 2016) or PixelVAEs (Gulrajani 2016) where local covariance is added through conditional factorization.

Questions
Q1) One criticism of the results is that the authors claim that the model achieves likelihood values competitive with state-of-the-art where the results seems to suggest that a more sound conclusion is that the the likelihoods are better than GANs but worse than MLE models such as PixelCNN++. Similarly for the Inception/FID scores where the model is better than VAEs but most of the time slightly worse than pure GAN models. ?
Q2) I find the claim “we propose a unified training approach that leverages coverage and quality based criteria” a bit overreaching. The loss in Eq(8) is simply the sum of a VAE and a GAN based loss and as such does not provide any unification of the two ?
Q3) Related to the previous question The KL-divergence interpretation of the GAN based loss assumes a bayes optimal discriminator. In practice this is usually never achieved why it is not really known what divergence GANs minimize if any (see e.g Fedus et. al. 2017 for a non-divergence minimization view). If the KL-divergenve minimization interpretation does not hold the proposed model is essentially a VAE with an auxiliary GAN loss biasing the model towards mode-seeking training?
Q4) There haven’t been many successful GAN usages outside of of the CNN-realm which suggests that a successful GAN is tightly connected to the inductive bias provided by CNNs for images. Have the authors tried there model on something else than images? (I suspect that the GAN based L_Q loss will be hard to apply outside of the realm of images)

Overall i find the paper well written and easy to understand and the experiments seems sound.  However I think that closely related variants of the main contributions in the paper have been tried elsewhere which somewhat reduces the novelty of the proposed model. Given my comments above I think the paper is marginally below acceptance however I could be be convinced otherwise by e.g. a solid use-case for models like this?
Lastly, I'm sorry for the late review (I was called upon late to review the paper) - I hope that you will find the time for a proper rebuttal to my questions.


ENABLING FACTORIZED PIANO MUSIC MODELING AND GENERATION WITH THE MAESTRO DATASET
Generating musical audio directly with neural networks is notoriously difficult because it requires coherently modeling structure at many different timescales. Fortunately, most music is also highly structured and can be represented as discrete note events played on musical instruments. Herein, we show that by using notes as an intermediate representation, we can train a suite of models capable of transcribing, composing, and synthesizing audio waveforms with coherent musical structure on timescales spanning six orders of magnitude (∼0.1 ms to ∼100 s), a process we call Wave2Midi2Wave. This large advance in the state of the art is enabled by our release of the new MAESTRO (MIDI and Audio Edited for Synchronous TRacks and Organization) dataset, composed of over 172 hours of virtuosic piano performances captured with fine alignment (≈3 ms) between note labels and audio waveforms. The networks and the dataset together present a promising approach toward creating new expressive and interpretable neural models of music.

This paper combines state of the art models for piano transcription, symbolic music synthesis, and waveform generation all using a shared piano-roll representation.  It also introduces a new dataset of 172 hours of aligned MIDI and audio from real performances recorded on Yamaha Disklavier pianos in the context of the piano-e-competition.
By using this shared representation and this dataset, it is able to expand the amount of time that it can coherently model music from a few seconds to a minute, necessary for truly modeling entire musical pieces.
Training an existing state of the art transcription model on this data improves performance on a standard benchmark by several percentage points (depending on the specific metric used).
Listening test results show that people still prefer the real recordings a plurality of the time, but that the syntheses are selected over them a fair amount.  One thing that is clear from the audio examples is that the different systems produce output with different equalization levels, which may lead to some of the listening results.  If some sort of automatic mastering were done to the outputs this might be avoided.
While the novelty of the individual algorithms is relatively meager, their combination is very synergistic and makes a significant contribution to the field.  Piano music modeling is a long-standing problem that the current paper has made significant progress towards solving.
The paper is very well written, but there are a few minor issues:
* Eq (1) this is really the joint distribution between audio and notes, not the marginal of audio
* Table 4: What do precision, recall, and f1 score mean for notes with velocity?  How close does the system have to be to the velocity to get it right?
* Table 6: NLL presumably stands for Negative Log Likelihood, but this should be made explicity
* Figure 2: Are the error bars the standard deviation of the mean or the standard error of the mean?


It has been common to argue or imply that a regularizer can be used to alter a statistical property of a hidden layer's representation and thus improve generalization or performance of deep networks. For instance, dropout has been known to improve performance by reducing co-adaptation, and representational sparsity has been argued as a good characteristic because many data-generation processes have only a small number of factors that are independent. In this work, we analytically and empirically investigate the popular characteristics of learned representations, including correlation, sparsity, dead unit, rank, and mutual information, and disprove many of the conventional wisdom. We first show that infinitely many Identical Output Networks (IONs) can be constructed for any deep network with a linear layer, where any invertible affine transformation can be applied to alter the layer's representation characteristics. The existence of ION proves that the correlation characteristics of representation can be either low or high for a well-performing network. Extensions to ReLU layers are provided, too. Then, we consider sparsity, dead unit, and rank to show that only loose relationships exist among the three characteristics. It is shown that a higher sparsity or additional dead units do not imply a better or worse performance when the rank of representation is fixed. We also develop a rank regularizer and show that neither representation sparsity nor lower rank is helpful for improving performance even when the data-generation process has only a small number of independent factors. Mutual information I(z l ; x) and I(z l ; y) are investigated as well, and we show that regularizers can affect I(z l ; x) and thus indirectly influence the performance. Finally, we explain how a rich set of regularizers can be used as a powerful tool for performance tuning.

This paper investigates the use of techniques for improving neural network training (regularization, normalization of covariance, sparsity) in terms of their generalization properties, empirically and analytically. The claim is that most of these tools do not help improve performance, with the exception of mutual information.
Pros: It's interesting to investigate and compare these different "regularization" techniques and compare them on different tasks empirically.
Cons:
Many of the points made in the paper are not properly capturing the nuance in the "conventional wisdom", and although it's good to be reminded and the empirical results are interesting to look at, in fact these are not really new discoveries, and sometimes the conclusions are very misleading.
1) There is test loss, and there is generalization loss, and it isn't exactly the same thing. For a hypothesis class H, we have
test loss <= train loss + generalization loss
where train loss measures how well we've fit a particular sample, and generalization loss measures how well a model that is trained on one sample can fit a new sample. Note that if I apply L2 loss to my model and have a regularization parameter --> infty, my train loss is huge but my generalization loss is 0. In other words, for a large enough regularization parameter, most of the methods experimented here WILL limit effective capacity and minimize generalization loss; it just will not give you the best test error performance. So the distinction here should be made much clearer--conventional wisdom for regularization limiting generalization error is not wrong.
2) The point that is trying to be made in section 3.1 is somewhat well-known in the general optimization literature. Let's consider a much simpler example: linear regression
min_x ||Ax-b||_2^2 + gamma ||x||_2^2
Let's consider first no regularization, gamma = 0. Assume that A has a huge nullspace. Then technically there are an infinite number of globally optimal solutions x, although if we solve this problem using SGD starting with x = 0, it is known that the minimum norm solution is always picked. You can also think of this as whitening, since large lambda smooths the spectrum of the Hessian. Now add in regularization. Now the solution is unique, even if A is ill-conditioned. It's true that it isn't super necessary to add this regularization, since SGD can get you a good solution, but now we can GUARANTEE that the generalization error is 0. In practice, also, regularization adds stability to the numerics.
In deep learning, the hidden layer z also acts as a coefficient matrix for determining y. I assume that is why people pursue low correlation, since it affects the conditioning of z.
3) Comments like  "for scaling and permutation, their influences are rather insignificant" seem a bit careless to me. In fact it is well known that scaling can affect training performance significantly. But of course, if I know the global solution for z which feeds into a softmax, then any scaling on z does not affect the output of the softmax. That, however, does not mean I don't care about the scaling of z when training.
4) Sparsity and rank BOTH limit the degrees of freedom. In fact, sparsity makes more sense when optimizing a nonlinear objective, which is always the case in deep learning. The reason to limit rank is when you wish to "learn your codewords", e.g. the eigenvectors, whereas in sparsity, the "codewords" are already learned, and you just learn the weighting. But if the codewords span the space, they both have equal representability.
4) It is not clear to me what the task is in 4.3. What is the "accuracy" in a data generation task? Is this the normal classification task? If so, is the accuracy reported train or test accuracy? How exactly is generalization error being measured?
5)I am not clear as to what conclusion is being drawn in section 5, with the last sentence "obviously, many of the statistical characteristics become meaningless for such a scalar representation, and it is high time to reconsider the so-called conventional wisdom on representation characteristics." why is this conclusion drawn based on the observation that, if a scalar z perfectly correlates with y, in fact this is the most generalization neural network?
6) Table 4: how did you choose your hyperparameters? (regularization performance is extremely sensitive to parameter choice.)
7) A major concern is that basically very little training is done in these comparisons, except in the very last section. As I previously mentioned, many of these regularization / normalization techniques are also meant to better condition the optimization itself, and thus this advantage should not be discarded.
minor comments:
 - page 5 last sentence "characteristics" should be singular
 - page 8 first sentence "to [a] deep network's performance"


This work presents a scalable solution to continuous visual speech recognition. To achieve this, we constructed the largest existing visual speech recognition dataset, consisting of pairs of text and video clips of faces speaking (3,886 hours of video). In tandem, we designed and trained an integrated lipreading system, consisting of a video processing pipeline that maps raw video to stable videos of lips and sequences of phonemes, a scalable deep neural network that maps the lip videos to sequences of phoneme distributions, and a production-level speech decoder that outputs sequences of words. The proposed system achieves a word error rate (WER) of 40.9% as measured on a held-out set. In comparison, professional lipreaders achieve either 86.4% or 92.9% WER on the same dataset when having access to additional types of contextual information. Our approach significantly improves on previous lipreading approaches, including variants of LipNet and of Watch, Attend, and Spell (WAS), which are only capable of 89.8% and 76.8% WER respectively.

The paper presents a large-scale lipreading system - no surprises there. This is good work and probably the strongest general purpose lip-reading system out there at this time, but i don't see both the work and the paper as a good fit for ICLR.
The authors take a large corpus of YouTube videos (on which Google has already trained direct acoustics-to-word speech recognizers, and which is manually transcribed), filter it, and extract regions that can be used for lipreading. They then describe a scalable preprocessing, and train a phone-based acoustic model using CTC. They seem to be using the (Miao et al., 2015) and Google WFST based decoding framework, and achieve a word error rate of ca 40%. That is impressive, but I don't see any novelty here, and the paper is full of contradictions, and leaves some important open questions:
- the authors argue for "phonemes and ctc", and no speech person would disagree with them; in fact (Miao et al., 2015) and many other papers show that the WERs with a good phoneme based dictionary in English are lower than with a character based model. it's just easier if one does not need a dictionary.
- why are the authors not using a viseme dictionary, or map their phoneme dictionary to a viseme dictionary. In visual space, their own "homonym" argument applies, too, and "mop" (or "mom") and "pop" should be mapped to the same "viseme" sequence - and the resulting uncertainty should be handled by the decoder, and not the classifier.
- how did the authors generate the one million word phoneme vocabulary? even google used around 100,000k words in their whole-word experiments, if i remember correctly? what happens if the authros reduce the vocabulary? could you provide some error analysis or at least deletions/ insertions/ substitutiosn, and compare them against an audio system?
- LipNet and the proposed architecture seem to be very similar - maybe you could provide some insight into which changes made the biggest difference?
- is the data going to be available?
- what is a "production-level speech decoder"? how come your model "is the first to combine a deep learning-based phoneme recognition model with production-grade word-level decoding techniques" if Google does essentially the same ("in production")?
- in Section 1, you say that "by design, the trained model only performs well when videos are shot at specific angles when a subject is facing the camera, [...] It does not perform well in other contexts". in Section 5, you demonstrate the "generalization power of our V2P approach"and find that it "is able to generalize well" - please clarify
- "speech impaired patients" often have non-canonical articulation, the proposed system may not work well for them
- it would be interesting to also know the absolute levels of insertions/ deletions/ substitutions for words and/ or phonemes, and for the audio only and visual systems, to be able to diagnose what the problems are.
- finally, Figure 10 is really hard to view - i'd be happy to be shown fewer faces, the main message is that the quality of the face detection is really good?


MODE NORMALIZATION
Normalization methods are a central building block in the deep learning toolbox. They accelerate and stabilize training, while decreasing the dependence on manually tuned learning rate schedules. When learning from multi-modal distributions, the effectiveness of batch normalization (BN), arguably the most prominent normalization method, is reduced. As a remedy, we propose a more flexible approach: by extending the normalization to more than a single mean and variance, we detect modes of data on-the-fly, jointly normalizing samples that share common features. We demonstrate that our method outperforms BN and other widely used normalization techniques in several experiments, including single and multi-task datasets.

Summary:
Batch Normalization (BN) suffers from 2 flaws: 1) It performs poorly when the batch size is small and 2) computing only one mean and one variance per feature might be a poor approximation for multi-modal features. To alleviate 2), this paper introduces Mode Normalization (MN) a new normalization technique based on BN. It uses a gating mechanism, similar to an attention mechanism, to project the examples in the mini-batch onto K different modes and then perform normalization on each of these modes.
Clarity:
The paper is clearly written, and the proposed normalization is well explained.
Novelty:
The proposed normalization is somewhat novel. I also found a similar paper on arXiv (submitted for review to IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018): M. M. Kalayeh, M. Shah, Training Faster by Separating Modes of Variation in Batch-normalized Models, arXiv 2018. I didn’t took the time to read this paper in details, but the mixture normalization they propose seems quite close to MN. Could the authors comment on this?
Pros and Cons:
+ Clearly written and motivated
+ Try to address BN’s weakness, which is an important direction in deep learning
- I found similar papier in the literature
- The proposed method aims to make BN perform better, but pushes it toward small batch settings, which is where BN performs poorly.
- Misses comparisons with other techniques (see detailed comments).
Detailed Comments:
1. Multi-modality:
It is not clear if the features are multimodal when performing classification tasks. Some histograms of a few features in the network would have help motivate the proposed normalization. However, it seems indeed to be an issue when training GANs: to make BN work when placed in the discriminator, the real and fake examples must be normalized separately, otherwise the network doesn't train properly. Moreover, when dealing with multimodal datasets (such as the one you created by aggregating different datasets), one can use the FiLM framework (V. Dumoulin et al., Feature-wise transformations, Distill 2018), and compute different means and variances for each datasets. How would the proposed method perform against such method?
2. Larger scale:
It would be nice to see how MN performs on bigger networks (such as the ResNet50, or a DenseNet), and maybe a more interesting fully-connected benchmark, such as the deep autoencoder.
3. Small batch regime:
It seems that the proposed method essentially pushes BN towards a regime of smaller mini-batch size, where it is known to performs poorly. For instance, the gain in performances on the ImageNet experiments drops quite a lot already, since the training is divided on several GPUs (and thus the effective mini-batch is already reduced quite a lot). This effect gets worse as the size of the network increases, since the effective mini-batch size gets smaller. This problem also appears when working on big segmentation tasks or videos: the mini-batch size is typically very small for those problems. So I fear that MN will scale poorly on bigger setups. I also think that this is the reason why you need to use extremely small K.
4. Validation set:
What validation sets are you using in your experiments? In section 4.1, the different dataset and their train / test splits are presented, but what about validation?
Conclusion:
Given the similarity with another paper already in the literature, I reject the paper. Also, it seems to me that the technique actually pushed BN towards a small batch regime, where it is known to perform poorly. Finally, it misses comparison with other techniques.
Revision:
After the rebuttal, I increased my rating to a 6. I feel this paper could still be improved by better motivating why multi-modality is important for single tasks (for example, by plotting histograms of activations from the network). I also think that the paper by Kalayeh & Shah should be presented in more details in the related work, and also be compared to in the experimental setup (for example on a small network), especially because the authors say they have experience with GMMs.


Adversarial examples are a pervasive phenomenon of machine learning models where seemingly imperceptible perturbations to the input lead to misclassifications for otherwise statistically accurate models. We propose a geometric framework, drawing on tools from the manifold reconstruction literature, to analyze the high-dimensional geometry of adversarial examples. In particular, we highlight the importance of codimension: for low-dimensional data manifolds embedded in high-dimensional space there are many directions off the manifold in which to construct adversarial examples. Adversarial examples are a natural consequence of learning a decision boundary that classifies the low-dimensional data manifold well, but classifies points near the manifold incorrectly. Using our geometric framework we prove (1) a tradeoff between robustness under different norms, (2) that adversarial training in balls around the data is sample inefficient, and (3) sufficient sampling conditions under which nearest neighbor classifiers and ball-based adversarial training are robust.

This paper studies the geometry of adversarial examples under the assumption that dataset encountered in practice exhibit lower dimensional structure despite being embedded in very high dimensional input spaces. Under the proposed framework, the authors analyze several interesting phenomena and give theoretical results related to the necessary number of samples needed to achieves robustness. However, the theory in this paper is not very deep.
Pros:
The logic of this paper is very clear and easy to follow. Definitions and theories are illustrated with well-designed figures.
This paper shows the tradeoff between robustness under two norm and infinity norm for the case when the manifolds of two classes of data are concentric spheres.
When data are distributed on a hypercube in a k dimensional subspace, the authors show that balls with radius \delta centered at data samples only covers a small part of the ‘\delta neighborhood’ of the manifold.
General theoretical results on robustness and minimum training set to guarantee robustness are given for nearest neighbor classifiers and other classifiers.
Cons:
Most of the theoretical results in this paper are not very general. The tradeoff between robustness in different norms are only shown for concentric spheres; the ‘X^\epsilon is a poor model of \mathcal{M}^\epsilon’ section is only shown for hypercubes in low dimensional subspaces.
Section 5 is not very convincing. As is discussed later in the paper, although $X^\delta$ only covers a small part of \mathcal{M}^\delta, robustness can be achieved by using balls centered at samples with larger radius.
Most of the analysis is based on the assumption that samples are perfectly distributed to achieve the best possible robustness result. A more interesting case is probably when samples are generated on the manifold following some probabilistic distributions.
Theorems given in Section 6 are reasonable, but not very significant. It is not very surprising that nearest neighbor classifier is more robust than ‘x^\epsilon based’ algorithms, especially when the samples are perfectly distributed.


RELGAN: RELATIONAL GENERATIVE ADVERSARIAL NETWORKS FOR TEXT GENERATION
Generative adversarial networks (GANs) have achieved great success at generating realistic images. However, the text generation still remains a challenging task for modern GAN architectures. In this work, we propose RelGAN, a new GAN architecture for text generation, consisting of three main components: a relational memory based generator for the long-distance dependency modeling, the Gumbel-Softmax relaxation for training GANs on discrete data, and multiple embedded representations in the discriminator to provide a more informative signal for the generator updates. Our experiments show that RelGAN outperforms current state-of-the-art models in terms of sample quality and diversity, and we also reveal via ablation studies that each component of RelGAN contributes critically to its performance improvements. Moreover, a key advantage of our method, that distinguishes it from other GANs, is the ability to control the trade-off between sample quality and diversity via the use of a single adjustable parameter. Finally, RelGAN is the first architecture that makes GANs with Gumbel-Softmax relaxation succeed in generating realistic text.

==========================
I have read the authors' response and other reviewers' comments. Thanks the authors for taking great effort in answering my questions. Generally, I feel satisfied with the repsonse, and prefer an acceptance recommendation.
==========================
Contributions:
The main contribution of this paper is the proposed RelGAN. First, instead of using a standard LSTM as generator, the authors propose using a relational memory based generator. Second, instead of using a single CNN as discriminator, the authors use multiple embedded representations. Third, Gumbel-softmax relaxation is also used for training GANs on discrete textual data. The authors also claim the proposed model has the ability to control the trade-off between sample quality and diversity via a single adjustable parameter.
Detailed comments:
(1) Novelty: This paper is not a breakthrough paper, mainly following previous work and propose new designs to improve the performance. However, it still contains some novelty inside, for example, the model choice of the generator and discriminator. I think the observation that the temperature control used in the Gumbel-softmax can reflect the trade-off between quality and diversity is interesting.
However, I feel the claim in the last sentence of the abstract and introduction is a little bit strong. Though this paper seems to be the first to really use Gumbel-softmax for text generation, similar techniques like using annealed softmax to approximate argmax has already been used in previous work (Zhang et al., 2017). Since this is similar to Gumbel-softmax, I think this may need additional one or two sentences to clarify this for more careful discussion.
Further, I would also recommend the authors discuss the following paper [a] to make this work more comprehensive as to the discussion of related work. [a] also uses annealed softmax approximation, and also divide the GAN approaches as RL-based and RL-free, similar in spirit as the discuss in this paper.
[a] Adversarial Text Generation via Feature-Mover's Distance, NIPS 2018.
(2) Presentation: This paper is carefully written and easy to follow. I enjoyed reading the paper.
(3) Evaluation: Experiments are generally well-executed, with ablation study also provided. However, human evaluation is lacked, which I think is essential for this line of work. I have a few questions listed below.
Questions:
(1) In section 2.4, it mentions that the generator needs pre-training. So, my question is: does the discriminator also need pre-training? If so, how the discriminator is pre-trained?
(2) In Table 1 & 2 & 3, how does your model compare with MaskGAN? If this can be provided, it would be better.
(3) Instead of using NLL_{gen}, a natural question is: what are the self-BLEU score results since it was used in previous work?
(4) The \beta_max value used in the synthetic and real datasets is quite different. For example, \beta_max = 1 or 2 in synthetic data, while \beta_max = 100 or 1000 is used in real data. What is the observation here? Can the authors provide some insights into this?
(5) I feel Figure 3 is interesting. As the authors noted, NLL_gen measures diversity, NLL_oracle measures quality. Looking at Figure 3, does this mean GAN model produces higher quality samples than MLE pretrained models, while GAN models also produces less diverse samples than MLE models? This is due to NLL_gen increases after pretraining, while NLL_oracle further decreases after pretraining. However, this conclusion also seems strange. Can the authors provide some discussion on this?
(6) Can human evaluation be performed since automatic metrics are not reliable enough?


This work describes a novel recurrent model for music composition, which accounts for the rich statistical structure of polyphonic music. There are many ways to factor the probability distribution over musical scores; we consider the merits of various approaches and propose a new factorization that decomposes a score into a collection of concurrent, coupled time series: "parts." The model we propose borrows ideas from both convolutional neural models and recurrent neural models; we argue that these ideas are natural for capturing music's pitch invariances, temporal structure, and polyphony.
We train generative models for homophonic and polyphonic composition on the KernScores dataset (Sapp, 2005), a collection of 2,300 musical scores comprised of around 2.8 million notes spanning time from the Renaissance to the early 20th century. While evaluation of generative models is known to be hard (Theis et al., 2016), we present careful quantitative results using a unit-adjusted cross entropy metric that is independent of how we factor the distribution over scores. We also present qualitative results using a blind discrimination test.

PROs
-seemingly reasonable approach to polyphonic music generation: figuring out a way to splitting the parts, share parameters appropriately, measuring entropy per time, all make sense
-the resulting outputs tend to have very short-term harmonic coherence (e.g. often a ‘standard chord’ with some resolving suspensions, etc), with individual parts often making very small stepwise motion (i.e. reasonable local voice leading)
-extensive comparison of architectural variations
-positive results from listening experiments
CONs
-musical outputs are *not* clearly better than some of the polyphonic systems described; despite the often small melodic steps, the individual lines are quite random sounding; this is perhaps a direct result of the short history
-I do not hear the rhythmic complexity that is described in the introduction
-the work by Johnson (2015) (ref. provided below) should be looked at and listened to; it too uses coupled networks, albeit in a different way but with a related motivation, and has rhythmic and polyphonic complexity and sounds quite good (better, in my opinion)
-some unclear sections (fixable, especially with an appendix; more detail below)
-despite the extensive architectural comparisons, I was not always clear about rationale behind certain choices, eg. if using recurrent nets, why not try LSTM or GRU? (more questions below)
-would like to have heard the listening tests; or at least read more about how samples were selected (again, perhaps in an appendix and additional sample files)
 quality, clarity, originality and significance of this work, including a list of its pros and cons (max 200000 characters).
Quality -- In this work, various good/reasonable choices are made. The quality of the actual output is fine. It is comparable to-- and to my ears not better than-- existing polyphonic systems such as the ones below (links to sample audio are provided here):
-Bachbot - https://soundcloud.com/bachbot (Liang et al 2017)
- tied parallel nets - http://www.hexahedria.com/2015/08/03/composing-music-with-recurrent-neural-networks/ (Johnson 2015, ref below)
-performanceRNN - https://magenta.tensorflow.org/performance-rnn - (Simon & Oore 2017)
..others as well..
Clarity -- Some of the writing is "locally" clear, but one large, poorly-organized section makes the whole thing confusing (details below). It is very helpful that the authors subsequently added a comment with a link to some sample scores; without that, it had been utterly impossible to evaluate the quality. There are a few points that could be better clarified:
	-p5”a multi-hot vector of notes N”. It sounds like N will be used to denote note-numbers, but in fact it seems like N is the total number of notes, i.e. the length of the vector, right? What value of N is used?
-p5 “a one-hot vector of durations D”. It sounds like D will be used to denote durations, but actually I think D is the length of the 1-hot vector encoding durations right? What value of D is used, and what durations do the elements of this vector represent?
-similarly, does T represent the size of the history? This should really be clarified.
	-p5 Polyphonic models.
		-Eq (2), (3), (4): Presumably the h’s are the hidden activations layers?
		-the networks here correspond to the blue circles in Fig 1, right? If so, make the relationship clear and explicit
		-Note that most variables in most equations are left undefined
		-actually defining the W’s in Eq(2-4)  would allow the authors to refer to the W’s later (e.g. in Section 5.2) when describing weight-sharing ideas. Otherwise, it’s all rather confusing. For example, the authors could write, “Thus, we can set W_p1 = W_p2 = W_p3 = W_p4” (or whatever is appropriate).
	-Generally, I found that pages 5-7 describe many ideas, and some of them are individually fairly clearly described, but it is not always clear when one idea is beginning, and one idea is ending, and which ideas can be combined or not. On my first readings, I thought that I was basically following it, until I got to Table 5, which then convinced me that I was in fact *not* quite following it. For example, I had been certain that all the networks described are recurrent (perhaps due to Fig1?), but then it turned out that many are in fact *not* recurrent, which made a lot more sense given the continual reference to the history and the length of the model’s Markov window etc. But the reader should not have had to deduce this. For example, one could write,
	“We will consider 3 types of architectures: convolutional, recurrent, .... In each architecture, we will have [...] modules, and we will try a variety of combinations of these modules. The modules/components are as follows:”. It’s a bit prosaic, but it can really help the reader.
-Appendices, presented well, could be immensely helpful in clarifying the exact architectures; obviously not all 22 architectures from Table 5 need to be shown, but at least a few of them shown explicitly would help clarify. For example, in Fig1, the purple boxes seem to represent notes (according to the caption), but do they actually represent networks? If they really do represent notes, then how can “notes” receive inputs from both the part-networks and the global network? Also, I was not entirely clear on the relationship of the architecture of the individual nets (for the parts) to that of the global integrating network. E.g. for experiment #20, the part-net is an RNN (with how many layers?? with regular or LSTM cells?) followed by a log-linear predictor (with one hidden layer of 300 units right? or are there multiple layers sometimes?), but then what is the global network? Why does the longest part-history vector appear to have length 10 based on Table 5, but according to Table 3 the best-performing history length was 20? Though, I am not sure the meaning of the “bottom/top” column was explained anywhere, so maybe I am completely misunderstanding that aspect of the table? Etc.
-Many piano scores do not easily deconstruct into clean 4-part polyphony; the example in Appendix A is an exception. It was not clear to me how piano scores were handled during training.
-Terminology: it is not entirely clear to me why one section is entitled “homophonic models”, instead of just “monophonic models”. Homophonic music usually involves a melody line that is supported by other voices, i.e. a sort of asymmetry in the part-wise structure. Here, the outputs are quite the opposite of that: the voices are independent, they generally function well together harmonically, and there is usually no sense of one voice containing a melody. If there’s some reason to call it homophonic, that would be fine, but otherwise it doesn’t really serve to clarify anything. However, the authors do say that the homophonic composition tasks are a “minor generalization of classic monophonic composition tasks”, so this suggests to me that there is something here that I am not quite understanding.
The last sentence of Section 5.3 is very confusing-- I don’t understand what lin_n is, or 1_n is, or how to read the corresponding entries of the table. The first part of the paragraph is fairly clear.
Table 4: “The first row” actually seems like it is referring to the second row. I know what the authors mean, but it is unnecessarily confusing to refer to it in this way. One might as well refer to “the zeroth row” as listing the duration of the clip :)
The experimental evaluation: I would like to hear some of the paired samples that were played for subjects. Were classical score excerpts chosen starting at random locations in the score, or at the beginning of the score? It is known that listening to a 10-second excerpt without context can sometimes not make sense. I would be curious to see the false positives versus the false negatives. Nevertheless, I certainly appreciate the authors’ warning to interpret the listening results with caution.

Originality & Significance -- So far, based both on the techniques and the output, I am not entirely convinced of the originality or significance of this particular system. The authors refer to “rhythmically simple polyphonic scores” such as Bachbot, but I cannot see what is rhythmically fundamentally more sophisticated about the scores being generated by the present system. One nice characteristic of the present system is the true and audible independence of the voices.
One of the contributions appears to be the construction of models that explicitly leverage with shared weights some of the patterns that occur in different “places” (pitch-wise and temporally) in music. This is both very reasonable, and also not an entirely novel idea; see for example the excellent work by Daniel Johnson, “Generating Polyphonic Music Using Tied Parallel Networks” (paper published 2017, first shared online, as far as I know, in 2015: links to all materials available at http://www.hexahedria.com/2015/08/03/composing-music-with-recurrent-neural-networks/  )
Another now common (and non exclusive) way to handle some of this is by augmenting the data with transposition. It seems that the authors are not doing this here. Why not? It usually helps.
Another contribution appears to be the use of a per-time measure of loss. This is reasonable, and I believe others have done this as well. I certainly appreciated the explicit justification for it, however.
Note that the idea of using a vector to indicate metric subdivision was also used in (Johnson 2015).
Playing through some of the scores, it is clear that melodies themselves are often quite unusual (check user studies), but the voices do stay closely connected harmonically, which is what gives the system a certain aural coherence. I would be interested to hear (and look at) what is generated in two-part harmony, and even what is generated-- as a sort of baseline-- with just a single part.
I encourage the authors to look at and listen to the work by Johnson:
-listening samples: http://www.hexahedria.com/2015/08/03/composing-music-with-recurrent-neural-networks/
-associated publication: http://www.hexahedria.com/files/2017generatingpolyphonic.pdf
Overall, I think that the problem of generating rhythmically and polyphonically complex music is a good one, the approaches seem to generally be reasonable, although they do not appear to be particularly novel, and the musical results are not particularly impressive. The architectural choices are not always clearly presented.