Neural Style Transfer  has become a popular technique for generating images of distinct artistic styles using convolutional neural networks. This recent success in image style transfer has raised the question of whether similar methods can be leveraged to alter the "style" of musical audio. In this work, we attempt long time-scale high-quality audio transfer and texture synthesis in the time-domain that captures harmonic, rhythmic, and timbral elements related to musical style, using examples that may have different lengths and musical keys. We demonstrate the ability to use randomly initialized convolutional neural networks to transfer these aspects of musical style from one piece onto another using 3 different representations of audio: the log-magnitude of the Short Time Fourier Transform (STFT), the Mel spectrogram, and the Constant-Q Transform spectrogram. We propose using these representations as a way of generating and modifying perceptually significant characteristics of musical audio content. We demonstrate each representation's shortcomings and advantages over others by carefully designing neural network structures that complement the nature of musical audio. Finally, we show that the most compelling "style" transfer examples make use of an ensemble of these representations to help capture the varying desired characteristics of audio signals.
This paper studies style transfer for musical audio, and largely proposes some additions to the framework proposed by Ulyanov and Lebedev.  The changes are designed to improve the long-term temporal structure and harmonic matching of the stylized audio.  They carry out a few experiments to demonstrate how their proposed approach improves upon the baseline model.
Overall, I don't think this paper provides sufficiently novel or justified contributions compared to the baseline approach of Ulyanov and Lebedev.  It largely studies what happens when a different spectrogram representation is used on the input, and when somewhat different network architectures are used.  These changes are interesting, but don't provide a lot of additional information which I believe would be interesting to the ICLR community.  They seem better suited for an (audio) signal processing venue, or a more informal venue.  In addition, the results are not terribly compelling.  If the proposed changes (which are heuristically, not theoretically motivated) resulted in huge improvements to sound quality, I might be convinced.  More concretely, the results are still very far away from being able to be used in a commercial application (in contrast with image style transfer, whose impressive results were immediately commercially applied).  One reason I think the results remain bad is that the audio signal is still fundamentally represented as a phase-invariant representation.  Even if you backpropagate through the time-frequency transformation, the transformation itself discards phase, and so various signals (with different phase characteristics) will appear the same after transformation.  I believe this contributes the most to the fact that the resulting audio sounds very artifact-ridden and unrealistic.  If the paper had been able to overcome this limitation, I might be more convinced, but as-is I don't think it warrants acceptance at ICLR.
Specific comments:
- The description of Ulyanov & Lebedev's algorithm in 3.1 is confusingly structured.  For example, the sentence "The L2
distance between the generated audio and the content audio’s feature maps..." is basically a concatenation of roughly 6 thoughts which should be separated into different sentences.  The location of the equations (1), (2) do not correspond to where they are introduced in the text.  In addition, I don't understand how S and C are generated.  It is written that S and C are the "log-magnitude feature maps for style and content".  But the "feature maps" X are themselves a log-magnitude time frequency representation (x) convolved with the filterbank.  So how are S and C "log-magnitude feature maps"?  Surely you aren't computing the log of the output of the filterbank?  More equations would be helpful here.  Finally, it would be helpful to provide an equation both for G and W instead of just saying that W is analogously defined.
- I don't see any reason to believe that a mel-scaled spectrogram would better capture longer time scales or rhythmic information.  Mel-scaling your spectrogram just changes the frequency axis to a mel scale, which makes it somewhat closer to our perception; it does not modify the way time is represented in any way.  In fact, in musical machine learning tasks usually swapping between CQT and mel-scaled spectrograms (with a comparable number of frequency bins) has little effect, so I don't see any compelling reason to use one for "rhythm".  You need to provide strong empirical or theoretical evidence to back up your claim that this is a principled approach.  Instead, I would expect that your change of convolutional structure (to the dilated convolutions, etc) for the "mel spectrogram" branch of your network would account more heavily for stronger modeling of longer timescales.
- You refer to "WaveNet auto-encoders" and cite van den Oord et al.  The original wavenet paper did not propose an auto-encoder structure; Engel et al. did.
- "neither representation is capable of representing spatial patterns along the frequency axis" What do you mean by this?  Mel or linear-frequency (or CQT) spectrograms exhibit very strong patterns along their frequency axis.
- The method for automatically setting the scales of the different loss terms seems interesting, but I can't find anywhere a description of how you apply each of the beta terms.  Are they analogous to the alpha and beta parameters in equation (4)?  If so, it appears that gamma is shared across each beta term; this would mean that changing the value of gamma simply changed the scale of all loss terms at once, which would have no effect on optimization.
- "This is entirely possible though the ensemble representation" typo, through -> through
- That instance normalization causes noisy audio is an interesting empirical result, but I'm interested in a principled explanation of why this would happen.
- "the property of having 0 mean and unit variance" - you use this to describe the SeLU nonlinearity.  That's not a property of the nonlinearity, it's a property of the activations of different layers when using the nonlinearity (given correct initialization).
- How are the "Inter-Onset Interval Length Distributions" computed?  How are you getting the onsets, etc?
- " the maximum cross-correlation value between the time-domain audio waveforms are not significantly affected by the length of this field" - there are many ways effective copying could happen without the time-domain cross-correlation being large.