A UNIVERSAL MUSIC TRANSLATION NETWORK
We present a method for translating music across musical instruments and styles. This method is based on unsupervised training of a multi-domain wavenet autoencoder, with a shared encoder and a domain-independent latent space that is trained end-to-end on waveforms. Employing a diverse training dataset and large net capacity, the single encoder allows us to translate also from musical domains that were not seen during training. We evaluate our method on a dataset collected from professional musicians, and achieve convincing translations. We also study the properties of the obtained translation and demonstrate translating even from a whistle, potentially enabling the creation of instrumental music by untrained humans.
The paper proposes a multi-domain music translation method. The model presents a Wavenet auto-encoder setting with a single (domain independent) encoder and multiple (domain specific) decoders.  From the model perspective, the paper builds up on several exciting ideas such as Wavenet and autoencoder based translation models that can perform the domain conversion without relying on parallel datasets. The two main modifications are the use of data augmentation, the use of multiple decoders (rather a single decoder conditioned on the output domain identity) and the use of a domain confusion loss to prevent the latent space to encode domain specific information. This last idea has been also used on prior work.
Up to my knowledge, this is the first autoencoder-based music translation method. While this problem is very similar to that of speaker conversion, modeling musical audio signal (with many instruments) is clearly more challenging. 
Summarizing, I think that the contributions in terms of methods are limited, but the results are very interesting. The paper gives an affirmative answer to the question of whether existing models could be adapted to handle the case of music translation, which is of value. The paper would be stronger in my view, if stronger baselines would be included. This would show that the technical contributions are better than alternative methods. Please read bellow some further comments and questions.
The authors perform two ablation studies: eliminating data augmentation and the domain confusion network. In both cases, the model without this add on fails to train. However, it seems to me that different studies are important. 
The paper seems to be missing baselines. The authors could compare their work with that of VQ-VAE. The authors claim that they could not make VQ-VAE work on this problem. The cited work by Dieleman et al provides some improvements to adapt VQ-VAE to be better suited to the music domain. Did you evaluate also autoregressive discrete autoencoders?
The proposed method uses an individual decoder per domain. This is unlike other conversion methods (such as the speech conversion studied in VQ-VAE). This modification is very costly and provides a very large capacity. Have you tried having a single decoder which is also conditioned on a one-hot vector indicating the domain? Is it reasonable to expect some transfer between domains or are they too different? Maybe this is the motivation behind using many decoders. It would be good to clarify. 
I understand that the emphasis of this work is on music translation, however, the model doesn't have anything specific to music. In that regard, maybe a way to compare to VQ-VAE is to run the proposed method to the voice conversion of the VQ-VAE.
Have you tried producing samples using the decoder in an unconditional setting?
The authors claim that the learned representation is disentangled. Why is this the case? Normally a representation is said to be disentangled if different properties are represented in different (disjoint) coordinates. I might not be understanding what is meant here.
The loss used by the authors, encourages the latent representation to not have domain specific information. The authors should cite the work [A], which has very similar motivation. It would be interesting to report the classification accuracy of the classifier to see how much of the domain information is left in the latent codes. Is it reduced to chance?
In Section 3.1 the authors describe some modifications to nv-wavenet. I imagine that this is because it leads to better performance or faster training. It would be good to give some more information. Did you perform ablation studies for these?
In the human lineup experiment (Figure 2 b,c and d). While the listeners fail to select the correct source, many of the domains are never chosen. This could suggest that some translations are consistently poorer than others or the translations themselves are poor. This cannot be deduced from this experiment. Have you evaluated this?  Maybe it would be better to present pairs of audios with reconstruction and a translation. 
While I consider the results quite good, I tend to agree with the posted public comment. It is very hard to claim that the model is effectively transferring styles. A perceptual test should include the question: is this piece on this given style? As the authors mentioned, it is clearly very difficult to evaluate generative models. But maybe the claims could be toned down.
[A] Louizos, Christos, et al. "The variational fair autoencoder." arXiv preprint arXiv:1511.00830 (2015).