Voice Conversion (VC) is a task of converting perceived speaker identity from a source speaker to a particular target speaker. The earlier approaches in the literature primarily find a mapping between the given source-target speaker-pairs. Developing mapping techniques for many-to-many VC using non-parallel data, including zero-shot learning, remains less explored areas in VC. Most of the many-to-many VC architectures require training data from all the target speakers for whom we want to convert the voices. In this paper, we propose a novel style transfer architecture, which can also be extended to generate voices even for target speakers whose data were not used in training (i.e., case of zero-shot learning). In particular, we propose Adaptive Generative Adversarial Network (AdaGAN), new architectural training procedure that helps in learning normalized speaker-independent latent representation, which will be used to generate speech with different speaking styles in the context of VC. We compare our results with the state-of-the-art StarGAN-VC architecture. In particular, the Ada-GAN achieves 31.73%, and 10.37% relative improvement compared to the Star-GAN in MOS tests for speech quality and speaker similarity, respectively. The key strength of the proposed architectures is that it yields these results with less computational complexity. AdaGAN is 88.6% less complex than StarGAN-VC in terms of FLoating Operation Per Second (FLOPS), and 85.46% less complex in terms of trainable parameters.
This paper presents a voice conversion approach using GANs based on adaptive instance normalization (AdaIN).  The authors give the mathematical formulation of the problem and provide the implementation of the so-called AdaGAN. Experiments are carried out on VCTK and the proposed AdaGAN is compared with StarGAN.  The idea is ok and the concept of using AdaIN for efficient voice conversion is also good.  But the paper has a lot of issues both technically and grammatically, which makes the paper hard to follow.
1. On writing
There are glaring grammar errors in numerous places. e.g.
  -- "Although, there are few GAN-based systems that produced state-of-the-art results for non-parallel VC. Among these algorithms, even fewer can be applied for many-to-many VC task. At last, there is the only system available for zero-shot VC proposed by Qian et al. (2019)."   This is hard to parse.
 -- "helps generator to make ..."  -> "helps the generator make ..."
 --  "let assume" -> "Let's assume" 
 --  "We know that the idea of transitivity as a way to regularize structured data has a long history."   what does it mean?
 --  "the generator of AdaGAN is consists of Encoder and Decoder."  -> "consist of"
 -- "After training of AdaGAN for large number of iteration of $\tau$ , where theoretically $\tau \rightarrow \infty$." where is the second half of the sentence?
2.  On math notation
 The math notation is messy and there are lots of inaccuracies.  e.g.
  --  $X_{i} \in p_{X}(\cdot|Z_{i},U_{i})$ should be $X_{i} \sim p_{X}(\cdot|Z_{i},U_{i})$
  --  "generate the distribution denoted by $\hat{X}_{Z_{1}\rightarrow Z_{2}}$"  -> why  $\hat{X}_{Z_{1}\rightarrow Z_{2}}$ becomes a distribution? 
  --  "$p_{N}(\cdot|Z_{1},U_{1})$, $p_{N}(\cdot|Z_{2},U_{1})$" in Eq.14,  $N$ should be replaced by the random variable.
  --  $S'_{X}$ and $S'_{Y}$ should be $S_{X'}$ and $S_{Y'}$ in line 15 in the algorithm
3. On technical details:
 -- In Fig.1 (b), why is there only one input to the discriminator?  How do you inject the adversarial samples and how do you generate adversarial samples? 
-- In section 4.4, "in encoder and decoder all layers are Linear layers".  Are you referring to fully-connected layers? Linear layers are usually referred to those with linear activation functions.  
-- The experiments are claimed to be zero-shot, but 3-5s of speech is required.  can you explain? 
Although the samples sound OK, given its current form, the paper needs significant re-work. 
P.S. rebuttal read.   I will stay with my score.