This work seeks the possibility of generating the human face from voice solely based on the audio-visual data without any human-labeled annotations. To this end, we propose a multi-modal learning framework that links the inference stage and generation stage. First, the inference networks are trained to match the speaker identity between the two different modalities. Then the trained inference networks cooperate with the generation network by giving conditional information about the voice. The proposed method exploits the recent development of GANs techniques and generates the human face directly from the speech waveform making our system fully end-to-end. We analyze the extent to which the network can naturally disentangle two latent factors that contribute to the generation of a face imageone that comes directly from a speech signal and the other that is not related to itand explore whether the network can learn to generate natural human face image distribution by modeling these factors. Experimental results show that the proposed network can not only match the relationship between the human face and speech, but can also generate the high-quality human face sample conditioned on its speech. Finally, the correlation between the generated face and the corresponding speech is quantitatively measured to analyze the relationship between the two modalities.
Summary - 
In this work the authors propose a two-stage procedure for training a GAN which generates plausible faces conditioned on the raw waveform of a speech signal. In the first stage two embedding functions are trained, one taking as input a frame from a video (of a person speaking), the other taking as input the raw waveform of the same video's audio. These embeddings are trained to maximize the inner product of positively sampled embeddings (frame and audio from same video), and minimize the inner product of negatively sampled embeddings (frame and audio from different videos). In the second stage a GAN is trained where the input to the generator is the a random latent code (z), concatenated with the learned embedding of a speech signal from stage 1 (c). They also initialize the first half (\phi) of the discriminator using the face embedding network from stage 1, and propose a modification of the relativistic GAN loss to prevent \phi from losing the ability to produce face embeddings that have low inner product with the wrong speech embedding.
The authors explore the properties of their learned pipeline in a number of experiments. In 4.1 they compare their self-supervised speaker matching pipeline with prior work, showing that it has competitive performance with prior work that relies on either supervised pretraining, or additional labels such as age and gender. In particular they show that as the number of negative samples increases from 1 to 9, their identity matching performance significantly improves over prior work (in the K=2 regime their method underperforms prior work). Qualitatively they show that their generator has some reasonable success (it is hard to judge what perfect success would look like, and how far they are from reaching it) at disentangling aspects of facial appearence that can and cannot be inferred from the speech signal (QLA1). They also show qualitatively that the output of their generator can be smoothly controlled by interpolating between speech conditioning vectors, producing reasonable faces at each intermediary step (QLA2). In experiment QTA1 they quantitatively validate the conditioning vector c is affecting the generated image. In experiment QTA2 the authors (with a small issue, see weaknesses) show that their proposed modification to the loss function causes their outputs to be better matched to the speech conditioning as measured by the fixed embedding network trained in stage 1. Finally in QT3 they measure how well their generated faces can be used with the original face embedding network to perform image retrieval (I am a bit unclear on the details of this experiment, see questions)
Strengths -
* The authors' proposed pipeline and modified loss function offers a generalized framework for jumpstarting conditional GAN training
* Their pipeline does not assume human-specified labels, but instead access to paired data from different modalities, which is can easily be obtained for many conditional image generation tasks (inpainting, super-resolution, colorization, etc.)
* Their pipeline also doesn't seem particularly finetuned for the speech-driven image synthesis task, so it seems reasonable to believe it could be adapted to other tasks
* Their results are qualitatively compelling, and they make a convincing efforts in experiments QTA1-3 to quantitatively show that the conditioning information is affecting the output
* The paper is generally well-written and easy to follow
Weaknesses - 
* Without completing the loop, and showing that the second stage of this pipeline helps with identity matching for speakers, I'm not clear on what the motivation for this particular form of conditional image generation is (this is unfortunate, because their framework is quite general, and it seems feasible that the authors could have applied it directly to a task where the output of conditional image generation is directly useful)
* I interpret the main purpose of experiment QTA 2 as validating the effectiveness of their proposed loss modification, it seems like a natural experiment to make this claim more convincing is to compare a generator trained with Eq 4. against real images (hopefully getting a much lower matching probability than the 76.65% reported in the first experiment of this section)
* It seems like an important ablation study is testing the effect of jumpstarting the GAN training with the pretrained networks of stage 1, and reporting what happens when one or both of these networks are initialized randomly (assuming that initializing randomly hurts performance significantly, this would be further evidence of their framework's strengths)
* The novelty of the proposed approach is limited so far as I can tell (slight architectural modifications, and adding a negative sampling term to the discriminator loss)
Initial Rating - Weak Accept
Questions - 
* In experiment QTA3 I'm confused by how including multiple faces from the same video clip affects measuring retrieval accuracy. Does retrieving any of the 50 faces from the same clip count as correct? 
Explanation of Rating - The novelty of the work is limited, and it doesn't seem clearly useful for any practical task. However the stated task is certainly a non-trivial one, and the qualitative results and experiments give compelling evidence that the authors are proposing a powerful framework for conditional image generation. I think that the high quality writing, experiments, and results; coupled with potential impact for other conditional image generation tasks warrant acceptance. However the paper is held back by the lack of novelty and lack of clear motivation.
Revised Rating After Rebuttal: Accept
Explanation of Revised Rating: The authors addressed my concrete concerns about missing experiments in the rebuttal. Despite my concerns about the motivation for this particular task, I think the good results produced by author's methodology indicate this work will be valuable in the context of conditional image generation more broadly. I am upgrading my rating to accept.