Although generative adversarial networks (GANs) have enabled us to convert images from one domain to another similar one, converting between different sensory modalities, such as images and sounds, has been difficult. This study aims to propose a network that reconstructs images from sounds. First, video data with both images and sounds are labeled with pre-trained classifiers. Second, image and sound features are extracted from the data using pre-trained classifiers. Third, multimodal layers are introduced to extract features that are common to both the images and sounds. These layers are trained to extract similar features regardless of the input modality, such as images only, sounds only, and both images and sounds. Once the multimodal layers have been trained, features are extracted from input sounds and converted into image features using a feature-to-feature GAN. Finally, the generated image features are used to reconstruct images. Experimental results show that this method can successfully convert from the sound domain into the image domain. When we applied a pre-trained classifier to both the generated and original images, 31.9% of the examples had at least one of their top 10 labels in common, suggesting reasonably good image generation. Our results suggest that common representations can be learned for different modalities, and that proposed method can be applied not only to sound-to-image conversion but also to other conversions, such as from images to sounds.
Summary:
This paper addresses the problem of generating images from sound. The general idea is to use conditional GANs. In particular, two stacked conditional autoencoder GANs, where the autoencoders have a U-Net architecture. First, sound features are mapped into multimodal features that contain image feature/class information. Such multimodal features condition the generation of image features with an initial GAN, and the image features condition the generation of the output image with a second GAN. Although the problem itself is rather difficult, the solution is almost entirely based on previous work. The most novel part of the paper is the learning of the multimodal features. The final results are not very compelling, the literature review is very limited. There is a very high-level description of the approach with very few details, which leave the reader with a lot of unanswered questions. There is no attempt to compare with previous work, the architecture is not studied in depth with an ablation study, or compared with more interesting baselines, other than verifying that images can be generated from features (something not very surprising, given StackGUN!). Probably the more important verification is that the multimodal feature can embody some class information. Overall it seems to be a limited contribution.
Comments on quality, clarity, originality and significance:
This paper provides a high-level description for an approach to a problem that is relatively difficult to address. The paper is not very well motivated, and therefore lacks clarity and leaves the reader with a lot of unanswered questions. The literature review is limited as it is the set of results and comparison with previous works.
The paper addresses an important problem; however, I feel the work is not very significant because it does not reveal new techniques, nor produces compelling results, nor performs a deep analysis.
I think the problem is interesting, but a deeper analysis is needed to lift this contribution up to a significant one.
Below is a summary of some of the additional questions gathered while reading the paper:
Why there is the need for the multimodal features at all? Why can’t the sound be converted into class labels and then StackGAN can generate images?
Why do you need a feature-to-feature GAN? Why not generating images directly from the generated multimodal features? Motivations are not provided clearly.
Unclear architecture from Figure 1. The Feature-to-Feature GAN and the Feature-to-Image GAN have the same architecture? What does the encoder and decoder do? How are they organized? 
Looks like every piece is trained alone, no end-to-end learning, right? Please clarify that point.
No attempt to compare with other approaches has been made. Also, no effort to formulate a baseline model. What would happen if one were to use solely the features generated by SoundNet?
Would you be able to compare your multimodal features with those generated by Ngiam et al. (2011), for instance?
Section 3 refers to a 90/10 training/evaluation split but then it is unclear in what experiments that exact split is used.
No description on hyperparameters.
No complexity, no architecture details, (also no equations that could provide more details).
It should be clarified what it means one-to-one conversion. It is brought up in several points in the paper, but it is never clear what it means and therefore how it relates to what the Author intends to stress. 
It is unclear why by performing first a feature-to-feature mapping and later a feature-to-image mapping the one-to-one conversion problem should be addressed. In StackGAN the problem addressed is the resolution increase. The problem addressed by this paper is unclear.
Unclear what is the “ensemble effect”, and what is the motivation for upsampling a feature vector in two dimensions.
What image features were used to generate the images in Figure 3? Which architecture was used and how was it trained? Where these the same multimodal features used in the full architecture?
The paragraph motivating the need for multimodal features is unclear.
How are the three type of losses weighted for learning the multimodal layers? No discussion provided on that.
Unclear why a multimodal vector should be upsampled in 2 dimensions. 
The training procedure and loss for training the feature-to-feature conditional GAN is not explained.
Despite the difficulty of the problem, the generated images do not look compelling.
16 references do not seem enough by todays’ high-quality standard conferences.