  We develop an approach for text-to-image generation that embraces additional
retrieval images, driven by a combination of implicit visual guidance loss and
generative objectives. Unlike most existing text-to-image generation methods
which merely take the text as input, our method dynamically feeds cross-modal
search results into a unified training stage, hence improving the quality,
controllability and diversity of generation results. We propose a novel
hypernetwork modulated visual-text encoding scheme to predict the weight update
of the encoding layer, enabling effective transfer from visual information
(e.g. layout, content) into the corresponding latent domain. Experimental
results show that our model guided with additional retrieval visual data
outperforms existing GAN-based models. On COCO dataset, we achieve better FID
of $9.13$ with up to $3.5 \times$ fewer generator parameters, compared with the
state-of-the-art method.

Computer Vision and Pattern Recognition