I created a Discord channel here 10.
Let’s try to finalize the details. I tried to summarize all the ideas from the discussion:


task: image generation from text


model

image encoder: JAX version of VQGAN 3 (@valhalla I assume it needs to be trained?)
text encoder: JAX GPT2
model/GAN structure

Option 1: build our own structure as suggested by @Vaibhavbrkn (text embeddings → image GAN) + maybe inspiration from DALLE-pytorch and CogView 1. Can be difficult (I always have trouble training GAN’s) but a great learning experience too!
Option 2: train XMC-GAN 3, I imagine this one is actually fully complete and we would just need to set up the dataset and plug everything in correctly.





dataset (thanks @valhalla):


Conceptual Caption 4 - ~3.3M image-text pairs

Conceptual 12M 2 - 12M

WIT 1- ~37M



training script: TODO


outcome: model that accepts a sentence and generate a realistic image from it


Also feel free to pick a specific area you would want to focus on!
On my side I can also help instrument the training with W&B so we can visualize metrics and some sample images generated live.