So this would be a hybrid dialog system combining intent recognition with text-to-image generation. The idea is to train the model to generate visual replies (static or dynamic GIFs, depending on how hard the latter is) to English-language text input. E.g.:
text input: ‘this restaurant is the bomb’
model output: generated picture of exploding restaurant
There are quite a few GIF and text-to-image datasets out there, but we’d need to combine them in a way that the model can be trained. A large part of the work will be to implement VQGAN and CLIP in JAX.