KoCLIP
Building on top of Korean Language models that are publicly available, we want to train a multimodal generative system. Specifically, we train CLIP on Korean datasets using KoBERT 10 and ViT as backbones.

Model
ViT, KoBERT (or any other Korean encoder LM)

Datasets
KETI has released a Korean image captioning dataset, available on AI Hub 11. We can also utilize the WIT dataset 13, which is a multilingual dataset scraped from Wikipedia.

Training Script
The training script is (almost) already available, see here 34.

Challenges

KoBERT likely performs slightly worse than the default English BERT. While this could be a bottleneck, we can always write the script in such a way that it is easy to plug-and-play different LMs. This way, when a better Korean LM is released, it can easily be used.
There might be minor architectural adjustments we have to make (e.g. adding projection layers).
Attaining fluency in JAX will take time and effort.


Desired Outcomes
The final deliverable of this project will most likely be an open source repository, accompanying documentation, model weights, and potentially a demo Streamlit app.