Artificial Intelligence, Research, Machine Learning, NLP.
it’s simple: tokenize images, states, and language into a shared vocabulary, then train a joint model for next-token prediction. To encode images, they use the Vision Transformer (ViT) and to encode text PaLM, both from Google and up to a combined 562 billion parameters (22B + 540B respectively).