Computer Vision, Artificial Intelligence, Machine Learning, Data Science, Year In Review.
we want to let LLMs loose in the real world as agents, they will need other senses with which to perceive the world. Multimodal LLMs (MLLMs) bridge this modality gap by giving an LLM the ability to accept tokens of more than one modality as input. In most cases, a pre-trained LLM is connected to a