TEAL: Tokenize and Embed ALL for Multi-modal Large Language Models
Despite Multi-modal Large Language Models (MM-LLMs) have made exciting
strides recently, they are still struggling to efficiently model the
interactions among multi-modal inputs and the generation in non-textual
modalities. In this work, we propose TEAL (Tokenize and Embed ALl)}, an
approach to treat the input from any modality as a token sequence and learn a
joint embedding space for all modalities. Specifically, for the input from any
modality, TEAL first discretizes it into a token sequence with the
off-the-shelf tokenizer and embeds the token sequence into a joint embedding
space with a learnable embedding matrix. MM-LLMs just need to predict the
multi-modal tokens autoregressively as the textual LLMs do. Finally, the
corresponding de-tokenizer is applied to generate the output in each modality
based on the predicted token sequence. With the joint embedding space, TEAL
enables the frozen LLMs to perform both understanding and generation tasks
involving non-textual modalities, such as image and audio. Thus, the textual
LLM can just work as an interface and maintain its high performance in textual
understanding and generation. Experiments show that TEAL achieves substantial
improvements in multi-modal understanding, and implements a simple scheme for
multi-modal generations.
654c4aa33e8861928003fc94
Zhen Yang
claimed_verified
2023-11-09T09:41:05.724Z
6447abfc4ca52cdb3188cdc1
/avatars/8439d36ac8e7d1fd08139e24131cb44d.svg
Zhen Yang
user
zhenyang042618
654c4aa33e8861928003fc95
Yingxue Zhang
654c4aa33e8861928003fc96
Fandong Meng
admin_assigned
2023-11-09T09:12:55.076Z
64cb254871a7bbb60c17d5fa
/avatars/5121fd5b7b55d275eba3947f3f4c034d.svg
Fandong Meng
user
fandong
654c4aa33e8861928003fc97
Jie Zhou
TEAL: Tokenize and Embed ALL for Multi-modal Large Language Models
Despite Multi-modal Large Language Models (MM-LLMs) have made exciting
strides recently, they are still struggling to efficiently model the
interactions among multi-modal inputs and the generation in non-textual
modalities. In this work, we propose TEAL (Tokenize and Embed ALl)}, an
approach to treat the input from any modality as a token sequence and learn a
joint embedding space for all modalities. Specifically, for the input from any
modality, TEAL first discretizes it into a token sequence with the
off-the-shelf tokenizer and embeds the token sequence into a joint embedding
space with a learnable embedding matrix. MM-LLMs just need to predict the
multi-modal tokens autoregressively as the textual LLMs do. Finally, the
corresponding de-tokenizer is applied to generate the output in each modality
based on the predicted token sequence. With the joint embedding space, TEAL
enables the frozen LLMs to perform both understanding and generation tasks
involving non-textual modalities, such as image and audio. Thus, the textual
LLM can just work as an interface and maintain its high performance in textual
understanding and generation. Experiments show that TEAL achieves substantial
improvements in multi-modal understanding, and implements a simple scheme for
multi-modal generations.
TEAL, a method that tokenizes and jointly embeds multi-modal inputs, enables existing frozen LLMs to handle multi-modal understanding and generation tasks with improved performance.
Multi-modal Large Language Models MM-LLMs Tokenize and Embed ALl token sequence joint embedding space off-the-shelf tokenizer learnable embedding matrix autoregressive prediction de-tokenizer multi-modal tokens multi-modal understanding multi-modal generations image audio textual understanding textual generation