title: TEAL: Tokenize and Embed ALL for Multi-modal Large Language Models
summary: Despite Multi-modal Large Language Models (MM-LLMs) have made exciting
  strides recently, they are still struggling to efficiently model the
  interactions among multi-modal inputs and the generation in non-textual
  modalities. In this work, we propose TEAL (Tokenize and Embed ALl)}, an
  approach to treat the input from any modality as a token sequence and learn a
  joint embedding space for all modalities. Specifically, for the input from any
  modality, TEAL first discretizes it into a token sequence with the
  off-the-shelf tokenizer and embeds the token sequence into a joint embedding
  space with a learnable embedding matrix. MM-LLMs just need to predict the
  multi-modal tokens autoregressively as the textual LLMs do. Finally, the
  corresponding de-tokenizer is applied to generate the output in each modality
  based on the predicted token sequence. With the joint embedding space, TEAL
  enables the frozen LLMs to perform both understanding and generation tasks
  involving non-textual modalities, such as image and audio. Thus, the textual
  LLM can just work as an interface and maintain its high performance in textual
  understanding and generation. Experiments show that TEAL achieves substantial
  improvements in multi-modal understanding, and implements a simple scheme for
  multi-modal generations.
paper_authors._id: 654c4aa33e8861928003fc94
paper_authors.name: Zhen Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-11-09T09:41:05.724Z
paper_authors._id: 6447abfc4ca52cdb3188cdc1
paper_authors.avatarUrl: /avatars/8439d36ac8e7d1fd08139e24131cb44d.svg
paper_authors.fullname: Zhen Yang
paper_authors.type: user
paper_authors.user: zhenyang042618
paper_authors._id: 654c4aa33e8861928003fc95
paper_authors.name: Yingxue Zhang
paper_authors._id: 654c4aa33e8861928003fc96
paper_authors.name: Fandong Meng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-09T09:12:55.076Z
paper_authors._id: 64cb254871a7bbb60c17d5fa
paper_authors.avatarUrl: /avatars/5121fd5b7b55d275eba3947f3f4c034d.svg
paper_authors.fullname: Fandong Meng
paper_authors.type: user
paper_authors.user: fandong
paper_authors._id: 654c4aa33e8861928003fc97
paper_authors.name: Jie Zhou
paper_title: TEAL: Tokenize and Embed ALL for Multi-modal Large Language Models
paper_summary: Despite Multi-modal Large Language Models (MM-LLMs) have made exciting
  strides recently, they are still struggling to efficiently model the
  interactions among multi-modal inputs and the generation in non-textual
  modalities. In this work, we propose TEAL (Tokenize and Embed ALl)}, an
  approach to treat the input from any modality as a token sequence and learn a
  joint embedding space for all modalities. Specifically, for the input from any
  modality, TEAL first discretizes it into a token sequence with the
  off-the-shelf tokenizer and embeds the token sequence into a joint embedding
  space with a learnable embedding matrix. MM-LLMs just need to predict the
  multi-modal tokens autoregressively as the textual LLMs do. Finally, the
  corresponding de-tokenizer is applied to generate the output in each modality
  based on the predicted token sequence. With the joint embedding space, TEAL
  enables the frozen LLMs to perform both understanding and generation tasks
  involving non-textual modalities, such as image and audio. Thus, the textual
  LLM can just work as an interface and maintain its high performance in textual
  understanding and generation. Experiments show that TEAL achieves substantial
  improvements in multi-modal understanding, and implements a simple scheme for
  multi-modal generations.
paper_ai_summary: TEAL, a method that tokenizes and jointly embeds multi-modal inputs, enables existing frozen LLMs to handle multi-modal understanding and generation tasks with improved performance.
paper_ai_keywords: Multi-modal Large Language Models MM-LLMs Tokenize and Embed ALl token sequence joint embedding space off-the-shelf tokenizer learnable embedding matrix autoregressive prediction de-tokenizer multi-modal tokens multi-modal understanding multi-modal generations image audio textual understanding textual generation