title: DeepSpeed-VisualChat: Multi-Round Multi-Image Interleave Chat via
    Multi-Modal Causal Attention
summary: Most of the existing multi-modal models, hindered by their incapacity to
  adeptly manage interleaved image-and-text inputs in multi-image, multi-round
  dialogues, face substantial constraints in resource allocation for training and
  data accessibility, impacting their adaptability and scalability across varied
  interaction realms. To address this, we present the DeepSpeed-VisualChat
  framework, designed to optimize Large Language Models (LLMs) by incorporating
  multi-modal capabilities, with a focus on enhancing the proficiency of Large
  Vision and Language Models in handling interleaved inputs. Our framework is
  notable for (1) its open-source support for multi-round and multi-image
  dialogues, (2) introducing an innovative multi-modal causal attention
  mechanism, and (3) utilizing data blending techniques on existing datasets to
  assure seamless interactions in multi-round, multi-image conversations.
  Compared to existing frameworks, DeepSpeed-VisualChat shows superior
  scalability up to 70B parameter language model size, representing a significant
  advancement in multi-modal language models and setting a solid foundation for
  future explorations.
paper_authors._id: 65127f81e7fb861a6aa86ee1
paper_authors.name: Zhewei Yao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-26T13:26:14.275Z
paper_authors._id: 6352e1953199c2612ec2ff01
paper_authors.avatarUrl: /avatars/f75444f1b7cba63ed085774a950a3fc7.svg
paper_authors.fullname: Zhewei Yao
paper_authors.type: user
paper_authors.user: zheweiyao
paper_authors._id: 65127f81e7fb861a6aa86ee2
paper_authors.name: Xiaoxia Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-26T18:51:25.710Z
paper_authors._id: 619721da1726b360d54d9ff7
paper_authors.avatarUrl: /avatars/f5f8bf0f372bef58848530dd7d7f58de.svg
paper_authors.fullname: wu
paper_authors.type: user
paper_authors.user: xiaoxia-microsoft
paper_authors._id: 65127f81e7fb861a6aa86ee3
paper_authors.name: Conglong Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-26T13:40:23.730Z
paper_authors._id: 617adc6d941993035fbaf2b3
paper_authors.avatarUrl: /avatars/8de7dc3a6e202a49ac9c061e93d0fc92.svg
paper_authors.fullname: Conglong Li
paper_authors.type: user
paper_authors.user: conglongli
paper_authors._id: 65127f81e7fb861a6aa86ee4
paper_authors.name: Minjia Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-26T13:40:39.286Z
paper_authors._id: 6305b5e39d2531fabd195c5f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6305b5e39d2531fabd195c5f/oGS5leXVrB8Ct_BF96VkR.jpeg
paper_authors.fullname: Zhang
paper_authors.type: user
paper_authors.user: Minjia
paper_authors._id: 65127f81e7fb861a6aa86ee5
paper_authors.name: Heyang Qi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-26T13:40:46.337Z
paper_authors._id: 63bc6a611374e3ef9134a24e
paper_authors.avatarUrl: /avatars/9df037b9cf7d7952a10d919e24a2ffa2.svg
paper_authors.fullname: Heyang Qin
paper_authors.type: user
paper_authors.user: heyangqin
paper_authors._id: 65127f81e7fb861a6aa86ee6
paper_authors.name: Olatunji Ruwase
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-26T13:40:53.473Z
paper_authors._id: 6499d685ea2cdac80992e742
paper_authors.avatarUrl: /avatars/c32741e7fc57c9a08722fab3877a7b81.svg
paper_authors.fullname: Olatunji Ruwase
paper_authors.type: user
paper_authors.user: tjruwase
paper_authors._id: 65127f81e7fb861a6aa86ee7
paper_authors.name: Ammar Ahmad Awan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-26T13:41:09.098Z
paper_authors._id: 6324b8417d6d0cbbe2c65019
paper_authors.avatarUrl: /avatars/7c1a468daaa984203a600a3b3526298f.svg
paper_authors.fullname: Ammar Awan
paper_authors.type: user
paper_authors.user: ammarawan
paper_authors._id: 65127f81e7fb861a6aa86ee8
paper_authors.name: Samyam Rajbhandari
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-26T13:41:16.702Z
paper_authors._id: 63238e79d444e1d90888ea03
paper_authors.avatarUrl: /avatars/cd0accb5ddec4f2ae23c94da4094b5e5.svg
paper_authors.fullname: Samyam Rajbhandari
paper_authors.type: user
paper_authors.user: samyam
paper_authors._id: 65127f81e7fb861a6aa86ee9
paper_authors.name: Yuxiong He
paper_title: DeepSpeed-VisualChat: Multi-Round Multi-Image Interleave Chat via
    Multi-Modal Causal Attention
paper_summary: Most of the existing multi-modal models, hindered by their incapacity to
  adeptly manage interleaved image-and-text inputs in multi-image, multi-round
  dialogues, face substantial constraints in resource allocation for training and
  data accessibility, impacting their adaptability and scalability across varied
  interaction realms. To address this, we present the DeepSpeed-VisualChat
  framework, designed to optimize Large Language Models (LLMs) by incorporating
  multi-modal capabilities, with a focus on enhancing the proficiency of Large
  Vision and Language Models in handling interleaved inputs. Our framework is
  notable for (1) its open-source support for multi-round and multi-image
  dialogues, (2) introducing an innovative multi-modal causal attention
  mechanism, and (3) utilizing data blending techniques on existing datasets to
  assure seamless interactions in multi-round, multi-image conversations.
  Compared to existing frameworks, DeepSpeed-VisualChat shows superior
  scalability up to 70B parameter language model size, representing a significant
  advancement in multi-modal language models and setting a solid foundation for
  future explorations.
paper_ai_summary: A framework called DeepSpeed-VisualChat optimizes LLMs for multi-modal, multi-image, and multi-round dialogues using an innovative causal attention mechanism and data blending techniques.
paper_ai_keywords: DeepSpeed-VisualChat Large Language Models (LLMs) Large Vision and Language Models multi-modal causal attention mechanism data blending techniques