title: What Matters in Training a GPT4-Style Language Model with Multimodal
    Inputs?
summary: Recent advancements in Large Language Models (LLMs) such as GPT4 have
  displayed exceptional multi-modal capabilities in following open-ended
  instructions given images. However, the performance of these models heavily
  relies on design choices such as network structures, training data, and
  training strategies, and these choices have not been extensively discussed in
  the literature, making it difficult to quantify progress in this field. To
  address this issue, this paper presents a systematic and comprehensive study,
  quantitatively and qualitatively, on training such models. We implement over 20
  variants with controlled settings. Concretely, for network structures, we
  compare different LLM backbones and model designs. For training data, we
  investigate the impact of data and sampling strategies. For instructions, we
  explore the influence of diversified prompts on the instruction-following
  ability of the trained models. For benchmarks, we contribute the first, to our
  best knowledge, comprehensive evaluation set including both image and video
  tasks through crowd-sourcing. Based on our findings, we present Lynx, which
  performs the most accurate multi-modal understanding while keeping the best
  multi-modal generation ability compared to existing open-sourced GPT4-style
  models.
paper_authors._id: 64a62bf085638d0e698cf2d7
paper_authors.name: Yan Zeng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-06T15:04:27.097Z
paper_authors._id: 640ef2b2f2d7c41a1e9b6013
paper_authors.avatarUrl: /avatars/556263c07df93942c22cda64e65b2e41.svg
paper_authors.fullname: Yan Zeng
paper_authors.type: user
paper_authors.user: zengyan-97
paper_authors._id: 64a62bf085638d0e698cf2d8
paper_authors.name: Hanbo Zhang
paper_authors._id: 64a62bf085638d0e698cf2d9
paper_authors.name: Jiani Zheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-11-21T15:00:53.949Z
paper_authors._id: 64531f631a57e1179c203e6b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64531f631a57e1179c203e6b/C_J7pXFLqoJoHYPPhK3J9.jpeg
paper_authors.fullname: zjn
paper_authors.type: user
paper_authors.user: garlicisnotmyfavor
paper_authors._id: 64a62bf085638d0e698cf2da
paper_authors.name: Jiangnan Xia
paper_authors._id: 64a62bf085638d0e698cf2db
paper_authors.name: Guoqiang Wei
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-17T07:35:07.884Z
paper_authors._id: 63160a1d29411a6864b0e585
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1668083049513-63160a1d29411a6864b0e585.jpeg
paper_authors.fullname: Guoqiang Wei
paper_authors.type: user
paper_authors.user: weigq
paper_authors._id: 64a62bf085638d0e698cf2dc
paper_authors.name: Yang Wei
paper_authors._id: 64a62bf085638d0e698cf2dd
paper_authors.name: Yuchen Zhang
paper_authors._id: 64a62bf085638d0e698cf2de
paper_authors.name: Tao Kong
paper_title: What Matters in Training a GPT4-Style Language Model with Multimodal
    Inputs?
paper_summary: Recent advancements in Large Language Models (LLMs) such as GPT4 have
  displayed exceptional multi-modal capabilities in following open-ended
  instructions given images. However, the performance of these models heavily
  relies on design choices such as network structures, training data, and
  training strategies, and these choices have not been extensively discussed in
  the literature, making it difficult to quantify progress in this field. To
  address this issue, this paper presents a systematic and comprehensive study,
  quantitatively and qualitatively, on training such models. We implement over 20
  variants with controlled settings. Concretely, for network structures, we
  compare different LLM backbones and model designs. For training data, we
  investigate the impact of data and sampling strategies. For instructions, we
  explore the influence of diversified prompts on the instruction-following
  ability of the trained models. For benchmarks, we contribute the first, to our
  best knowledge, comprehensive evaluation set including both image and video
  tasks through crowd-sourcing. Based on our findings, we present Lynx, which
  performs the most accurate multi-modal understanding while keeping the best
  multi-modal generation ability compared to existing open-sourced GPT4-style
  models.
paper_ai_summary: A comprehensive study examines design choices impacting multi-modal capabilities of Large Language Models, introducing Lynx as the most accurate and versatile multi-modal model among existing GPT4-style models.
paper_ai_keywords: Large Language Models (LLMs) GPT4 multi-modal capabilities network structures LLM backbones model designs training data data and sampling strategies instruction-following ability comprehensive evaluation set multi-modal understanding multi-modal generation ability Lynx