title: u-LLaVA: Unifying Multi-Modal Tasks via Large Language Model
summary: Recent advances such as LLaVA and Mini-GPT4 have successfully integrated
  visual information into LLMs, yielding inspiring outcomes and giving rise to a
  new generation of multi-modal LLMs, or MLLMs. Nevertheless, these methods
  struggle with hallucinations and the mutual interference between tasks. To
  tackle these problems, we propose an efficient and accurate approach to adapt
  to downstream tasks by utilizing LLM as a bridge to connect multiple expert
  models, namely u-LLaVA. Firstly, we incorporate the modality alignment module
  and multi-task modules into LLM. Then, we reorganize or rebuild multi-type
  public datasets to enable efficient modality alignment and instruction
  following. Finally, task-specific information is extracted from the trained LLM
  and provided to different modules for solving downstream tasks. The overall
  framework is simple, effective, and achieves state-of-the-art performance
  across multiple benchmarks. We also release our model, the generated data, and
  the code base publicly available.
paper_authors._id: 654db51d0011607c312089c7
paper_authors.name: Jinjin Xu
paper_authors._id: 654db51d0011607c312089c8
paper_authors.name: Liwu Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-10T11:14:18.355Z
paper_authors._id: 641f47743c49fc33ced8f414
paper_authors.avatarUrl: /avatars/e8269d87541f3241034762ae8aaae296.svg
paper_authors.fullname: xlw
paper_authors.type: user
paper_authors.user: liwu1995
paper_authors._id: 654db51d0011607c312089c9
paper_authors.name: Yuzhe Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-10T11:14:33.679Z
paper_authors._id: 631c3672c73939ffc0714b1f
paper_authors.avatarUrl: /avatars/c6f8f69294881532034404ab8e2fe238.svg
paper_authors.fullname: yangyuzhe
paper_authors.type: user
paper_authors.user: nekoosuki
paper_authors._id: 654db51d0011607c312089ca
paper_authors.name: Xiang Li
paper_authors._id: 654db51d0011607c312089cb
paper_authors.name: Yanchun Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-10T11:15:08.170Z
paper_authors._id: 62a69c16db0b844f8b15e6e1
paper_authors.avatarUrl: /avatars/971811e630bd5354a008f6abe3bffcce.svg
paper_authors.fullname: yanchun xie
paper_authors.type: user
paper_authors.user: xyc
paper_authors._id: 654db51d0011607c312089cc
paper_authors.name: Yi-Jie Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-10T11:15:21.281Z
paper_authors._id: 648136af7d65d9ac17346bac
paper_authors.avatarUrl: /avatars/7fe65fe712b69f16d81123f0d67719ab.svg
paper_authors.fullname: Yijie Huang
paper_authors.type: user
paper_authors.user: asdfagafgs
paper_authors._id: 654db51d0011607c312089cd
paper_authors.name: Yaqian Li
paper_title: u-LLaVA: Unifying Multi-Modal Tasks via Large Language Model
paper_summary: Recent advances such as LLaVA and Mini-GPT4 have successfully integrated
  visual information into LLMs, yielding inspiring outcomes and giving rise to a
  new generation of multi-modal LLMs, or MLLMs. Nevertheless, these methods
  struggle with hallucinations and the mutual interference between tasks. To
  tackle these problems, we propose an efficient and accurate approach to adapt
  to downstream tasks by utilizing LLM as a bridge to connect multiple expert
  models, namely u-LLaVA. Firstly, we incorporate the modality alignment module
  and multi-task modules into LLM. Then, we reorganize or rebuild multi-type
  public datasets to enable efficient modality alignment and instruction
  following. Finally, task-specific information is extracted from the trained LLM
  and provided to different modules for solving downstream tasks. The overall
  framework is simple, effective, and achieves state-of-the-art performance
  across multiple benchmarks. We also release our model, the generated data, and
  the code base publicly available.
paper_ai_summary: A framework u-LLaVA integrates visual information into LLMs using modality alignment and multi-task modules to address hallucinations and task interference, achieving state-of-the-art performance.
paper_ai_keywords: LLaVA Mini-GPT4 multi-modal LLMs MLLMs modality alignment module multi-task modules task-specific information