title: Phantom of Latent for Large Language and Vision Models
summary: The success of visual instruction tuning has accelerated the development of
  large language and vision models (LLVMs). Following the scaling laws of
  instruction-tuned large language models (LLMs), LLVMs either have further
  increased their sizes, reaching 26B, 34B, and even 80B parameters. While this
  increase in model size has yielded significant performance gains, it demands
  substantially more hardware resources for both training and inference.
  Consequently, there naturally exists a strong need for efficient LLVMs that
  achieve the performance of larger models while being smaller in size. To
  achieve this need, we present a new efficient LLVM family with model sizes of
  0.5B, 1.8B, 3.8B, and 7B parameters, Phantom, which significantly enhances
  learning capabilities within limited structures. By temporarily increasing the
  latent hidden dimension during multi-head self-attention (MHSA), we make LLVMs
  prepare to look and understand much more vision-language knowledge on the
  latent, without substantially increasing physical model sizes. To maximize its
  advantage, we introduce Phantom Optimization (PO) using both autoregressive
  supervised fine-tuning (SFT) and direct preference optimization (DPO)-like
  concept, which effectively follows correct answers while eliminating incorrect
  and ambiguous ones. Phantom outperforms numerous larger open- and closed-source
  LLVMs, positioning itself as a leading solution in the landscape of efficient
  LLVMs.
paper_authors._id: 66f23c6aaef962b23792f12c
paper_authors.name: Byung-Kwan Lee
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-24T07:52:40.382Z
paper_authors._id: 657152eb12f162153b50ec9d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/657152eb12f162153b50ec9d/qnldHP35PclV0pDz_05q8.jpeg
paper_authors.fullname: Byung-Kwan Lee
paper_authors.type: user
paper_authors.user: BK-Lee
paper_authors._id: 66f23c6aaef962b23792f12d
paper_authors.name: Sangyun Chung
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-03T14:06:23.017Z
paper_authors._id: 65a4bf8e90b5e87bcdff41c7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65a4bf8e90b5e87bcdff41c7/I6OSoZigV7Fl6OKRl_Yqy.jpeg
paper_authors.fullname: Sangyun Chung
paper_authors.type: user
paper_authors.user: topyun
paper_authors._id: 66f23c6aaef962b23792f12e
paper_authors.name: Chae Won Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-24T07:52:38.696Z
paper_authors._id: 65aa2e84e2a2c86356263340
paper_authors.avatarUrl: /avatars/cba9ecceff0f67b2b5dde02e7c0a9991.svg
paper_authors.fullname: Chae Won Kim
paper_authors.type: user
paper_authors.user: chae-won-kim
paper_authors._id: 66f23c6aaef962b23792f12f
paper_authors.name: Beomchan Park
paper_authors._id: 66f23c6aaef962b23792f130
paper_authors.name: Yong Man Ro
paper_title: Phantom of Latent for Large Language and Vision Models
paper_summary: The success of visual instruction tuning has accelerated the development of
  large language and vision models (LLVMs). Following the scaling laws of
  instruction-tuned large language models (LLMs), LLVMs either have further
  increased their sizes, reaching 26B, 34B, and even 80B parameters. While this
  increase in model size has yielded significant performance gains, it demands
  substantially more hardware resources for both training and inference.
  Consequently, there naturally exists a strong need for efficient LLVMs that
  achieve the performance of larger models while being smaller in size. To
  achieve this need, we present a new efficient LLVM family with model sizes of
  0.5B, 1.8B, 3.8B, and 7B parameters, Phantom, which significantly enhances
  learning capabilities within limited structures. By temporarily increasing the
  latent hidden dimension during multi-head self-attention (MHSA), we make LLVMs
  prepare to look and understand much more vision-language knowledge on the
  latent, without substantially increasing physical model sizes. To maximize its
  advantage, we introduce Phantom Optimization (PO) using both autoregressive
  supervised fine-tuning (SFT) and direct preference optimization (DPO)-like
  concept, which effectively follows correct answers while eliminating incorrect
  and ambiguous ones. Phantom outperforms numerous larger open- and closed-source
  LLVMs, positioning itself as a leading solution in the landscape of efficient
  LLVMs.
paper_ai_keywords: multi-head self-attention (MHSA) latent hidden dimension autoregressive supervised fine-tuning (SFT) direct preference optimization (DPO) large language and vision models (LLVMs) large language models (LLMs)