title: TroL: Traversal of Layers for Large Language and Vision Models
summary: Large language and vision models (LLVMs) have been driven by the
  generalization power of large language models (LLMs) and the advent of visual
  instruction tuning. Along with scaling them up directly, these models enable
  LLVMs to showcase powerful vision language (VL) performances by covering
  diverse tasks via natural language instructions. However, existing open-source
  LLVMs that perform comparably to closed-source LLVMs such as GPT-4V are often
  considered too large (e.g., 26B, 34B, and 110B parameters), having a larger
  number of layers. These large models demand costly, high-end resources for both
  training and inference. To address this issue, we present a new efficient LLVM
  family with 1.8B, 3.8B, and 7B LLM model sizes, Traversal of Layers (TroL),
  which enables the reuse of layers in a token-wise manner. This layer traversing
  technique simulates the effect of looking back and retracing the answering
  stream while increasing the number of forward propagation layers without
  physically adding more layers. We demonstrate that TroL employs a simple layer
  traversing approach yet efficiently outperforms the open-source LLVMs with
  larger model sizes and rivals the performances of the closed-source LLVMs with
  substantial sizes.
paper_authors._id: 6672470286f6b3d1267ddce3
paper_authors.name: Byung-Kwan Lee
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-19T07:38:10.524Z
paper_authors._id: 657152eb12f162153b50ec9d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/657152eb12f162153b50ec9d/qnldHP35PclV0pDz_05q8.jpeg
paper_authors.fullname: Byung-Kwan Lee
paper_authors.type: user
paper_authors.user: BK-Lee
paper_authors._id: 6672470286f6b3d1267ddce4
paper_authors.name: Sangyun Chung
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-19T08:09:46.674Z
paper_authors._id: 65a4bf8e90b5e87bcdff41c7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65a4bf8e90b5e87bcdff41c7/I6OSoZigV7Fl6OKRl_Yqy.jpeg
paper_authors.fullname: Sangyun Chung
paper_authors.type: user
paper_authors.user: topyun
paper_authors._id: 6672470286f6b3d1267ddce5
paper_authors.name: Chae Won Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-24T07:33:15.975Z
paper_authors._id: 65aa2e84e2a2c86356263340
paper_authors.avatarUrl: /avatars/cba9ecceff0f67b2b5dde02e7c0a9991.svg
paper_authors.fullname: Chae Won Kim
paper_authors.type: user
paper_authors.user: chae-won-kim
paper_authors._id: 6672470286f6b3d1267ddce6
paper_authors.name: Beomchan Park
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-19T07:38:08.480Z
paper_authors._id: 65390e7ce279d2fc80a3f800
paper_authors.avatarUrl: /avatars/a930c1f20fa1eef372ca92139cb32887.svg
paper_authors.fullname: Beomchan Park
paper_authors.type: user
paper_authors.user: bpbpbp0810
paper_authors._id: 6672470286f6b3d1267ddce7
paper_authors.name: Yong Man Ro
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-19T08:10:04.741Z
paper_authors._id: 660603529e3555d648e3baf1
paper_authors.avatarUrl: /avatars/0f54479afcfc19df00b25d5aedb4cf67.svg
paper_authors.fullname: Yong Man Ro
paper_authors.type: user
paper_authors.user: dwightro
paper_title: TroL: Traversal of Layers for Large Language and Vision Models
paper_summary: Large language and vision models (LLVMs) have been driven by the
  generalization power of large language models (LLMs) and the advent of visual
  instruction tuning. Along with scaling them up directly, these models enable
  LLVMs to showcase powerful vision language (VL) performances by covering
  diverse tasks via natural language instructions. However, existing open-source
  LLVMs that perform comparably to closed-source LLVMs such as GPT-4V are often
  considered too large (e.g., 26B, 34B, and 110B parameters), having a larger
  number of layers. These large models demand costly, high-end resources for both
  training and inference. To address this issue, we present a new efficient LLVM
  family with 1.8B, 3.8B, and 7B LLM model sizes, Traversal of Layers (TroL),
  which enables the reuse of layers in a token-wise manner. This layer traversing
  technique simulates the effect of looking back and retracing the answering
  stream while increasing the number of forward propagation layers without
  physically adding more layers. We demonstrate that TroL employs a simple layer
  traversing approach yet efficiently outperforms the open-source LLVMs with
  larger model sizes and rivals the performances of the closed-source LLVMs with
  substantial sizes.
paper_ai_summary: The TroL model efficiently reduces the size of large language and vision models while maintaining or surpassing performance through a layer traversal technique that reuses layers token-wise.
paper_ai_keywords: LLVMs LLMs VL performances visual instruction tuning GPT-4V layer traversing token-wise forward propagation layers