title: Sorted LLaMA: Unlocking the Potential of Intermediate Layers of Large
    Language Models for Dynamic Inference Using Sorted Fine-Tuning (SoFT)
summary: The rapid advancement of large language models (LLMs) has revolutionized
  natural language processing (NLP). While these models excel at understanding
  and generating human-like text, their widespread deployment can be
  prohibitively expensive. SortedNet is a recent training technique for enabling
  dynamic inference for deep neural networks. It leverages network modularity to
  create sub-models with varying computational loads, sorting them based on
  computation/accuracy characteristics in a nested manner. We extend SortedNet to
  generative NLP tasks, making large language models dynamic without any
  pretraining and by only replacing standard Supervised Fine-Tuning (SFT) with
  Sorted Fine-Tuning (SoFT) at the same costs. Our approach boosts model
  efficiency, eliminating the need for multiple models for various scenarios
  during inference. We show that using this approach, we are able to unlock the
  potential of intermediate layers of transformers in generating the target
  output. Our sub-models remain integral components of the original model,
  minimizing storage requirements and transition costs between different
  computational/latency budgets. By applying this approach on LLaMa 2 13B for
  tuning on the Stanford Alpaca dataset and comparing it to normal tuning and
  early exit via PandaLM benchmark, we show that Sorted Fine-Tuning can deliver
  models twice as fast as the original model while maintaining or exceeding
  performance.
paper_authors._id: 65090daa7e0d56c2715720d3
paper_authors.name: Parsa Kavehzadeh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T13:15:58.939Z
paper_authors._id: 630e8c0f8df86f1e5bf3a510
paper_authors.avatarUrl: /avatars/b9e06d64339d0ebbfc0a8f5837ca13eb.svg
paper_authors.fullname: Parsa Kavehzadeh
paper_authors.type: user
paper_authors.user: parsareal
paper_authors._id: 65090daa7e0d56c2715720d4
paper_authors.name: Mojtaba Valipour
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-19T11:29:54.141Z
paper_authors._id: 63ecf846f765928ceeaae5bc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63ecf846f765928ceeaae5bc/PKDo1dERknoi-GFjmK2_L.png
paper_authors.fullname: Mojtaba Valipour
paper_authors.type: user
paper_authors.user: vpcom
paper_authors._id: 65090daa7e0d56c2715720d5
paper_authors.name: Marzieh Tahaei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T13:16:06.784Z
paper_authors._id: 6306331214e7d8892aaa3929
paper_authors.avatarUrl: /avatars/c5c1007487f18730c3e4df9c30c18337.svg
paper_authors.fullname: Marzieh Tahaei
paper_authors.type: user
paper_authors.user: marzieh7
paper_authors._id: 65090daa7e0d56c2715720d6
paper_authors.name: Ali Ghodsi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T13:16:14.918Z
paper_authors._id: 63bc8dc4b8c61b8aa49cbc28
paper_authors.avatarUrl: /avatars/c982663d88901584d65b7393af8d14a2.svg
paper_authors.fullname: Ali Ghodsi
paper_authors.type: user
paper_authors.user: alighodsi
paper_authors._id: 65090daa7e0d56c2715720d7
paper_authors.name: Boxing Chen
paper_authors._id: 65090daa7e0d56c2715720d8
paper_authors.name: Mehdi Rezagholizadeh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T13:16:29.230Z
paper_authors._id: 6348322f0744c555b6292073
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1665675864273-6348322f0744c555b6292073.jpeg
paper_authors.fullname: Mehdi Rezagholizadeh
paper_authors.type: user
paper_authors.user: mrgzadeh
paper_title: Sorted LLaMA: Unlocking the Potential of Intermediate Layers of Large
    Language Models for Dynamic Inference Using Sorted Fine-Tuning (SoFT)
paper_summary: The rapid advancement of large language models (LLMs) has revolutionized
  natural language processing (NLP). While these models excel at understanding
  and generating human-like text, their widespread deployment can be
  prohibitively expensive. SortedNet is a recent training technique for enabling
  dynamic inference for deep neural networks. It leverages network modularity to
  create sub-models with varying computational loads, sorting them based on
  computation/accuracy characteristics in a nested manner. We extend SortedNet to
  generative NLP tasks, making large language models dynamic without any
  pretraining and by only replacing standard Supervised Fine-Tuning (SFT) with
  Sorted Fine-Tuning (SoFT) at the same costs. Our approach boosts model
  efficiency, eliminating the need for multiple models for various scenarios
  during inference. We show that using this approach, we are able to unlock the
  potential of intermediate layers of transformers in generating the target
  output. Our sub-models remain integral components of the original model,
  minimizing storage requirements and transition costs between different
  computational/latency budgets. By applying this approach on LLaMa 2 13B for
  tuning on the Stanford Alpaca dataset and comparing it to normal tuning and
  early exit via PandaLM benchmark, we show that Sorted Fine-Tuning can deliver
  models twice as fast as the original model while maintaining or exceeding
  performance.
paper_ai_keywords: SortedNet dynamic inference network modularity sub-models computational loads accuracy characteristics nested manner generative NLP tasks Supervised Fine-Tuning (SFT) Sorted Fine-Tuning (SoFT) computational/latency budgets intermediate layers transformers Stanford Alpaca dataset PandaLM benchmark early exit