title: Mini-GPTs: Efficient Large Language Models through Contextual Pruning
summary: In AI research, the optimization of Large Language Models (LLMs) remains a
  significant challenge, crucial for advancing the field's practical applications
  and sustainability. Building upon the foundational work of Professor Song Han's
  lab at MIT, this paper introduces a novel approach in developing Mini-GPTs via
  contextual pruning. Our methodology strategically prunes the computational
  architecture of traditional LLMs, like Phi-1.5, focusing on retaining core
  functionalities while drastically reducing model sizes. We employ the technique
  across diverse and complex datasets, including US law, Medical Q&A, Skyrim
  dialogue, English-Taiwanese translation, and Economics articles. The results
  underscore the efficiency and effectiveness of contextual pruning, not merely
  as a theoretical concept but as a practical tool in developing domain-specific,
  resource-efficient LLMs. Contextual pruning is a promising method for building
  domain-specific LLMs, and this research is a building block towards future
  development with more hardware compute, refined fine-tuning, and quantization.
paper_authors._id: 6583b2452031da25daaf4aca
paper_authors.name: Tim Valicenti
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-12-21T03:47:10.646Z
paper_authors._id: 6553f93939bce75e6cbc6b79
paper_authors.avatarUrl: /avatars/6e67ed9430a9e8e8ed6ac150080e4795.svg
paper_authors.fullname: Tim Valicenti
paper_authors.type: user
paper_authors.user: tval2
paper_authors._id: 6583b2452031da25daaf4acb
paper_authors.name: Justice Vidal
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2023-12-21T03:34:29.820Z
paper_authors._id: 656fdc362c8849c4ac0ac5ff
paper_authors.avatarUrl: /avatars/b8cf43771d9131e89fc208b7eac94ec1.svg
paper_authors.fullname: Justice Vidal
paper_authors.type: user
paper_authors.user: jmvidal08
paper_authors._id: 6583b2452031da25daaf4acc
paper_authors.name: Ritik Patnaik
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2023-12-21T03:34:29.820Z
paper_authors._id: 64274f4a13e5e91b2428775e
paper_authors.avatarUrl: /avatars/b97c577a1c209bf2d9effad1237bacca.svg
paper_authors.fullname: Ritik Patnaik
paper_authors.type: user
paper_authors.user: rik01
paper_title: Mini-GPTs: Efficient Large Language Models through Contextual Pruning
paper_summary: In AI research, the optimization of Large Language Models (LLMs) remains a
  significant challenge, crucial for advancing the field's practical applications
  and sustainability. Building upon the foundational work of Professor Song Han's
  lab at MIT, this paper introduces a novel approach in developing Mini-GPTs via
  contextual pruning. Our methodology strategically prunes the computational
  architecture of traditional LLMs, like Phi-1.5, focusing on retaining core
  functionalities while drastically reducing model sizes. We employ the technique
  across diverse and complex datasets, including US law, Medical Q&A, Skyrim
  dialogue, English-Taiwanese translation, and Economics articles. The results
  underscore the efficiency and effectiveness of contextual pruning, not merely
  as a theoretical concept but as a practical tool in developing domain-specific,
  resource-efficient LLMs. Contextual pruning is a promising method for building
  domain-specific LLMs, and this research is a building block towards future
  development with more hardware compute, refined fine-tuning, and quantization.
paper_ai_keywords: Mini-GPTs contextual pruning computational architecture Phi-1.5 US law Medical Q&A Skyrim dialogue English-Taiwanese translation Economics articles domain-specific resource-efficient LLMs quantization