title: Flacuna: Unleashing the Problem Solving Power of Vicuna using FLAN
    Fine-Tuning
summary: Recently, the release of INSTRUCTEVAL has provided valuable insights into the
  performance of large language models (LLMs) that utilize encoder-decoder or
  decoder-only architecture. Interestingly, despite being introduced four years
  ago, T5-based LLMs, such as FLAN-T5, continue to outperform the latest
  decoder-based LLMs, such as LLAMA and VICUNA, on tasks that require general
  problem-solving skills. This performance discrepancy can be attributed to three
  key factors: (1) Pre-training data, (2) Backbone architecture, and (3)
  Instruction dataset. In this technical report, our main focus is on
  investigating the impact of the third factor by leveraging VICUNA, a large
  language model based on LLAMA, which has undergone fine-tuning on ChatGPT
  conversations. To achieve this objective, we fine-tuned VICUNA using a
  customized instruction dataset collection called FLANMINI. This collection
  includes a subset of the large-scale instruction dataset known as FLAN, as well
  as various code-related datasets and conversational datasets derived from
  ChatGPT/GPT-4. This dataset comprises a large number of tasks that demand
  problem-solving skills. Our experimental findings strongly indicate that the
  enhanced problem-solving abilities of our model, FLACUNA, are obtained through
  fine-tuning VICUNA on the FLAN dataset, leading to significant improvements
  across numerous benchmark datasets in INSTRUCTEVAL. FLACUNA is publicly
  available at https://huggingface.co/declare-lab/flacuna-13b-v1.0.
paper_authors._id: 64a619c8bec7a6392342be9b
paper_authors.name: Deepanway Ghosal
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-06T14:52:12.589Z
paper_authors._id: 631246a8f568fb0098f5e72e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/631246a8f568fb0098f5e72e/kTcs1cgwFqd1Wjr3tolcL.jpeg
paper_authors.fullname: Deepanway
paper_authors.type: user
paper_authors.user: deepanway
paper_authors._id: 64a619c8bec7a6392342be9c
paper_authors.name: Yew Ken Chia
paper_authors._id: 64a619c8bec7a6392342be9d
paper_authors.name: Navonil Majumder
paper_authors._id: 64a619c8bec7a6392342be9e
paper_authors.name: Soujanya Poria
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-06T14:43:23.752Z
paper_authors._id: 626b626405fe1cb65725aca1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/626b626405fe1cb65725aca1/aa-Lata46I3fXOmMetvXH.jpeg
paper_authors.fullname: Soujanya Poria
paper_authors.type: user
paper_authors.user: soujanyaporia
paper_title: Flacuna: Unleashing the Problem Solving Power of Vicuna using FLAN
    Fine-Tuning
paper_summary: Recently, the release of INSTRUCTEVAL has provided valuable insights into the
  performance of large language models (LLMs) that utilize encoder-decoder or
  decoder-only architecture. Interestingly, despite being introduced four years
  ago, T5-based LLMs, such as FLAN-T5, continue to outperform the latest
  decoder-based LLMs, such as LLAMA and VICUNA, on tasks that require general
  problem-solving skills. This performance discrepancy can be attributed to three
  key factors: (1) Pre-training data, (2) Backbone architecture, and (3)
  Instruction dataset. In this technical report, our main focus is on
  investigating the impact of the third factor by leveraging VICUNA, a large
  language model based on LLAMA, which has undergone fine-tuning on ChatGPT
  conversations. To achieve this objective, we fine-tuned VICUNA using a
  customized instruction dataset collection called FLANMINI. This collection
  includes a subset of the large-scale instruction dataset known as FLAN, as well
  as various code-related datasets and conversational datasets derived from
  ChatGPT/GPT-4. This dataset comprises a large number of tasks that demand
  problem-solving skills. Our experimental findings strongly indicate that the
  enhanced problem-solving abilities of our model, FLACUNA, are obtained through
  fine-tuning VICUNA on the FLAN dataset, leading to significant improvements
  across numerous benchmark datasets in INSTRUCTEVAL. FLACUNA is publicly
  available at https://huggingface.co/declare-lab/flacuna-13b-v1.0.
paper_ai_summary: Fine-tuning VICUNA on a customized instruction dataset called FLANMINI enhances its problem-solving skills, leading to improved performance across various benchmarks.
paper_ai_keywords: INSTRUCTEVAL large language models encoder-decoder decoder-only T5-based FLAN-T5 LLAMA VICUNA fine-tuning ChatGPT conversations FLAN problem-solving skills benchmark datasets