title: Orca: Progressive Learning from Complex Explanation Traces of GPT-4
summary: Recent research has focused on enhancing the capability of smaller models
  through imitation learning, drawing on the outputs generated by large
  foundation models (LFMs). A number of issues impact the quality of these
  models, ranging from limited imitation signals from shallow LFM outputs; small
  scale homogeneous training data; and most notably a lack of rigorous evaluation
  resulting in overestimating the small model's capability as they tend to learn
  to imitate the style, but not the reasoning process of LFMs. To address these
  challenges, we develop Orca (We are working with our legal team to publicly
  release a diff of the model weights in accordance with LLaMA's release policy
  to be published at https://aka.ms/orca-lm), a 13-billion parameter model that
  learns to imitate the reasoning process of LFMs. Orca learns from rich signals
  from GPT-4 including explanation traces; step-by-step thought processes; and
  other complex instructions, guided by teacher assistance from ChatGPT. To
  promote this progressive learning, we tap into large-scale and diverse
  imitation data with judicious sampling and selection. Orca surpasses
  conventional state-of-the-art instruction-tuned models such as Vicuna-13B by
  more than 100% in complex zero-shot reasoning benchmarks like Big-Bench Hard
  (BBH) and 42% on AGIEval. Moreover, Orca reaches parity with ChatGPT on the BBH
  benchmark and shows competitive performance (4 pts gap with optimized system
  message) in professional and academic examinations like the SAT, LSAT, GRE, and
  GMAT, both in zero-shot settings without CoT; while trailing behind GPT-4. Our
  research indicates that learning from step-by-step explanations, whether these
  are generated by humans or more advanced AI models, is a promising direction to
  improve model capabilities and skills.
paper_authors._id: 647e843218274bce0307a091
paper_authors.name: Subhabrata Mukherjee
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-06-06T01:20:13.175Z
paper_authors._id: 5fd2dbbd9e5a0a03458e1fc8
paper_authors.avatarUrl: /avatars/d0868aef6c180db8610184396443731a.svg
paper_authors.fullname: Subhabrata Mukherjee
paper_authors.type: user
paper_authors.user: subho
paper_authors._id: 647e843218274bce0307a092
paper_authors.name: Arindam Mitra
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-06T17:23:01.086Z
paper_authors._id: 609ee7bfa4d9efdf9ce888aa
paper_authors.avatarUrl: /avatars/b5429d85c030a9f08e1d2bb37ce9974d.svg
paper_authors.fullname: Arindam Mitra
paper_authors.type: user
paper_authors.user: ari9dam
paper_authors._id: 647e843218274bce0307a093
paper_authors.name: Ganesh Jawahar
paper_authors._id: 647e843218274bce0307a094
paper_authors.name: Sahaj Agarwal
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-17T06:49:24.105Z
paper_authors._id: 64b12642540a2ba57874349a
paper_authors.avatarUrl: /avatars/e0ed6aa70e706865a6755c5fd10d5766.svg
paper_authors.fullname: Sahaj Agarwal
paper_authors.type: user
paper_authors.user: sahajgg
paper_authors._id: 647e843218274bce0307a095
paper_authors.name: Hamid Palangi
paper_authors._id: 647e843218274bce0307a096
paper_authors.name: Ahmed Awadallah
paper_title: Orca: Progressive Learning from Complex Explanation Traces of GPT-4
paper_summary: Recent research has focused on enhancing the capability of smaller models
  through imitation learning, drawing on the outputs generated by large
  foundation models (LFMs). A number of issues impact the quality of these
  models, ranging from limited imitation signals from shallow LFM outputs; small
  scale homogeneous training data; and most notably a lack of rigorous evaluation
  resulting in overestimating the small model's capability as they tend to learn
  to imitate the style, but not the reasoning process of LFMs. To address these
  challenges, we develop Orca (We are working with our legal team to publicly
  release a diff of the model weights in accordance with LLaMA's release policy
  to be published at https://aka.ms/orca-lm), a 13-billion parameter model that
  learns to imitate the reasoning process of LFMs. Orca learns from rich signals
  from GPT-4 including explanation traces; step-by-step thought processes; and
  other complex instructions, guided by teacher assistance from ChatGPT. To
  promote this progressive learning, we tap into large-scale and diverse
  imitation data with judicious sampling and selection. Orca surpasses
  conventional state-of-the-art instruction-tuned models such as Vicuna-13B by
  more than 100% in complex zero-shot reasoning benchmarks like Big-Bench Hard
  (BBH) and 42% on AGIEval. Moreover, Orca reaches parity with ChatGPT on the BBH
  benchmark and shows competitive performance (4 pts gap with optimized system
  message) in professional and academic examinations like the SAT, LSAT, GRE, and
  GMAT, both in zero-shot settings without CoT; while trailing behind GPT-4. Our
  research indicates that learning from step-by-step explanations, whether these
  are generated by humans or more advanced AI models, is a promising direction to
  improve model capabilities and skills.
paper_ai_keywords: imitation learning large foundation models (LFMs) shallow LFM outputs homogeneous training data evaluation Orca 13-billion parameter model GPT-4 explanation traces step-by-step thought processes teacher assistance ChatGPT large-scale and diverse imitation data judicious sampling and selection Vicuna-13B Big-Bench Hard (BBH) AGIEval parity SAT LSAT GRE GMAT zero-shot reasoning zero-shot settings CoT (Chain of Thought)