title: SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective
    Depth Up-Scaling
summary: We introduce depth up-scaling (DUS), a novel technique to up-scale base LLMs
  efficiently and effectively in a simple manner. In contrast to
  mixture-of-experts (MoE), DUS does not require complex changes to train and
  inference. Using DUS, we build SOLAR 10.7B, a large language model (LLM) with
  10.7 billion parameters, demonstrating superior performance in various natural
  language processing (NLP) tasks. Comparative evaluations show that SOLAR 10.7B
  outperforms existing open-source pretrained LLMs, such as Llama 2 and Mistral
  7B. We additionally present SOLAR 10.7B-Instruct, a variant fine-tuned for
  instruction-following capabilities, surpassing Mixtral-8x7B. SOLAR 10.7B is
  publicly available under the Apache 2.0 license, promoting broad access and
  application in the LLM field.
paper_authors._id: 658b86fe4dd2ca8497d27b6c
paper_authors.name: Dahyun Kim
paper_authors._id: 658b86fe4dd2ca8497d27b6d
paper_authors.name: Chanjun Park
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-09T19:19:22.023Z
paper_authors._id: 64b9fd6910430817357cd9b2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b9fd6910430817357cd9b2/Zm_Ngp20IlHlLIOFlNXPo.jpeg
paper_authors.fullname: Chanjun Park
paper_authors.type: user
paper_authors.user: Chanjun
paper_authors._id: 658b86fe4dd2ca8497d27b6e
paper_authors.name: Sanghoon Kim
paper_authors._id: 658b86fe4dd2ca8497d27b6f
paper_authors.name: Wonsung Lee
paper_authors._id: 658b86fe4dd2ca8497d27b70
paper_authors.name: Wonho Song
paper_authors._id: 658b86fe4dd2ca8497d27b71
paper_authors.name: Yunsu Kim
paper_authors._id: 658b86fe4dd2ca8497d27b72
paper_authors.name: Hyeonwoo Kim
paper_authors._id: 658b86fe4dd2ca8497d27b73
paper_authors.name: Yungi Kim
paper_authors._id: 658b86fe4dd2ca8497d27b74
paper_authors.name: Hyeonju Lee
paper_authors._id: 658b86fe4dd2ca8497d27b75
paper_authors.name: Jihoo Kim
paper_authors._id: 658b86fe4dd2ca8497d27b76
paper_authors.name: Changbae Ahn
paper_authors._id: 658b86fe4dd2ca8497d27b77
paper_authors.name: Seonghoon Yang
paper_authors._id: 658b86fe4dd2ca8497d27b78
paper_authors.name: Sukyung Lee
paper_authors._id: 658b86fe4dd2ca8497d27b79
paper_authors.name: Hyunbyung Park
paper_authors._id: 658b86fe4dd2ca8497d27b7a
paper_authors.name: Gyoungjin Gim
paper_authors._id: 658b86fe4dd2ca8497d27b7b
paper_authors.name: Mikyoung Cha
paper_authors._id: 658b86fe4dd2ca8497d27b7c
paper_authors.name: Hwalsuk Lee
paper_authors._id: 658b86fe4dd2ca8497d27b7d
paper_authors.name: Sunghun Kim
paper_title: SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective
    Depth Up-Scaling
paper_summary: We introduce depth up-scaling (DUS), a novel technique to up-scale base LLMs
  efficiently and effectively in a simple manner. In contrast to
  mixture-of-experts (MoE), DUS does not require complex changes to train and
  inference. Using DUS, we build SOLAR 10.7B, a large language model (LLM) with
  10.7 billion parameters, demonstrating superior performance in various natural
  language processing (NLP) tasks. Comparative evaluations show that SOLAR 10.7B
  outperforms existing open-source pretrained LLMs, such as Llama 2 and Mistral
  7B. We additionally present SOLAR 10.7B-Instruct, a variant fine-tuned for
  instruction-following capabilities, surpassing Mixtral-8x7B. SOLAR 10.7B is
  publicly available under the Apache 2.0 license, promoting broad access and
  application in the LLM field.
paper_ai_summary: A novel technique, depth up-scaling (DUS), efficiently enhances large language models (LLMs) without complex changes, building SOLAR 10.7B that outperforms existing open-source models in various NLP tasks, including instruction-following.
paper_ai_keywords: depth up-scaling DUS mixture-of-experts MoE large language model LLM SOLAR 10.7B NLP Llama 2 Mistral 7B SOLAR 10.7B-Instruct Mixtral-8x7B Apache 2.0 license