SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective
  Depth Up-Scaling
We introduce depth up-scaling (DUS), a novel technique to up-scale base LLMs
efficiently and effectively in a simple manner. In contrast to
mixture-of-experts (MoE), DUS does not require complex changes to train and
inference. Using DUS, we build SOLAR 10.7B, a large language model (LLM) with
10.7 billion parameters, demonstrating superior performance in various natural
language processing (NLP) tasks. Comparative evaluations show that SOLAR 10.7B
outperforms existing open-source pretrained LLMs, such as Llama 2 and Mistral
7B. We additionally present SOLAR 10.7B-Instruct, a variant fine-tuned for
instruction-following capabilities, surpassing Mixtral-8x7B. SOLAR 10.7B is
publicly available under the Apache 2.0 license, promoting broad access and
application in the LLM field.
658b86fe4dd2ca8497d27b6c
Dahyun Kim
658b86fe4dd2ca8497d27b6d
Chanjun Park
claimed_verified
2024-04-09T19:19:22.023Z
64b9fd6910430817357cd9b2
https://cdn-avatars.huggingface.co/v1/production/uploads/64b9fd6910430817357cd9b2/Zm_Ngp20IlHlLIOFlNXPo.jpeg
Chanjun Park
user
Chanjun
658b86fe4dd2ca8497d27b6e
Sanghoon Kim
658b86fe4dd2ca8497d27b6f
Wonsung Lee
658b86fe4dd2ca8497d27b70
Wonho Song
658b86fe4dd2ca8497d27b71
Yunsu Kim
658b86fe4dd2ca8497d27b72
Hyeonwoo Kim
658b86fe4dd2ca8497d27b73
Yungi Kim
658b86fe4dd2ca8497d27b74
Hyeonju Lee
658b86fe4dd2ca8497d27b75
Jihoo Kim
658b86fe4dd2ca8497d27b76
Changbae Ahn
658b86fe4dd2ca8497d27b77
Seonghoon Yang
658b86fe4dd2ca8497d27b78
Sukyung Lee
658b86fe4dd2ca8497d27b79
Hyunbyung Park
658b86fe4dd2ca8497d27b7a
Gyoungjin Gim
658b86fe4dd2ca8497d27b7b
Mikyoung Cha
658b86fe4dd2ca8497d27b7c
Hwalsuk Lee
658b86fe4dd2ca8497d27b7d
Sunghun Kim
SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective
  Depth Up-Scaling
We introduce depth up-scaling (DUS), a novel technique to up-scale base LLMs
efficiently and effectively in a simple manner. In contrast to
mixture-of-experts (MoE), DUS does not require complex changes to train and
inference. Using DUS, we build SOLAR 10.7B, a large language model (LLM) with
10.7 billion parameters, demonstrating superior performance in various natural
language processing (NLP) tasks. Comparative evaluations show that SOLAR 10.7B
outperforms existing open-source pretrained LLMs, such as Llama 2 and Mistral
7B. We additionally present SOLAR 10.7B-Instruct, a variant fine-tuned for
instruction-following capabilities, surpassing Mixtral-8x7B. SOLAR 10.7B is
publicly available under the Apache 2.0 license, promoting broad access and
application in the LLM field.
A novel technique, depth up-scaling (DUS), efficiently enhances large language models (LLMs) without complex changes, building SOLAR 10.7B that outperforms existing open-source models in various NLP tasks, including instruction-following.
depth up-scaling DUS mixture-of-experts MoE large language model LLM SOLAR 10.7B NLP Llama 2 Mistral 7B SOLAR 10.7B-Instruct Mixtral-8x7B Apache 2.0 license