LIMA: Less Is More for Alignment
Large language models are trained in two stages: (1) unsupervised pretraining
from raw text, to learn general-purpose representations, and (2) large scale
instruction tuning and reinforcement learning, to better align to end tasks and
user preferences. We measure the relative importance of these two stages by
training LIMA, a 65B parameter LLaMa language model fine-tuned with the
standard supervised loss on only 1,000 carefully curated prompts and responses,
without any reinforcement learning or human preference modeling. LIMA
demonstrates remarkably strong performance, learning to follow specific
response formats from only a handful of examples in the training data,
including complex queries that range from planning trip itineraries to
speculating about alternate history. Moreover, the model tends to generalize
well to unseen tasks that did not appear in the training data. In a controlled
human study, responses from LIMA are either equivalent or strictly preferred to
GPT-4 in 43% of cases; this statistic is as high as 58% when compared to Bard
and 65% versus DaVinci003, which was trained with human feedback. Taken
together, these results strongly suggest that almost all knowledge in large
language models is learned during pretraining, and only limited instruction
tuning data is necessary to teach models to produce high quality output.
646ac57431fbfc5df81f6f22
Chunting Zhou
646ac57431fbfc5df81f6f23
Pengfei Liu
claimed_verified
2023-05-23T06:39:37.453Z
6144a0c4ff1146bbd84d9865
https://cdn-avatars.huggingface.co/v1/production/uploads/1661715958139-6144a0c4ff1146bbd84d9865.png
Pengfei Liu
user
Pengfei
646ac57431fbfc5df81f6f24
Puxin Xu
646ac57431fbfc5df81f6f25
Srini Iyer
646ac57431fbfc5df81f6f26
Jiao Sun
claimed_verified
2023-05-22T09:36:50.111Z
642cd2f7ad221e8f41d2f05e
/avatars/64ad33b67c10e3cb7b9b07744d108d8c.svg
Jiao Sun
user
sunjiao123sun
646ac57431fbfc5df81f6f27
Yuning Mao
646ac57431fbfc5df81f6f28
Xuezhe Ma
claimed_verified
2024-04-22T06:56:26.978Z
6624517c204cf7d22a90306e
/avatars/82da7b7194f1d9ad49e8e1a470cf02ce.svg
Xuezhe Ma
user
maxma1987
646ac57431fbfc5df81f6f29
Avia Efrat
646ac57431fbfc5df81f6f2a
Ping Yu
646ac57431fbfc5df81f6f2b
Lili Yu
646ac57431fbfc5df81f6f2c
Susan Zhang
646ac57431fbfc5df81f6f2d
Gargi Ghosh
646ac57431fbfc5df81f6f2e
Mike Lewis
646ac57431fbfc5df81f6f2f
Luke Zettlemoyer
646ac57431fbfc5df81f6f30
Omer Levy
claimed_verified
2023-05-22T14:04:23.568Z
610b71d7f5dd7bcefedaff55
/avatars/b489c021e906010e661313da0b4aafab.svg
Omer Levy
user
omerlevy
LIMA: Less Is More for Alignment
Large language models are trained in two stages: (1) unsupervised pretraining
from raw text, to learn general-purpose representations, and (2) large scale
instruction tuning and reinforcement learning, to better align to end tasks and
user preferences. We measure the relative importance of these two stages by
training LIMA, a 65B parameter LLaMa language model fine-tuned with the
standard supervised loss on only 1,000 carefully curated prompts and responses,
without any reinforcement learning or human preference modeling. LIMA
demonstrates remarkably strong performance, learning to follow specific
response formats from only a handful of examples in the training data,
including complex queries that range from planning trip itineraries to
speculating about alternate history. Moreover, the model tends to generalize
well to unseen tasks that did not appear in the training data. In a controlled
human study, responses from LIMA are either equivalent or strictly preferred to
GPT-4 in 43% of cases; this statistic is as high as 58% when compared to Bard
and 65% versus DaVinci003, which was trained with human feedback. Taken
together, these results strongly suggest that almost all knowledge in large
language models is learned during pretraining, and only limited instruction
tuning data is necessary to teach models to produce high quality output.
A 65B parameter LLaMa language model trained with minimal instructional data matches or outperforms models with extensive human preference modeling in most cases, indicating that pretraining is predominantly responsible for knowledge acquisition.
large language models unsupervised pretraining LLaMa instruction tuning reinforcement learning supervised loss response formats human study GPT-4 Bard DaVinci003 knowledge acquisition