Patience Is The Key to Large Language Model Reasoning
Recent advancements in the field of large language models, particularly
through the Chain of Thought (CoT) approach, have demonstrated significant
improvements in solving complex problems. However, existing models either tend
to sacrifice detailed reasoning for brevity due to user preferences, or require
extensive and expensive training data to learn complicated reasoning ability,
limiting their potential in solving complex tasks. To bridge this gap,
following the concept of scaling test-time, we propose a simple method by
encouraging models to adopt a more patient reasoning style without the need of
introducing new knowledge or skills. To employ a preference optimization
approach, we generate detailed reasoning processes as positive examples and
simple answers as negative examples, thereby training the model to favor
thoroughness in its responses. Our results demonstrate a performance increase
of up to 6.7% on GSM8k with training just on a lightweight dataset.
673ebf5a4edfc6d3b57b8376
Yijiong Yu
admin_assigned
2024-11-22T13:21:54.207Z
6374c494958cd71fa7ea0a9d
https://cdn-avatars.huggingface.co/v1/production/uploads/6374c494958cd71fa7ea0a9d/2YCKv6tXCZXtsIOFIIXjs.png
yuyijiong
user
yuyijiong
Patience Is The Key to Large Language Model Reasoning
Recent advancements in the field of large language models, particularly
through the Chain of Thought (CoT) approach, have demonstrated significant
improvements in solving complex problems. However, existing models either tend
to sacrifice detailed reasoning for brevity due to user preferences, or require
extensive and expensive training data to learn complicated reasoning ability,
limiting their potential in solving complex tasks. To bridge this gap,
following the concept of scaling test-time, we propose a simple method by
encouraging models to adopt a more patient reasoning style without the need of
introducing new knowledge or skills. To employ a preference optimization
approach, we generate detailed reasoning processes as positive examples and
simple answers as negative examples, thereby training the model to favor
thoroughness in its responses. Our results demonstrate a performance increase
of up to 6.7% on GSM8k with training just on a lightweight dataset.
A method encouraging detailed reasoning in large language models via preference optimization improves performance on complex tasks using a lightweight dataset.
Chain of Thought large language models patient reasoning preference optimization GSM8k