title: Tuning Language Models by Proxy
summary: Despite the general capabilities of large pretrained language models, they
  consistently benefit from further adaptation to better achieve desired
  behaviors. However, tuning these models has become increasingly
  resource-intensive, or impossible when model weights are private. We introduce
  proxy-tuning, a lightweight decoding-time algorithm that operates on top of
  black-box LMs to achieve the result of directly tuning the model, but by
  accessing only its prediction over the output vocabulary. Our method instead
  tunes a smaller LM, then applies the difference between the predictions of the
  small tuned and untuned LMs to shift the original predictions of the base model
  in the direction of tuning, while retaining the benefits of larger scale
  pretraining. In experiments, when we apply proxy-tuning to Llama2-70B using
  proxies of only 7B size, we can close 88% of the gap between Llama2-70B and its
  truly-tuned chat version, when evaluated across knowledge, reasoning, and
  safety benchmarks. Interestingly, when tested on TruthfulQA, proxy-tuned models
  are actually more truthful than directly tuned models, possibly because
  decoding-time guidance better retains the model's factual knowledge. We then
  demonstrate the generality of proxy-tuning by applying it for domain adaptation
  on code, and task-specific finetuning on question-answering and math problems.
  Our work demonstrates the promise of using small tuned LMs to efficiently
  customize large, potentially proprietary LMs through decoding-time guidance.
paper_authors._id: 65a772455c58475cf91ca68d
paper_authors.name: Alisa Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-17T09:42:45.304Z
paper_authors._id: 62449da46b55ef458d430371
paper_authors.avatarUrl: /avatars/b95954a88c7b15524a3d7776a4abe083.svg
paper_authors.fullname: Alisa Liu
paper_authors.type: user
paper_authors.user: alisawuffles
paper_authors._id: 65a772455c58475cf91ca68e
paper_authors.name: Xiaochuang Han
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-17T09:42:51.194Z
paper_authors._id: 635ea3969f24f6db0a1e2d0b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/635ea3969f24f6db0a1e2d0b/RxaJKAx4MYMShb58yuUxy.png
paper_authors.fullname: Xiaochuang Han
paper_authors.type: user
paper_authors.user: xhan77
paper_authors._id: 65a772455c58475cf91ca68f
paper_authors.name: Yizhong Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-17T09:43:01.186Z
paper_authors._id: 6269be67d1ac0cde592aba29
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6269be67d1ac0cde592aba29/EP3zZbD7-jWK9ITvHGbnZ.jpeg
paper_authors.fullname: Yizhong Wang
paper_authors.type: user
paper_authors.user: yizhongw
paper_authors._id: 65a772455c58475cf91ca690
paper_authors.name: Yulia Tsvetkov
paper_authors._id: 65a772455c58475cf91ca691
paper_authors.name: Yejin Choi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-17T09:43:14.472Z
paper_authors._id: 64d42729f63b01b7f676b176
paper_authors.avatarUrl: /avatars/52e54bdd6a1fb6c774a40cd70f3d7925.svg
paper_authors.fullname: Yejin Choi
paper_authors.type: user
paper_authors.user: yejinchoinka
paper_authors._id: 65a772455c58475cf91ca692
paper_authors.name: Noah A. Smith
paper_title: Tuning Language Models by Proxy
paper_summary: Despite the general capabilities of large pretrained language models, they
  consistently benefit from further adaptation to better achieve desired
  behaviors. However, tuning these models has become increasingly
  resource-intensive, or impossible when model weights are private. We introduce
  proxy-tuning, a lightweight decoding-time algorithm that operates on top of
  black-box LMs to achieve the result of directly tuning the model, but by
  accessing only its prediction over the output vocabulary. Our method instead
  tunes a smaller LM, then applies the difference between the predictions of the
  small tuned and untuned LMs to shift the original predictions of the base model
  in the direction of tuning, while retaining the benefits of larger scale
  pretraining. In experiments, when we apply proxy-tuning to Llama2-70B using
  proxies of only 7B size, we can close 88% of the gap between Llama2-70B and its
  truly-tuned chat version, when evaluated across knowledge, reasoning, and
  safety benchmarks. Interestingly, when tested on TruthfulQA, proxy-tuned models
  are actually more truthful than directly tuned models, possibly because
  decoding-time guidance better retains the model's factual knowledge. We then
  demonstrate the generality of proxy-tuning by applying it for domain adaptation
  on code, and task-specific finetuning on question-answering and math problems.
  Our work demonstrates the promise of using small tuned LMs to efficiently
  customize large, potentially proprietary LMs through decoding-time guidance.
paper_ai_summary: Proxy-tuning, a lightweight decoding-time algorithm, customizes large language models efficiently using smaller LMs without accessing their weights, achieving competitive performance and better factual truthfulness.
paper_ai_keywords: proxy-tuning decoding-time algorithm black-box LMs prediction over the output vocabulary domain adaptation task-specific fine-tuning Llama2-70B TruthfulQA