title: Tailoring Self-Rationalizers with Multi-Reward Distillation
summary: Large language models (LMs) are capable of generating free-text rationales to
  aid question answering. However, prior work 1) suggests that useful
  self-rationalization is emergent only at significant scales (e.g., 175B
  parameter GPT-3); and 2) focuses largely on downstream performance, ignoring
  the semantics of the rationales themselves, e.g., are they faithful, true, and
  helpful for humans? In this work, we enable small-scale LMs (approx. 200x
  smaller than GPT-3) to generate rationales that not only improve downstream
  task performance, but are also more plausible, consistent, and diverse,
  assessed both by automatic and human evaluation. Our method, MaRio
  (Multi-rewArd RatIOnalization), is a multi-reward conditioned
  self-rationalization algorithm that optimizes multiple distinct properties like
  plausibility, diversity and consistency. Results on five difficult
  question-answering datasets StrategyQA, QuaRel, OpenBookQA, NumerSense and QASC
  show that not only does MaRio improve task accuracy, but it also improves the
  self-rationalization quality of small LMs across the aforementioned axes better
  than a supervised fine-tuning (SFT) baseline. Extensive human evaluations
  confirm that MaRio rationales are preferred vs. SFT rationales, as well as
  qualitative improvements in plausibility and consistency.
paper_authors._id: 6549c6d51718116e3508e005
paper_authors.name: Sahana Ramnath
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2023-12-04T07:31:46.669Z
paper_authors._id: 656d803c6f39f1565858aa9c
paper_authors.avatarUrl: /avatars/d6e30599cf32bd13e0b96a85cff3f9ad.svg
paper_authors.fullname: Sahana Ramnath
paper_authors.type: user
paper_authors.user: SahanaRamnath
paper_authors._id: 6549c6d51718116e3508e006
paper_authors.name: Brihi Joshi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-07T17:19:37.667Z
paper_authors._id: 64ee4d611c9470d52f5d8aaf
paper_authors.avatarUrl: /avatars/223617a9933c833e7ea424bfcc3883c0.svg
paper_authors.fullname: Brihi Joshi
paper_authors.type: user
paper_authors.user: brihijoshi
paper_authors._id: 6549c6d51718116e3508e007
paper_authors.name: Skyler Hallinan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-07T17:19:44.231Z
paper_authors._id: 637d38fcb8e573d75bedc033
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/637d38fcb8e573d75bedc033/MbzfbMBGN2ajH7oZtitcB.png
paper_authors.fullname: Skyler Hallinan
paper_authors.type: user
paper_authors.user: hallisky
paper_authors._id: 6549c6d51718116e3508e008
paper_authors.name: Ximing Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-07T17:19:50.642Z
paper_authors._id: 640928bd3461c51cf7378707
paper_authors.avatarUrl: /avatars/429d1ba8327df82a8f8360c0f55e199c.svg
paper_authors.fullname: Ximing Lu
paper_authors.type: user
paper_authors.user: Ximing
paper_authors._id: 6549c6d51718116e3508e009
paper_authors.name: Liunian Harold Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-07T17:19:57.434Z
paper_authors._id: 60b7c68c105296d03a0530ca
paper_authors.avatarUrl: /avatars/e5823c7a8f7b77b4d75125192c1adc66.svg
paper_authors.fullname: Liunian Harold Li
paper_authors.type: user
paper_authors.user: harold
paper_authors._id: 6549c6d51718116e3508e00a
paper_authors.name: Aaron Chan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-07T17:20:05.715Z
paper_authors._id: 6426440722bddcea3d27ee8a
paper_authors.avatarUrl: /avatars/daac33a79a6b35c36154274b6f109f8c.svg
paper_authors.fullname: Aaron_Chan
paper_authors.type: user
paper_authors.user: AaronChan
paper_authors._id: 6549c6d51718116e3508e00b
paper_authors.name: Jack Hessel
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-07T17:20:13.003Z
paper_authors._id: 625498644f4edf771516b2cb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1664497251391-625498644f4edf771516b2cb.jpeg
paper_authors.fullname: Jack Hessel
paper_authors.type: user
paper_authors.user: jmhessel
paper_authors._id: 6549c6d51718116e3508e00c
paper_authors.name: Yejin Choi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-07T17:20:20.082Z
paper_authors._id: 64d42729f63b01b7f676b176
paper_authors.avatarUrl: /avatars/52e54bdd6a1fb6c774a40cd70f3d7925.svg
paper_authors.fullname: Yejin Choi
paper_authors.type: user
paper_authors.user: yejinchoinka
paper_authors._id: 6549c6d51718116e3508e00d
paper_authors.name: Xiang Ren
paper_title: Tailoring Self-Rationalizers with Multi-Reward Distillation
paper_summary: Large language models (LMs) are capable of generating free-text rationales to
  aid question answering. However, prior work 1) suggests that useful
  self-rationalization is emergent only at significant scales (e.g., 175B
  parameter GPT-3); and 2) focuses largely on downstream performance, ignoring
  the semantics of the rationales themselves, e.g., are they faithful, true, and
  helpful for humans? In this work, we enable small-scale LMs (approx. 200x
  smaller than GPT-3) to generate rationales that not only improve downstream
  task performance, but are also more plausible, consistent, and diverse,
  assessed both by automatic and human evaluation. Our method, MaRio
  (Multi-rewArd RatIOnalization), is a multi-reward conditioned
  self-rationalization algorithm that optimizes multiple distinct properties like
  plausibility, diversity and consistency. Results on five difficult
  question-answering datasets StrategyQA, QuaRel, OpenBookQA, NumerSense and QASC
  show that not only does MaRio improve task accuracy, but it also improves the
  self-rationalization quality of small LMs across the aforementioned axes better
  than a supervised fine-tuning (SFT) baseline. Extensive human evaluations
  confirm that MaRio rationales are preferred vs. SFT rationales, as well as
  qualitative improvements in plausibility and consistency.
paper_ai_keywords: Large language models (LMs) free-text rationales self-rationalization parameter GPT-3 small-scale LMs MaRio (Multi-rewArd RatIOnalization) multi-reward conditioned self-rationalization algorithm plausibility diversity consistency automatic evaluation human evaluation StrategyQA QuaRel OpenQuaQA NumerSense QASC task accuracy self-rationalization quality supervised fine-tuning (SFT) baseline