title: Predicting Rewards Alongside Tokens: Non-disruptive Parameter Insertion
    for Efficient Inference Intervention in Large Language Model
summary: Transformer-based large language models (LLMs) exhibit limitations such as
  generating unsafe responses, unreliable reasoning, etc. Existing inference
  intervention approaches attempt to mitigate these issues by finetuning
  additional models to produce calibration signals (such as rewards) that guide
  the LLM's decoding process. However, this solution introduces substantial time
  and space overhead due to the separate models required. This work proposes
  Non-disruptive parameters insertion (Otter), inserting extra parameters into
  the transformer architecture to predict calibration signals along with the
  original LLM output. Otter offers state-of-the-art performance on multiple
  demanding tasks while saving up to 86.5\% extra space and 98.5\% extra time.
  Furthermore, Otter seamlessly integrates with existing inference engines,
  requiring only a one-line code change, and the original model response remains
  accessible after the parameter insertion. Our code is publicly available at
  https://github.com/chenhan97/Otter
paper_authors._id: 66c5786109ea098de4af206e
paper_authors.name: Chenhan Yuan
paper_authors._id: 66c5786109ea098de4af206f
paper_authors.name: Fei Huang
paper_authors._id: 66c5786109ea098de4af2070
paper_authors.name: Ru Peng
paper_authors._id: 66c5786109ea098de4af2071
paper_authors.name: Keming Lu
paper_authors._id: 66c5786109ea098de4af2072
paper_authors.name: Bowen Yu
paper_authors._id: 66c5786109ea098de4af2073
paper_authors.name: Chang Zhou
paper_authors._id: 66c5786109ea098de4af2074
paper_authors.name: Jingren Zhou
paper_title: Predicting Rewards Alongside Tokens: Non-disruptive Parameter Insertion
    for Efficient Inference Intervention in Large Language Model
paper_summary: Transformer-based large language models (LLMs) exhibit limitations such as
  generating unsafe responses, unreliable reasoning, etc. Existing inference
  intervention approaches attempt to mitigate these issues by finetuning
  additional models to produce calibration signals (such as rewards) that guide
  the LLM's decoding process. However, this solution introduces substantial time
  and space overhead due to the separate models required. This work proposes
  Non-disruptive parameters insertion (Otter), inserting extra parameters into
  the transformer architecture to predict calibration signals along with the
  original LLM output. Otter offers state-of-the-art performance on multiple
  demanding tasks while saving up to 86.5\% extra space and 98.5\% extra time.
  Furthermore, Otter seamlessly integrates with existing inference engines,
  requiring only a one-line code change, and the original model response remains
  accessible after the parameter insertion. Our code is publicly available at
  https://github.com/chenhan97/Otter
paper_ai_summary: Non-disruptive parameter insertion into transformer architecture enhances large language models by predicting calibration signals without additional overhead.
paper_ai_keywords: Transformer-based large language models LLMs inference intervention calibration signals reward decoding process non-disruptive parameters insertion Otter transformer architecture space overhead time overhead inference engines