Predicting Rewards Alongside Tokens: Non-disruptive Parameter Insertion
  for Efficient Inference Intervention in Large Language Model
Transformer-based large language models (LLMs) exhibit limitations such as
generating unsafe responses, unreliable reasoning, etc. Existing inference
intervention approaches attempt to mitigate these issues by finetuning
additional models to produce calibration signals (such as rewards) that guide
the LLM's decoding process. However, this solution introduces substantial time
and space overhead due to the separate models required. This work proposes
Non-disruptive parameters insertion (Otter), inserting extra parameters into
the transformer architecture to predict calibration signals along with the
original LLM output. Otter offers state-of-the-art performance on multiple
demanding tasks while saving up to 86.5\% extra space and 98.5\% extra time.
Furthermore, Otter seamlessly integrates with existing inference engines,
requiring only a one-line code change, and the original model response remains
accessible after the parameter insertion. Our code is publicly available at
https://github.com/chenhan97/Otter
66c5786109ea098de4af206e
Chenhan Yuan
66c5786109ea098de4af206f
Fei Huang
66c5786109ea098de4af2070
Ru Peng
66c5786109ea098de4af2071
Keming Lu
66c5786109ea098de4af2072
Bowen Yu
66c5786109ea098de4af2073
Chang Zhou
66c5786109ea098de4af2074
Jingren Zhou
Predicting Rewards Alongside Tokens: Non-disruptive Parameter Insertion
  for Efficient Inference Intervention in Large Language Model
Transformer-based large language models (LLMs) exhibit limitations such as
generating unsafe responses, unreliable reasoning, etc. Existing inference
intervention approaches attempt to mitigate these issues by finetuning
additional models to produce calibration signals (such as rewards) that guide
the LLM's decoding process. However, this solution introduces substantial time
and space overhead due to the separate models required. This work proposes
Non-disruptive parameters insertion (Otter), inserting extra parameters into
the transformer architecture to predict calibration signals along with the
original LLM output. Otter offers state-of-the-art performance on multiple
demanding tasks while saving up to 86.5\% extra space and 98.5\% extra time.
Furthermore, Otter seamlessly integrates with existing inference engines,
requiring only a one-line code change, and the original model response remains
accessible after the parameter insertion. Our code is publicly available at
https://github.com/chenhan97/Otter
Non-disruptive parameter insertion into transformer architecture enhances large language models by predicting calibration signals without additional overhead.
Transformer-based large language models LLMs inference intervention calibration signals reward decoding process non-disruptive parameters insertion Otter transformer architecture space overhead time overhead inference engines