tal feedback, RLHF incorporates human insights, addressing challenges in crafting precise reward
functions and mitigating unintended behaviors. Nonetheless, this feedback often lacks immediacy
as it traditionally integrates during training phases rather than in real-time. Furthermore, corrections
might not necessarily be reflected in specific queries during inference.
With LLMs’ evolution, the notion of inference-time self-correction gained prominence. The dis-
course on self-correction now pivots around whether these advanced models can recognize the ap-
propriateness or accuracy of their outputs and, if needed, provide refined answers (Bai et al., 2022;
Madaan et al., 2023; Welleck et al., 2023, inter alia). For example, in the context of mathematical
reasoning, an LLM might initially solve a complex problem but make an error in one of the calcu-
lation steps. In an ideal self-correction scenario, the model is expected to recognize the potential
LARGELANGUAGEMODELSCANNOTSELF-CORRECT REASONINGYET