title: Almost Surely Safe Alignment of Large Language Models at Inference-Time
summary: Even highly capable large language models (LLMs) can produce biased or unsafe
  responses, and alignment techniques, such as RLHF, aimed at mitigating this
  issue, are expensive and prone to overfitting as they retrain the LLM. This
  paper introduces a novel inference-time alignment approach that ensures LLMs
  generate safe responses almost surely, i.e., with a probability approaching
  one. We achieve this by framing the safe generation of inference-time responses
  as a constrained Markov decision process within the LLM's latent space.
  Crucially, we augment a safety state that tracks the evolution of safety
  constraints and enables us to demonstrate formal safety guarantees upon solving
  the MDP in the latent space. Building on this foundation, we propose
  InferenceGuard, a practical implementation that safely aligns LLMs without
  modifying the model weights. Empirically, we demonstrate InferenceGuard
  effectively balances safety and task performance, outperforming existing
  inference-time alignment methods in generating safe and aligned responses.
paper_authors._id: 67a1e729a6c7d65cad72b3d7
paper_authors.name: Xiaotong Ji
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-04T17:40:16.111Z
paper_authors._id: 6682989afc0e69f80acf2845
paper_authors.avatarUrl: /avatars/35d48738965fe21fdd79198a17d6c8cc.svg
paper_authors.fullname: jixiaotong
paper_authors.type: user
paper_authors.user: xiaotong9515
paper_authors._id: 67a1e729a6c7d65cad72b3d8
paper_authors.name: Shyam Sundhar Ramesh
paper_authors._id: 67a1e729a6c7d65cad72b3d9
paper_authors.name: Matthieu Zimmer
paper_authors._id: 67a1e729a6c7d65cad72b3da
paper_authors.name: Ilija Bogunovic
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-04T17:39:45.148Z
paper_authors._id: 65fd58a41a58717c800d3650
paper_authors.avatarUrl: /avatars/9dce361a0417465116c816abdf53e916.svg
paper_authors.fullname: Bogunovic
paper_authors.type: user
paper_authors.user: ilijabogunovic
paper_authors._id: 67a1e729a6c7d65cad72b3db
paper_authors.name: Jun Wang
paper_authors._id: 67a1e729a6c7d65cad72b3dc
paper_authors.name: Haitham Bou Ammar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-04T17:39:59.356Z
paper_authors._id: 631c375768f7da9ad2496bf6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/631c375768f7da9ad2496bf6/1sDOoecA6e1v_hn_VAgUq.jpeg
paper_authors.fullname: Haitham Bou Ammar
paper_authors.type: user
paper_authors.user: hba123
paper_title: Almost Surely Safe Alignment of Large Language Models at Inference-Time
paper_summary: Even highly capable large language models (LLMs) can produce biased or unsafe
  responses, and alignment techniques, such as RLHF, aimed at mitigating this
  issue, are expensive and prone to overfitting as they retrain the LLM. This
  paper introduces a novel inference-time alignment approach that ensures LLMs
  generate safe responses almost surely, i.e., with a probability approaching
  one. We achieve this by framing the safe generation of inference-time responses
  as a constrained Markov decision process within the LLM's latent space.
  Crucially, we augment a safety state that tracks the evolution of safety
  constraints and enables us to demonstrate formal safety guarantees upon solving
  the MDP in the latent space. Building on this foundation, we propose
  InferenceGuard, a practical implementation that safely aligns LLMs without
  modifying the model weights. Empirically, we demonstrate InferenceGuard
  effectively balances safety and task performance, outperforming existing
  inference-time alignment methods in generating safe and aligned responses.
paper_ai_summary: A novel inference-time alignment approach ensures large language models generate safe responses with high probability using a constrained Markov decision process in the latent space.
paper_ai_keywords: large language models LLMs alignment techniques RLHF inference-time alignment constrained Markov decision process latent space safety state safety constraints formal safety guarantees InferenceGuard