AlignGuard-LoRA: Alignment-Preserving Fine-Tuning via Fisher-Guided
  Decomposition and Riemannian-Geodesic Collision Regularization
Low-rank adaptation (LoRA) has become a standard tool for efficiently
fine-tuning large language models (LLMs). Yet, even minor LoRA updates can
induce alignment drift, weakening safety and behavioral constraints through
entangled parameter changes. To address this, we propose AlignGuard-LoRA (AGL),
a principled framework for preserving alignment during finetuning. AGL
introduces several key components: a primary task loss for supervision, Fisher
Information Matrix-based regularization to restrict updates in
alignment-sensitive subspaces, and task-specific regularization to stabilize
the integration of new knowledge. We further introduce collision-aware
regularization, blending Riemannian overlap -- which penalizes coordinate-wise
interference -- and geodesic separation -- which encourages disjoint update
geometry. We curate DriftCaps, a targeted diagnostic benchmark of safe and
unsafe prompts designed to quantify alignment drift and safety degradation.
Empirical evaluations show that AGL mitigates alignment drift by up to 50% on
safety-critical benchmarks without degrading downstream task performance.
Comprehensive ablation confirms that each component contributes distinctly to
preserving latent safety behaviors. Finally, we derive and validate a scaling
law for catastrophic forgetting, revealing that AGL flattens post-finetuning
loss escalation while preserving adaptation dynamics. AGL is a structurally
grounded refinement of LoRA, ensuring alignment preservation with minimal
trade-offs. To encourage further exploration and development, we open-source
our implementation.
6892c2a18da45ffb0a2b241e
Amitava Das
6892c2a18da45ffb0a2b241f
Abhilekh Borah
6892c2a18da45ffb0a2b2420
Vinija Jain
6892c2a18da45ffb0a2b2421
Aman Chadha
AlignGuard-LoRA: Alignment-Preserving Fine-Tuning via Fisher-Guided
  Decomposition and Riemannian-Geodesic Collision Regularization
Low-rank adaptation (LoRA) has become a standard tool for efficiently
fine-tuning large language models (LLMs). Yet, even minor LoRA updates can
induce alignment drift, weakening safety and behavioral constraints through
entangled parameter changes. To address this, we propose AlignGuard-LoRA (AGL),
a principled framework for preserving alignment during finetuning. AGL
introduces several key components: a primary task loss for supervision, Fisher
Information Matrix-based regularization to restrict updates in
alignment-sensitive subspaces, and task-specific regularization to stabilize
the integration of new knowledge. We further introduce collision-aware
regularization, blending Riemannian overlap -- which penalizes coordinate-wise
interference -- and geodesic separation -- which encourages disjoint update
geometry. We curate DriftCaps, a targeted diagnostic benchmark of safe and
unsafe prompts designed to quantify alignment drift and safety degradation.
Empirical evaluations show that AGL mitigates alignment drift by up to 50% on
safety-critical benchmarks without degrading downstream task performance.
Comprehensive ablation confirms that each component contributes distinctly to
preserving latent safety behaviors. Finally, we derive and validate a scaling
law for catastrophic forgetting, revealing that AGL flattens post-finetuning
loss escalation while preserving adaptation dynamics. AGL is a structurally
grounded refinement of LoRA, ensuring alignment preservation with minimal
trade-offs. To encourage further exploration and development, we open-source
our implementation.
AlignGuard-LoRA (AGL) is a framework that preserves alignment during fine-tuning of large language models by introducing regularization techniques and a diagnostic benchmark to mitigate alignment drift.
LoRA AlignGuard-LoRA primary task loss Fisher Information Matrix-based regularization task-specific regularization collision-aware regularization Riemannian overlap geodesic separation DriftCaps alignment drift catastrophic forgetting adaptation dynamics