Steering Conceptual Bias via Transformer Latent-Subspace Activation
This work examines whether activating latent subspaces in language models
(LLMs) can steer scientific code generation toward a specific programming
language. Five causal LLMs were first evaluated on scientific coding prompts to
quantify their baseline bias among four programming languages. A static
neuron-attribution method, perturbing the highest activated MLP weight for a
C++ or CPP token, proved brittle and exhibited limited generalization across
prompt styles and model scales. To address these limitations, a
gradient-refined adaptive activation steering framework (G-ACT) was developed:
per-prompt activation differences are clustered into a small set of steering
directions, and lightweight per-layer probes are trained and refined online to
select the appropriate steering vector. In LLaMA-3.2 3B, this approach reliably
biases generation towards the CPP language by increasing the average probe
classification accuracy by 15% and the early layers (0-6) improving the probe
classification accuracy by 61.5% compared to the standard ACT framework. For
LLaMA-3.3 70B, where attention-head signals become more diffuse, targeted
injections at key layers still improve language selection. Although per-layer
probing introduces a modest inference overhead, it remains practical by
steering only a subset of layers and enables reproducible model behavior. These
results demonstrate a scalable, interpretable and efficient mechanism for
concept-level control for practical agentic systems.
685a176c0e4ad7e21975858b
Vansh Sharma
claimed_verified
2025-06-24T08:08:13.705Z
665ca52693b937f83e1b970e
/avatars/a5b202dadb31fb51688a0e5b7a482727.svg
Vansh Sharma
user
vanshs1
685a176c0e4ad7e21975858c
Venkat Raman
Steering Conceptual Bias via Transformer Latent-Subspace Activation
This work examines whether activating latent subspaces in language models
(LLMs) can steer scientific code generation toward a specific programming
language. Five causal LLMs were first evaluated on scientific coding prompts to
quantify their baseline bias among four programming languages. A static
neuron-attribution method, perturbing the highest activated MLP weight for a
C++ or CPP token, proved brittle and exhibited limited generalization across
prompt styles and model scales. To address these limitations, a
gradient-refined adaptive activation steering framework (G-ACT) was developed:
per-prompt activation differences are clustered into a small set of steering
directions, and lightweight per-layer probes are trained and refined online to
select the appropriate steering vector. In LLaMA-3.2 3B, this approach reliably
biases generation towards the CPP language by increasing the average probe
classification accuracy by 15% and the early layers (0-6) improving the probe
classification accuracy by 61.5% compared to the standard ACT framework. For
LLaMA-3.3 70B, where attention-head signals become more diffuse, targeted
injections at key layers still improve language selection. Although per-layer
probing introduces a modest inference overhead, it remains practical by
steering only a subset of layers and enables reproducible model behavior. These
results demonstrate a scalable, interpretable and efficient mechanism for
concept-level control for practical agentic systems.
A gradient-refined adaptive activation steering framework enables LLMs to reliably generate scientific code in a specific programming language by selectively steering latent subspaces.
latent subspaces language models (LLMs) causal LLMs scientific coding prompts neuron-attribution method MLP weight gradient-refined adaptive activation steering framework (G-ACT) steering directions per-layer probes probe classification accuracy attention-head signals concept-level control