ARCHead: Activation-Metric Residual Correction for Large Language Model Output Heads
Paper • 2608.02703 • Published
open/ acc
community
AI & ML interests
None defined yet.
Recent Activity
View all activity
suayptalha
authored a
paper 10 minutes ago
Post
122
Hey! We've released PrunaVAED, a drop-in replacement decoder for LTX-2.3
- ~1.7-2.1x faster decoder & ~50% lower peak VRAM
- Near-original visual quality & no changes to the latent encoding
Check it here: PrunaAI/PrunaVAED
- ~1.7-2.1x faster decoder & ~50% lower peak VRAM
- Near-original visual quality & no changes to the latent encoding
Check it here: PrunaAI/PrunaVAED
Post
4002
who's working on an NVFP4 version of Kimi-K3?
- 4 replies
RiverRider
posted an update 11 days ago
Post
163
Train Once, Read Everywhere
Paper title:
Train Once, Read Everywhere: Substrate Invariance of the Linearly Readable Structure in Frozen Language Models
Paper URL:
https://github.com/space-bacon/SRT/blob/main/arxiv_program/paper.md
Repository URL:
https://github.com/space-bacon/SRT
The consolidated findings of the SRT research program are now available.
The program treats frozen production-scale language models as substrates whose internal states carry structure that small, inspectable instruments can read. Results include:
- A ~12 M-parameter adapter that surfaces per-token semiotic signals from a frozen 7 B backbone with zero cross-entropy degradation
- An activation verbalizer that recovers text from single hidden states up to a calibrated paraphrase ceiling
- Linear readout ports spanning dense 3 B models to 94-layer 235 B mixture-of-experts models
- A 22 MB linear head that gives a frozen multimodal chat model image-to-text retrieval performance matching fully trained 2018 dual encoders on the COCO benchmark
The central claim is substrate invariance. The readable structure is a stable property of the model class. A head trained once on one host reads, with no retraining and at most a 42 KB recalibration, across:
- Hosts ten times smaller (31 B → 3 B)
- 4-bit weight precision
- Entirely different silicon and kernels (CUDA/bf16 to Apple Silicon/MLX-Q4)
Deployment tiers differ in latency and cost, never in capability.
All instruments, measurement protocols, invariance evidence, negative results, and artifacts are in the repository.
Paper title:
Train Once, Read Everywhere: Substrate Invariance of the Linearly Readable Structure in Frozen Language Models
Paper URL:
https://github.com/space-bacon/SRT/blob/main/arxiv_program/paper.md
Repository URL:
https://github.com/space-bacon/SRT
The consolidated findings of the SRT research program are now available.
The program treats frozen production-scale language models as substrates whose internal states carry structure that small, inspectable instruments can read. Results include:
- A ~12 M-parameter adapter that surfaces per-token semiotic signals from a frozen 7 B backbone with zero cross-entropy degradation
- An activation verbalizer that recovers text from single hidden states up to a calibrated paraphrase ceiling
- Linear readout ports spanning dense 3 B models to 94-layer 235 B mixture-of-experts models
- A 22 MB linear head that gives a frozen multimodal chat model image-to-text retrieval performance matching fully trained 2018 dual encoders on the COCO benchmark
The central claim is substrate invariance. The readable structure is a stable property of the model class. A head trained once on one host reads, with no retraining and at most a 42 KB recalibration, across:
- Hosts ten times smaller (31 B → 3 B)
- 4-bit weight precision
- Entirely different silicon and kernels (CUDA/bf16 to Apple Silicon/MLX-Q4)
Deployment tiers differ in latency and cost, never in capability.
All instruments, measurement protocols, invariance evidence, negative results, and artifacts are in the repository.
- 1 reply
osanseviero
authored a
paper 27 days ago
osanseviero
submitted a
paper to Daily Papers 28 days ago
suayptalha
authored a
paper about 1 month ago
RiverRider
posted an update about 1 month ago
Post
165
🔮 Gemma-4-31B-it SRT-Sunstone
A read-out that reads images — trained only on words.
A 12.3M side-channel head on a frozen google/gemma-4-31B-it, taught meaning from text alone — to separate discourse communities in the residual stream. It never saw a single picture in training. Hand it a picture and it names what the image means, with zero image training.
Cross-modal transfer. Give it a photo and it lands next to the right words: bicycle → bicycle, rose → flower, dog → pet. It groups images by what they mean, not how they look: image→referent kNN 0.64 against chance 0.10.
Why this is a semiotic read-out, not an image classifier. A classifier is trained on labelled images and learns a fixed map from pixels to a closed label set; it only knows the labels it was shown. This read-out is different in kind. It never saw an image in training — it was trained only on text, to separate discourse communities in the residual stream of a frozen gemma-4-31B. It can interpret a picture because gemma-4 already fuses image and word into one representational stream, and the read-out taps the shared interpretant: the meaning a sign carries, whatever form it arrived in. So it does not classify the image. It tells you what the image means to a system that learned meaning from words — a transfer across modality, from a head that was never cross-trained. That is the result.
Each picture gets two readings: the words it means — the load-bearing evidence — and the discourse it evokes, the nearest of 35 communities it learned from text. Read the second as a flavour, not a category: cars into the automotive community, deer and mushrooms into gardening, cats and dogs into the cozy-domestic communities. Never a class label.
Scored offline through a frozen google/gemma-4-31B-it (62.5 GB, too large to run live)
Try it: RiverRider/srt-sunstone
Model: RiverRider/Gemma-4-31B-it-SRT-Sunstone
Code: https://github.com/space-bacon/SRT
A read-out that reads images — trained only on words.
A 12.3M side-channel head on a frozen google/gemma-4-31B-it, taught meaning from text alone — to separate discourse communities in the residual stream. It never saw a single picture in training. Hand it a picture and it names what the image means, with zero image training.
Cross-modal transfer. Give it a photo and it lands next to the right words: bicycle → bicycle, rose → flower, dog → pet. It groups images by what they mean, not how they look: image→referent kNN 0.64 against chance 0.10.
Why this is a semiotic read-out, not an image classifier. A classifier is trained on labelled images and learns a fixed map from pixels to a closed label set; it only knows the labels it was shown. This read-out is different in kind. It never saw an image in training — it was trained only on text, to separate discourse communities in the residual stream of a frozen gemma-4-31B. It can interpret a picture because gemma-4 already fuses image and word into one representational stream, and the read-out taps the shared interpretant: the meaning a sign carries, whatever form it arrived in. So it does not classify the image. It tells you what the image means to a system that learned meaning from words — a transfer across modality, from a head that was never cross-trained. That is the result.
Each picture gets two readings: the words it means — the load-bearing evidence — and the discourse it evokes, the nearest of 35 communities it learned from text. Read the second as a flavour, not a category: cars into the automotive community, deer and mushrooms into gardening, cats and dogs into the cozy-domestic communities. Never a class label.
Scored offline through a frozen google/gemma-4-31B-it (62.5 GB, too large to run live)
Try it: RiverRider/srt-sunstone
Model: RiverRider/Gemma-4-31B-it-SRT-Sunstone
Code: https://github.com/space-bacon/SRT
jeffboudier
posted an update about 1 month ago
Post
300
@alvarobartt published a step-by-step guide to deploy zai-org/GLM-5.2 on AMD GPUs, using newly released features in Microsoft Foundry. The FP8 model fits on a single node of MI300X GPUs, which cuts the bill in half vs. H100.
https://alvarobartt.com/goal-glm-5.2-on-foundry/
https://alvarobartt.com/goal-glm-5.2-on-foundry/
bofenghuang
authored 2
papers about 1 month ago
RiverRider
posted an update about 2 months ago
Post
277
SRT Showcase: Watch a Frozen Model Think, Token by Token
A frozen Qwen-2.5-7B now narrates its own interpretation in real time. SRT Showcase is the most complete public demonstration of computational semiotics to date, running the backbone with the SRT Adapter and Activation Verbalizer. As the model generates, every token is tinted by its predictive effort, and at the highest-effort positions the Verbalizer decodes the hidden state directly into natural language. You see what the model is representing at the exact moment its computation is most active.
Every verbalization is validated, not asserted. Each decoded thought is re-encoded and compared back to the original hidden state, and the reconstruction closely approximates it. The "this is what the model was thinking" claim carries its own fidelity badge. This is grounded introspection, not plausible narration.
The Showcase goes further than the trace. An A/B panel runs the same prompt with SRT injection on and off under an identical seed, so the side-channel's effect is directly observable. A curated gallery walks through confident recall, false premises, misconceptions, reasoning pivots, genuine uncertainty, and safety boundaries. Live entropy and divergence meters track the crystallization process token by token, with per-layer traces and reflexivity estimates on hover.
None of the backbone weights are touched. The entire mechanism is a lightweight reflexive layer over a frozen model, which is why the same read-out heads already port from Qwen-2.5-7B up to a 235B Mixture of Experts. Frozen models can now be verbalized in real time. No retraining. No fine-tuning. No black box.
First request is a brief cold start while ZeroGPU acquires a GPU. Bring your own prompt.
Try it: RiverRider/srt-showcase
Repository: https://github.com/space-bacon/SRT
A frozen Qwen-2.5-7B now narrates its own interpretation in real time. SRT Showcase is the most complete public demonstration of computational semiotics to date, running the backbone with the SRT Adapter and Activation Verbalizer. As the model generates, every token is tinted by its predictive effort, and at the highest-effort positions the Verbalizer decodes the hidden state directly into natural language. You see what the model is representing at the exact moment its computation is most active.
Every verbalization is validated, not asserted. Each decoded thought is re-encoded and compared back to the original hidden state, and the reconstruction closely approximates it. The "this is what the model was thinking" claim carries its own fidelity badge. This is grounded introspection, not plausible narration.
The Showcase goes further than the trace. An A/B panel runs the same prompt with SRT injection on and off under an identical seed, so the side-channel's effect is directly observable. A curated gallery walks through confident recall, false premises, misconceptions, reasoning pivots, genuine uncertainty, and safety boundaries. Live entropy and divergence meters track the crystallization process token by token, with per-layer traces and reflexivity estimates on hover.
None of the backbone weights are touched. The entire mechanism is a lightweight reflexive layer over a frozen model, which is why the same read-out heads already port from Qwen-2.5-7B up to a 235B Mixture of Experts. Frozen models can now be verbalized in real time. No retraining. No fine-tuning. No black box.
First request is a brief cold start while ZeroGPU acquires a GPU. Bring your own prompt.
Try it: RiverRider/srt-showcase
Repository: https://github.com/space-bacon/SRT
cfahlgren1
submitted a
paper to Daily Papers about 2 months ago
RiverRider
posted an update about 2 months ago
Post
208
Real-Time Introspection for Qwen3 235B
An SRT adapter is now available for Qwen3 235B, a large open Mixture of Experts model. This adapter adds a lightweight monitoring layer that provides signals about the models internal state during generation without modifying any of its original weights.
The adapter can indicate whether the model is operating in a more stable or more divergent internal condition. It also estimates how self referential the models processing is at each step and tracks shifts in representation across layers. In addition it produces information about the kind of discourse style the model appears to be using at any moment.
This is the first adapter of its kind released on a model of this scale with fully open weights. It was trained in a read only configuration so the base model remains unchanged. The strongest result is in regime detection where the adapter achieved very high accuracy in distinguishing between different internal operating states on held out data.
For researchers focused on model interpretability this provides a practical way to inspect internal dynamics at frontier scale. For those working in semiotics it offers empirical access to how processes of meaning and interpretation unfold inside large contemporary language models.
Model: RiverRider/srt-adapter-qwen3-235b
Repository and documentation: https://github.com/space-bacon/SRT
An SRT adapter is now available for Qwen3 235B, a large open Mixture of Experts model. This adapter adds a lightweight monitoring layer that provides signals about the models internal state during generation without modifying any of its original weights.
The adapter can indicate whether the model is operating in a more stable or more divergent internal condition. It also estimates how self referential the models processing is at each step and tracks shifts in representation across layers. In addition it produces information about the kind of discourse style the model appears to be using at any moment.
This is the first adapter of its kind released on a model of this scale with fully open weights. It was trained in a read only configuration so the base model remains unchanged. The strongest result is in regime detection where the adapter achieved very high accuracy in distinguishing between different internal operating states on held out data.
For researchers focused on model interpretability this provides a practical way to inspect internal dynamics at frontier scale. For those working in semiotics it offers empirical access to how processes of meaning and interpretation unfold inside large contemporary language models.
Model: RiverRider/srt-adapter-qwen3-235b
Repository and documentation: https://github.com/space-bacon/SRT
RiverRider
posted an update about 2 months ago
Post
5142
ATTENTION: The SRT-Introspect framework moves past surface-level output commentary by supplying real-time natural language interpretations of a model’s latent states. These verbalizations are validated, not merely asserted, through a round-trip reconstruction procedure. Natural language descriptions derived from hidden activations are passed through an encoder that reconstructs the corresponding activation vector; the recovered vector closely approximates the original. High reconstruction fidelity indicates that the verbalizations encode genuine structural information about the internal state rather than offering plausible but ungrounded speculation.
This validated introspection converts what has often remained a theoretical or post-hoc exercise into a practical instrument for auditing model behavior, diagnosing failure modes, and providing high-level semantic guidance—all without modifying the base model or incurring the costs of fine-tuning. Because the mechanism operates on frozen configurations, it can be applied to production systems where any change to weights or architecture is undesirable. Thank you for your attention.
Run a trace: RiverRider/srt-introspect
Repo: https://github.com/space-bacon/SRT
This validated introspection converts what has often remained a theoretical or post-hoc exercise into a practical instrument for auditing model behavior, diagnosing failure modes, and providing high-level semantic guidance—all without modifying the base model or incurring the costs of fine-tuning. Because the mechanism operates on frozen configurations, it can be applied to production systems where any change to weights or architecture is undesirable. Thank you for your attention.
Run a trace: RiverRider/srt-introspect
Repo: https://github.com/space-bacon/SRT
RiverRider
posted an update about 2 months ago
Post
3631
This is not a pipe.
Everyone is born a semiotician, no one is born knowing it. Go easy on yourself (and me) for not understanding this yet.
Computational semiotics is now an empirical study.
LLMs are not proto-minds. They are verifiably semiotic infrastructure.
This repository (or attached demo) can show you, in real time, how any frozen model (Qwen for demo) arrives at any answer by reading its latent states directly during generation.
Any questions?
RiverRider/srt-introspect
Repo:
https://github.com/space-bacon/SRT
Grok insist my intro is condescending … This is certainly true, as is the statement in my condescended opinion. I expect heat for it, let’s think this through?
Everyone is born a semiotician, no one is born knowing it. Go easy on yourself (and me) for not understanding this yet.
Computational semiotics is now an empirical study.
LLMs are not proto-minds. They are verifiably semiotic infrastructure.
This repository (or attached demo) can show you, in real time, how any frozen model (Qwen for demo) arrives at any answer by reading its latent states directly during generation.
Any questions?
RiverRider/srt-introspect
Repo:
https://github.com/space-bacon/SRT
Grok insist my intro is condescending … This is certainly true, as is the statement in my condescended opinion. I expect heat for it, let’s think this through?
RiverRider
posted an update 2 months ago
Post
204
Words do not have determined meanings.
The vocabulary itself is reflexive. It is self-referential, looping back into its own structure rather than anchoring in fixed reality. What we treat as stable meaning is continually reconstituted in the act of using it. The observers own interpretations molding each word like clay with every utterance.
All large language models to date treat words otherwise. At the moment of softmax crystallization they determine the meaning of every token. Probabilities collapse into a single output. Meaning is not found. It is fixed, token by token, in that final distribution.
SRT-Introspect is a demo for observing what Qwen actually thinks at the points of highest effort. It surfaces the internal representations during generation, making visible the reflexive vocabulary at work and the precise crystallization process: the weights, the assumptions, the decisions that resolve ambiguity into output. This includes accounting for anisotropy collapse in hidden states by centering representations around the layer-mean before analysis.
Feel free to comment your prompts
RiverRider/srt-introspect
Repo
https://github.com/space-bacon/SRT
The vocabulary itself is reflexive. It is self-referential, looping back into its own structure rather than anchoring in fixed reality. What we treat as stable meaning is continually reconstituted in the act of using it. The observers own interpretations molding each word like clay with every utterance.
All large language models to date treat words otherwise. At the moment of softmax crystallization they determine the meaning of every token. Probabilities collapse into a single output. Meaning is not found. It is fixed, token by token, in that final distribution.
SRT-Introspect is a demo for observing what Qwen actually thinks at the points of highest effort. It surfaces the internal representations during generation, making visible the reflexive vocabulary at work and the precise crystallization process: the weights, the assumptions, the decisions that resolve ambiguity into output. This includes accounting for anisotropy collapse in hidden states by centering representations around the layer-mean before analysis.
Feel free to comment your prompts
RiverRider/srt-introspect
Repo
https://github.com/space-bacon/SRT
- 1 reply
RiverRider
posted an update 2 months ago
Post
2823
This is not the end of words. It is the end of pretending their meanings are determined.
Meaning Forks. SRT detects it.
Paste any text to identify contested terms
RiverRider/srt-introspect
Try any prompt (attached link) to see exactly what an LLM is thinking at every meaningful step of its answer
RiverRider/srt-introspect
Repository
https://github.com/space-bacon/SRT
Paper
https://github.com/space-bacon/SRT/blob/main/paper_nla.md
Explainer
https://github.com/space-bacon/SRT/blob/main/docs/EXPLAINERS.md
Meaning Forks. SRT detects it.
Paste any text to identify contested terms
RiverRider/srt-introspect
Try any prompt (attached link) to see exactly what an LLM is thinking at every meaningful step of its answer
RiverRider/srt-introspect
Repository
https://github.com/space-bacon/SRT
Paper
https://github.com/space-bacon/SRT/blob/main/paper_nla.md
Explainer
https://github.com/space-bacon/SRT/blob/main/docs/EXPLAINERS.md
Ritvik19
authored a
paper 2 months ago
RiverRider
posted an update 2 months ago
Post
4849
SRT-introspect: Live Token-by-Token Readout of LLM Internal Reasoning
I have released SRT-introspect, a new public demonstration that makes the hidden reasoning process of a frozen large language model visible in real time.
The interface runs a Qwen-2.5-7B backbone equipped with the SRT Adapter and Activation Verbalizer. As the model generates each token, the system continuously measures divergence across attention heads, identifies high-signal moments, and translates the corresponding hidden-state object representations into natural-language verbalizations. You see exactly what the model is internally representing at the precise points where its computation is most active, complete with divergence scores, reflexivity estimates, and per-layer traces.
This is not a summary of the final output. It is a direct window into the model’s latent conceptual landscape, showing the dominant training-data attractors that activate even when the prompt asks for first-principles reasoning. The adaptive scheduler concentrates verbalizations precisely where the real internal work occurs, turning what used to be opaque black-box generation into observable, analyzable data.
The result is the clearest public demonstration yet that modern LLMs possess a rich, structured semiotic infrastructure that can now be audited without retraining or fine-tuning.
Try it:
RiverRider/srt-introspect
I have released SRT-introspect, a new public demonstration that makes the hidden reasoning process of a frozen large language model visible in real time.
The interface runs a Qwen-2.5-7B backbone equipped with the SRT Adapter and Activation Verbalizer. As the model generates each token, the system continuously measures divergence across attention heads, identifies high-signal moments, and translates the corresponding hidden-state object representations into natural-language verbalizations. You see exactly what the model is internally representing at the precise points where its computation is most active, complete with divergence scores, reflexivity estimates, and per-layer traces.
This is not a summary of the final output. It is a direct window into the model’s latent conceptual landscape, showing the dominant training-data attractors that activate even when the prompt asks for first-principles reasoning. The adaptive scheduler concentrates verbalizations precisely where the real internal work occurs, turning what used to be opaque black-box generation into observable, analyzable data.
The result is the clearest public demonstration yet that modern LLMs possess a rich, structured semiotic infrastructure that can now be audited without retraining or fine-tuning.
Try it:
RiverRider/srt-introspect