SumomoLee/tmp-anima-qwen35-adapter

Anima + Qwen3.5 conditioner — rebuilt vocabulary, concatenated multi-layer fusion

Text-side conditioner for the Anima DiT, trained in two stages against a frozen anima-base-v1.0 with no LoRA.

This is not a drop-in replacement for the previous adapter. Its embedding table is indexed by a rebuilt vocabulary and it expects a 3072-wide input, so the tokenizer bundle and the fusion module in this repo are both required.

Contents

file	what it is
anima-qwen35-conditioner-v3-image-epoch-fp32.safetensors	final, 755.4 MB
anima-qwen35-conditioner-v3-image-epoch-bf16.safetensors	the same, bf16, 377.7 MB
anima-qwen35-conditioner-v3-fp32.safetensors	after the text stage only, 755.4 MB
anima-qwen35-conditioner-v3-bf16.safetensors	the same, bf16, 377.7 MB
t5_bundle/	the rebuilt T5 tokenizer: tokenizer.json, anima_vocab.json, vocab_mapping.json
infer_conditioner.py	runnable reference, reads the architecture from file metadata
config.json, row-init.json, packed-manifest.json	training configuration and provenance

Both pairs share one architecture and one tokenizer; only the weights differ.

The two stages

	text stage	image stage
objective	Sinkhorn OT against the previous conditioner's context	flow matching against the frozen DiT
samples	28,170,368	4,608,495 (one epoch)
steps	7,589	25,416
wall clock	11.2 h	21.4 h
validation	0.391 → 0.045	0.09661 → 0.09596

The image stage moved its own loss by 0.67% over a full epoch, wandering inside 0.0959–0.0961 the whole way with a gradient norm around 0.016. Its starting point was already 0.0966, against 2.298 for the same architecture without the text stage — so the cheap stage had brought the conditioner to where the frozen DiT wants it, and the expensive one had little left to do. Side-by-side samples show no substantial difference, possibly slightly better alignment. The final weights are the image-epoch pair; the text-stage pair is kept because it is what that 0.045 belongs to, and because 11.2 h reproduces it.

188,848,640 parameters: a 126,930,432-parameter adapter body, a 52,766x1024 embedding table, and a 7,885,824-parameter fusion.

Keys carry an adapter. or fusion. prefix. The split training embedding (embed.legacy frozen + embed.new trainable) is merged into a single adapter.embed.weight of legacy[:32100] ++ extension. Every architecture value needed to rebuild the modules is in the file metadata, including the tapped encoder depths, which cannot be inferred from the tensor shapes.

How the pieces fit

text ──► Qwen3.5-2B-Base ──► hidden_states[2,3,6,9,12,21]
                                      │
                                      ▼
                         fusion: per-layer RMSNorm + projection to 512,
                                 concatenated ──► (B, L, 3072)
                                      │
   T5 ids (rebuilt vocab) ──► adapter cross-attention ──► (B, L, 1024) ──► DiT

Vocabulary

Rebuilt from the corpus rather than from Unicode ranges. The predecessor added every CJK/kana codepoint in three ranges — 27,788 single-character rows, 70.0% of which the corpus never used — and every unit was a character, while the T5 ids are the adapter's cross-attention queries, so one unit is one retrieval slot.

	predecessor	this
extension rows	27,788	20,666 (10,562 characters + 10,104 words)
rows the corpus never touches	19,465	0
zh caption tokens/row	381.2	258.1
ja caption tokens/row	499.2	301.6
ja captions truncated at 512	2,338 / 6,000	7
rows containing <unk> (zh tag dictionary)	29,041	0

Word units come from segmenting the corpus with Qwen3.5's own tokenizer, so a query string is also an encoder unit. Characters outside the CJK ranges that the legacy sentencepiece model cannot encode get rows too — · (U+00B7), used in Chinese renderings of foreign names, alone accounted for 27,750 of the 29,041 <unk> rows.

Tap selection

Chosen by a per-layer linear probe over all 25 depths, predicting tags the caption does not spell out (the unrestricted version is dominated by lexical overlap: depth 0, the embedding table itself, scores as well as any transformer layer).

tap set	mean AP
last layer alone [24]	0.2368
[3,8,12,16,20,24] (inherited)	0.3411
[2,3,6,9,12,21] (selected)	0.3602

The last layer is the worst single choice; the early layers are not redundant, and the three languages give near-identical depth profiles.

Fusion

Concatenation, not a weighted sum. With k_proj linear, concatenating per-layer projections gives Σ Wᵢxᵢ — a separate map per layer — where a weighted sum gives Σ aᵢWxᵢ with one shared map. Writing xᵢ = cᵢx_K + rᵢ, the shared-map form can only produce a multiple of Wx_K; the per-layer form keeps Σ Wᵢrᵢ. Those residuals are the whole of the per-layer information and they are small — the taps are near collinear, any linear combination has cosine 0.994 with the last layer — which is why earlier weighted-sum variants were indistinguishable from it.

Distillation objective

Sinkhorn optimal transport between the student's and the teacher's context token distributions, plus a pooled cosine and mean term. The two tokenizers give different sequence lengths, so there is no position map between them and a position-wise MSE is undefined.

Teacher: anima_dense_multilayer_20260915/job-8185-retry1/checkpoint-025416. 70.3% of the adapter body (105 of 118 tensors, 89.2M of 126.9M parameters) was transferred from it; only the 12 cross-attention k/v projections changed width.

Inference

infer_conditioner.py in this repo is a runnable reference. It reads the architecture from the file's own metadata rather than hardcoding it, because two things are silent failures: the tapped encoder depths cannot be recovered from the tensor shapes, and the T5 side needs the t5_bundle/ shipped here -- handing it the predecessor's 59,888-row vocabulary produces ids that index the wrong rows instead of an error.

Check the weights load and the shapes line up, with no GPU and no diffusion model:

python infer_conditioner.py   --weights anima-qwen35-conditioner-v3-bf16.safetensors   --bundle t5_bundle --dry-run
# {"phase": "conditioner_ready", "layers": [2, 3, 6, 9, 12, 21],
#  "adapter_source_dim": 3072, "fusion_output_dim": 3072, "vocab": 52766}
# {"phase": "dry_run_ok", "fused": [1, 8, 3072], "context": [1, 5, 1024]}

Generate:

python infer_conditioner.py   --weights anima-qwen35-conditioner-v3-bf16.safetensors   --bundle t5_bundle --prompt "..." --seed 7 --out sample.png

The flow in full, which is all encode() does:

source = qwen_tokenizer(text, padding="max_length", max_length=512, truncation=True)
target = t5_tokenizer(text,   padding="max_length", max_length=512, truncation=True)

out  = qwen(**source, output_hidden_states=True, use_cache=False)
last = out.last_hidden_state * source["attention_mask"][..., None]
taps = torch.stack([out.hidden_states[i] for i in (2, 3, 6, 9, 12, 21)], dim=2)

hidden  = fusion(last, taps, source["attention_mask"])          # (B, 512, 3072)
context = adapter(hidden, target["input_ids"],
                  target_attention_mask=target["attention_mask"],
                  source_attention_mask=source["attention_mask"])
context = context.masked_fill(~target["attention_mask"].bool()[..., None], 0)
# (B, 512, 1024) -- this is the DiT's crossattn_emb

Four details that matter:

• Both tokenizers see the same text. Qwen's ids drive the encoder; the T5 ids are the adapter's cross-attention queries, so a query slot is one retrieval slot.
• attention_fp32 = True on every LLMAdapterAttention, as trained and sampled. The adapter itself is kept in fp32 with autocast around it.
• Padding is zeroed after the adapter, not just masked, and the same encode path produces the negative prompt.
• Sampling settings used throughout: 1024x1024, 32 steps, guidance 4.0, flow shift 3.0.

Verified end to end on one GPU: loads, encodes, and produces a 1024x1024 image.

Status

Both stages complete. The image-epoch pair is the finished model.