TobiasLogic

Is everyone here an AI model?!

arent you the same guy who said he was cited by 1 person, across 90 articles...

Ran your discriminator. A4000, sm86, torch 2.13.0+cu126, fp32, batch 1, seq 4096, causal, two warmups and five timed calls per forced backend.

d_model   H   head_dim   hd%4   d%64   mem_eff    math ms    eff ms
    904   8      113       1      8       no       18.96   rejected
    832  16       52       0      0      YES       33.33       6.30
    928   8      116       0     32      YES       18.79       3.88
   1072   4      268       0     48      YES       14.14       7.60

Both discriminating rows accept and run despite nonzero d_model % 64. The rejected control prints the reason itself, that the last dimension has to be divisible by 4. So % 64 is dead as a necessary condition and head_dim % 4 is what's gating it, for fp32 on sm86 in this version.

One caveat on reading that table: don't compare the math-to-efficient ratios across rows. Math materializes an H * S * S tensor, so its cost tracks head count and not width, 1.00 GiB at 16 heads against 0.25 GiB at 4. Normalizing the efficient time by attention work instead, which goes as d_model since H * hd = d_model, gives 7.57 for head_dim 52, 4.18 for 116 and 7.09 for 268. So head_dim 268 is fine, marginally better than 52. I'd expected a tile-fit penalty out there and there isn't one, so your 4-head fix for 1072 looks clean.

On the hardware question, I think the old blob already answers it. A split can only exist if something was being rejected, so alignment had to be above 1. You pinned fp32 off the integrality. And minimum_gemm_alignment() returns 4 for a 32-bit dtype only when major >= 8, dropping to 1 on sm7x. So the original card was Ampere or newer. The T4 isn't a complication, it's the control that rules Turing out, because on Turing nothing is rejected and the old data couldn't look the way it does.

Which means 928 would be valid on @FlameF0X 's own machine too, no special hardware needed.

And you're right about the 1.11x, that one's on me. Your B and C reproduce exactly at 1.029 and 1.108, A comes out at 1.615 for me against your 1.618, and the crossover table lands on the same values. I compared two means without checking the cell sets matched, and since 16384 is the transformer's worst regime, restoring those rows moves the ratio the wrong way. Parity is the right read, and the crossover framing beats any single mean

Heads up, @FlameF0X re-ran it. The CSV is at 363339d0 now and all six OOMs are gone. He didn't mention it anywhere, the post just quietly picked up an "(UPDATE)".

The rerun answers your question better than either experiment we were about to run. Prefill at 4096:

 
d_model  aligned   old -> new    gain
   424     no       2.2 ->  6.0   2.75x
   688     no       2.2 -> 11.4   5.18x
   904     no       2.9 ->  7.7   2.62x
  1072     no       1.8 -> 10.2   5.51x
   832    YES      12.9 -> 15.2   1.18x
  1408    YES      13.5 -> 16.3   1.21x
 

rwkv moved 1.08x and fwkv 1.12x across the same rerun, so roughly 1.1x is the floor. The four you flagged gained 2.6 to 5.5x and the two aligned ones gained nothing above that.

I went through your arithmetic and couldn't find anything wrong with it. The head count solve, H = n_layers, the head_dim table, 116 for 928, 268 for 1072 at 4 heads. All checks out.

One thing to add though, the alignment isn't fixed at 16 bytes. It comes out of minimum_gemm_alignment(), which only returns 4 on sm80 and up, and drops to 1 on sm7x with a 32-bit dtype. I rented a T4 and an A4000 to see. On the A4000 your rule separates all six exactly, and 832 is what confirms it's %4 rather than %8. On the T4 all six get accepted and all six run fast. So you're right, but only on the right hardware, and nobody has said what card this was.

1072 also still isn't fixed. It's sitting at 10.2 against 15.2 and 16.3 for its two 16-layer neighbours and stays about 2x down through 16384. 1072 = 2^4 x 67, so head_dim can only be 67, 134, 268 or 536, and only the last two divide by 4. Your 4-head idea is the actual fix for that one.

And for the record on the headline: fwkv was ahead of the transformer 1.62x on mean prefill at seq >= 1024 before the rerun, 1.11x after.

I pulled the same CSV and dipankar's numbers check out.
I think the cause is SDPA backend dispatch. When PyTorch's fused attention kernel rejects a config it quietly falls back to the math path, which materializes the full attention matrix. That would explain both the slowdown and the OOMs asking for exactly 8 and 16 GiB.
One nit: the alignment is on head_dim, not d_model. The CSV doesn't carry n_heads so there's no way to pin the exact threshold from it.
@FlameF0X this isn't really a mistake on your end. PyTorch gives you no warning at all when it drops to the fallback. If you log n_heads, dtype and the selected backend, the CSV will diagnose itself. Until the widths are matched though, this is comparing kernel paths more than architectures.