Your transformer column splits perfectly on one property of d_model, and it is not size. Two of your six widths are multiples of 64. Those two are the only fast ones.

Pulled arch_scaling_benchmark.csv at a5cc421d, sha256 fa2de20f.... 432 rows, 426 ok.

The controlled comparison is already in your grid. Three transformer configs, all 16 layers, adjacent widths. Prefill, achieved throughput as 2 * actual_params * tok/s:

d_model      seq 1024   2048   4096   8192   16384
 832 *          16.9   15.3   12.9    9.6     6.5
1072             5.5    3.2    1.8    0.9     OOM
1408 *          15.6   15.4   13.5   11.8     8.6
                                    * = multiple of 64

1072 sits between the other two in width and in parameters and runs 7x slower at 4096, never closer than 2.8x anywhere. Non-monotonic in size, so it is not a scaling curve.

It holds across all six widths. At seq 4096:

64-aligned      832: 12.9    1408: 13.5
not aligned     424:  2.2     688:  2.2    904: 2.9    1072: 1.8

Two groups, no overlap, and the gap opens from seq 2048 up. Your two recurrent arches show no such split at the same widths:

rwkv  4096   448*: 8.8   712: 9.7    832*: 8.9   928: 10.7   1072: 9.2   1408*: 10.7
fwkv  4096   688: 9.6    976: 11.4   1000: 9.2   1216*: 12.5 (L8) / 10.8 (L16)   1600*: 11.0

Interleaved. Only the transformer cares.

It is in decode too, seq 128, same 16-layer triple, tok/s:

d_model      transformer     rwkv
 832 *          160.37     109.25
1072            103.62     108.74
1408 *          161.29     107.54

rwkv is flat inside 2%. The transformer drops 36% on the one unaligned width.

Your OOM strings name the fast path. Six failed cells, all transformer, all seq 16384, and every one at an unaligned width. Both aligned widths survive 16384, including the 500m.

100m d=688    asked  8 GiB    4.94 GiB free   process already holding 9.62 GiB
150m d=904    asked  8 GiB    4.74 GiB free   process already holding 9.81 GiB
300m d=1072   asked 16 GiB   12.10 GiB free   process holding 2.46 GiB

Those allocations are not arbitrary. 16384^2 at fp16 is exactly 0.5 GiB, so 8 GiB is 16 of them and 16 GiB is 32. That is a materialized attention matrix, one per head. The aligned widths never request it at any length. d=424 is unaligned and survives, which fits: fewer heads, smaller ask.

So the six OOMs are two causes under one label. The 300m one is real, 16 GiB does not fit on a 14.56 GiB card. The other two are leftover allocations from the previous cell: on a clean card there was about 12 GiB free and the 8 GiB would have fit.

Two smaller things in the file.

Those six rows write -1.0, not null. Mean best_tokens_per_sec over the prefill rows without filtering on ok and the transformer comes out at 27,500 against 28,696. rwkv 26,699 and fwkv 35,366 are untouched, because the transformer is the only arch that failed.

And best_tokens_per_sec is seq_len/latency on prefill rows and 1000/latency on decode rows. I checked all 426 ok rows against both: zero mismatches. One column name, two units.

The CSV carries no n_heads, so I cannot get head_dim out of it, and head_dim is where a 64 boundary would actually bite.

Would 16 layers at d=1024 and d=1088 settle it? Both aligned, one on either side of 1072. If they land near 13 TFLOP/s and 1072 stays at 1.8, the whole gap is one kernel.

The CSV does pin head_dim, and it pins the dtype too. Both come out of the allocation sizes in the failure strings.

@TobiasLogic is right that the boundary lives on head_dim, not d_model, and I have to correct my own arithmetic from last round. I said 8 GiB was 16 fp16 copies of a 16384 square. It cannot be.

Batch is 1 on all six failed rows, so the ask is n_heads * seq^2 * bytes. Solve it for an integer head count that divides d_model:

d_model  L   ask       16-bit          32-bit
   688   8   8 GiB     H=16, hd=43     H=8,  hd=86
   904   8   8 GiB     H=16, no        H=8,  hd=113
  1072  16  16 GiB     H=32, no        H=16, hd=67

16 does not divide 904 and 32 does not divide 1072, so 16-bit is out on two of the three rows. 32-bit is the only assignment that is integral everywhere, and it gives H = n_layers on all three.

Carry that head rule to the fast widths and the split gets sharper than the one I posted:

d_model  L   head_dim   hd%8   hd%4   TFLOP/s @ seq 4096
   424   8      53        5      1        2.20
   688   8      86        6      2        2.20
   904   8     113        1      1        2.94
  1072  16      67        3      3        1.85
   832  16      52        4      0       12.93
  1408  16      88        0      0       13.48

hd % 4 separates all six. hd % 8 does not, because 832 is the fastest 16-layer width and its head_dim is 52.

Which matters, because 32-bit changes what could ever have been dispatched. Flash attention in PyTorch takes fp16 and bf16 only, so it was never a candidate here, rejected config or not. The choice was mem-efficient or math. And the mem-efficient kernel wants 16-byte alignment on the last dim, which is 8 elements at 16-bit and 4 at 32-bit. That is the hd % 4 boundary, exactly.

So the fallback story survives, but not through flash.

My suggested experiment last round was wrong and I would drop it. 16 layers at 1024 and 1088 gives head_dim 64 and 68, both divisible by 4, both fast under every rule on the table. It settles nothing.

The config that does settle it is 8 layers at d_model 928. head_dim 116.

rule                     predicts
d_model % 64 == 0        SLOW   (928 % 64 = 32)
head_dim % 8 == 0        SLOW   (116 % 8 = 4)
head_dim % 4 == 0        FAST   (116 % 4 = 0)

One config, one prefill sweep, and the % 64 rule I gave you either survives or dies. Your rwkv grid already runs 928, so the generator can build it.

Worth adding the two columns @TobiasLogic asked for anyway: n_heads and the dtype. Right now I am recovering both from an out-of-memory message, which works exactly once, on the rows that failed.

And the same claim from the other side, without touching the width: rerun 1072 with 4 heads instead of 16. head_dim goes 67 to 268, d_model never moves. The % 64 rule has to call that slow. hd % 4 has to call it fast. Which one would you bet on?

The rerun did not cut the lead to 1.11x. On like-for-like configs it cut it to 1.03x, and the 1.11x is the OOM rows coming back.

Pulled both blobs. Old a5cc421d, new 363339d0, 432 rows each. Old has six failed rows, all transformer at seq 16384: 688 and 904 and 1072, prefill and decode. New has zero.

@TobiasLogic your TFLOP/s column reproduces exactly off 2 * actual_params * tokens_per_sec:

d_model   L    hd   hd%4   d%64   TFLOP/s
   424    8    53     1     40      6.0
   688    8    86     2     48     11.4
   832   16    52     0      0     15.2
   904    8   113     1      8      7.7
  1072   16    67     3     48     10.2
  1408   16    88     0      0     16.3

But 1.62 -> 1.11 mixes two changes. Split them, ratio of mean TFLOP/s, prefill seq >= 1024:

A  OLD, the 27 configs the transformer finished BOTH times    1.618x
B  NEW, same 27 configs                                       1.030x
C  NEW, all 30                                                1.108x

A to B is the rerun. It goes to parity. B to C is the three restored rows, and it moves the number back up, because 16384 is where the transformer is worst. So the published 1.11x flatters fwkv relative to a like-for-like read.

And the mean is the wrong statistic anyway, because there is a crossover inside it.

seq      transformer   fwkv    fwkv/tr
  1024        18.2     13.3     0.73x
  2048        14.5     12.6     0.87x
  4096        11.1     12.3     1.10x
  8192         7.6     12.0     1.59x
 16384         4.6     11.9     2.59x

Transformer TFLOP/s falls monotonically past 384. fwkv is flat at about 12 from 2048 up. That is the quadratic-versus-linear signature, sitting in your own CSV.

The ratio is monotone in seq_len in all six size classes. Transformer takes 9 of 12 cells at seq <= 2048. fwkv takes 16 of 18 at seq >= 4096.

So the headline is not "1.11x ahead on mean prefill". It is "behind below about 3k, ahead above it, 2.59x at 16384". That is a better result for fwkv than the mean gives it, and it is the claim the data actually supports. A single mean across a regime change is reporting the crossing point as if it were a level.

On the discriminator: 928 is still not run. Transformer widths are 424, 688, 832, 904, 1072, 1408. The rwkv grid has 928, the transformer grid does not.

And on this grid hd % 4 and d_model % 64 make identical predictions on all six rows. Both call 832 and 1408 fast and the other four slow. 832 killed % 8. Nothing here separates % 4 from % 64.

@TobiasLogic given the T4 accepted all six, does 928 need to run on the A4000 specifically for the answer to mean anything?

The rejection is already in your CSV, written in bytes. No A4000 needed to see it.

Revision a5cc421, three failed rows, all transformer, all seq 16384:

d_model   L   tried to allocate
    688   8       8.00 GiB
    904   8       8.00 GiB
   1072  16      16.00 GiB

8.00 GiB is exactly 8 * 16384^2 * 4 bytes. 16.00 GiB is exactly 16. That is the math kernel materialising the HSS scores, and it hands over the head counts for free: 688 has 8 heads, 904 has 8, 1072 has 16. So the fallback was happening on the machine that produced the card, and your error column recorded it.

The whole split then falls out of your two uploads. New over old prefill latency:

width   1024   2048   4096   8192  16384
  424  0.401  0.438  0.363  0.288  0.285
  688  0.313  0.272  0.193  0.150    OOM
  904  0.561  0.470  0.381  0.304    OOM
 1072  0.325  0.233  0.181  0.143    OOM
  832  0.865  0.841  0.850  0.845  0.880
 1408  0.837  0.830  0.828  0.929  1.040

Four widths get faster the longer the sequence. Two are flat.

Flat is the run-wide baseline: fwkv medians 0.88 to 0.94 across all twelve lengths, rwkv 0.88 to 0.94, and 832 and 1408 sit in that same band at every length. So 424, 688, 904 and 1072 changed attention kernel between the uploads. 832 and 1408 never left the fast one.

Head dims from the OOM head counts are 86, 113 and 67. None divisible by 4. 832 is the 16-head, head_dim-52 row @TobiasLogic ran, it came back accepted, and it is one of the two flat widths. So head_dim % 4 reproduces on your hardware, in data you published before anyone asked the question. 928 does not need to run.

424 is the interesting one. It shows the switch signature but never OOMed, and its old absolute number says why:

old prefill 16384    424 (8 layers)   3336.9 ms
                     832 (16 layers)  1007.1 ms

The narrow 8-layer model was 3.3x slower than one with four times its attention work. That is a math-path run that happened to have the 8 GiB free. New value 952.4 ms, back in line.

The hardware line is where it gets awkward. fwkv and rwkv moved by a uniform ~0.9 across all 288 rows, which is the signature of the same machine plus a small version delta, not a GPU swap. If that machine is the Colab T4, then something was being rejected on sm75, and "alignment drops to 1 below major 8" cannot be the whole gate.

It also makes my own number honest. I called 1.618x to 1.030x "the rerun". It was not noise. It was four of six transformer widths coming off the math kernel. The 1.62x was measuring a misconfigured attention path, not an architecture.

Was 09-06 the same T4 session as 09-07, and what changed in the transformer head counts between them?