6.10 GB/s
6.10 GB/s
1.25 GB/s
1.25 GB/s
2.25 GB/s

2130 ms
1090 ms
738 ms
164 ms
87 ms

under the condition that approximately half of the
model size is available for use in DRAM.

Predictors. In the Falcon 7B model, predictors
of rank r = 256 are used for the initial 28 layers,
and r = 1152 for the last four layers.

Window Configuration. Our model reserves
memory for a window containing the last 4 tokens.
This setup utilizes 33% of the Feed Forward Net-
work (FFN). In terms of memory allocation, em-
beddings take 4.2% of the model size, attention
weights account for 19.4%, and predictors require
4%. The active portion of the FFN, given our win-
dow size, is 25.3% (calculated as 0.33 × 76.8).
Overall, this amounts to 52.93% of the model’s
total size.
LLM in a flash