Latency Analysis: Using a window size of 5,
each token requires access to 2.4% of the Feed
Forward Network (FFN) neurons. For a 32-bit
model, the data chunk size per read is 2dmodel ×
4 bytes = 32 KiB, as it involves concatenated rows
and columns. On an M1 Max, this results in a la-
tency of 125ms per token for loading from flash
and 65ms for memory management (involving neu-
ron deletion and addition). Thus, the total memory-
related latency is less than 190ms per token (refer
to Figure 1). In contrast, the baseline approach,
which requires loading 13.4GB of data at a speed
of 6.1GB/s, leads to a latency of approximately
2330ms per token. Therefore, our method repre-
sents a substantial improvement over the baseline.
For a 16-bit model on a GPU machine, the flash
load time is reduced to 40.5ms, and memory man-
agement takes 40ms, slightly higher due to the
additional overhead of transferring data from CPU
to GPU. Nevertheless, the baseline method’s I/O
LLM in a flash