4 Results
Experimental Setup: Our experiment is designed
to optimize inference efficiency on personal de-
vices. To this end, we process sequences individ-
ually, running only one sequence at a time. This
approach allows us to allocate a specific portion of
DRAM for the Key-Value (KV) cache while pri-
marily focusing on the model size. This strategy is
particularly effective when dealing with only one
sequence/query at a time.2

2For OPT 6.7 B model with context length 2048 KV-cache
requires 2048 × 2dmodel elements which is only 8% of model
size. Also the KV-cache itself can be held in flash memory.

7
LLM in a flash