To circumvent these limitations, we advocate
two primary strategies, which can be employed
jointly. The first involves reading larger chunks
of data. Although throughput growth is not linear
(larger chunks take longer to transfer), the latency
for the initial byte becomes a smaller fraction of the
total request time, resulting in more efficient data
reading. This principle is depicted in Figure 2b.
Perhaps a counterintuitive yet interesting obser-
vation is that in some scenarios, it will be faster
to read more than needed (but in larger chunks)
and then discard, than only reading necessary parts
but in smaller chunks. The second strategy lever-
ages parallelized reads, utilizing the inherent paral-
lelism within storage stacks and flash controllers.
Our results indicate that throughputs appropriate
for sparse LLM inference are achievable on stan-
dard hardware using 32KiB or larger random reads
across multiple threads.
LLM in a flash