1

Figure 1: Inference latency of 1 token when half the
memory of the model is available.

wide range of natural language tasks. However, the
unprecedented capabilities of these models come
with substantial computational and memory re-
quirements for inference. LLMs can contain hun-
dreds of billions or even trillions of parameters,
making it challenging to load and run them effi-
ciently, especially on resource-constrained devices.
Currently, the standard approach is to load the en-
tire model into DRAM for inference (Rajbhandari
et al., 2021; Aminabadi et al., 2022). However, this
severely limits the maximum model size that can
be run. For example, a 7 billion parameter model
requires over 14GB of memory just to load the
parameters in half-precision floating point format,
exceeding the capabilities of most edge devices.
LLM in a flash