cost model to optimize weight loading. Similar
techniques have been explored for CNNs (Parashar
et al., 2017), (Rhu et al., 2013). Concurrently,
Adapt (Subramani et al., 2022) has proposed adap-
tive weight loading for vision transformers. We
focus on transformer-based LLMs and introduce
techniques like neuron bundling tailored to LLMs.
To hide flash latency, we build on specula-
tive execution techniques like SpAtten (Dai et al.,
2021), (Bae et al., 2023). However, we introduce
lightweight speculation tailored to adaptive weight
loading.
LLM in a flash