Detailed comparisons of how each method im-

pacts performance are provided in Table 2.

4.2 Results for Falcon 7B Model
To verify that our findings generalize beyond OPT
models we also apply the idea of LLM in flash to
Falcon model. Since, the base line Falcon model is
not sparse, we used a sparsified (relufied) version
with almost the same performance as that of the
base version (Mirzadeh et al., 2023). Similar to
previous section, we present the results obtained

8

Table 2: The I/O latency of OPT 6.7B 16 bit on M1 max for different techniques when half the memory is available

Configuration

Performance Metrics

Hybrid Predictor Windowing Bundling DRAM (GB) Flash→ DRAM(GB) Throughput (GB/s)

I/O Latency (ms)

✗
✓
✓
✓
✓

✗
✗
✓
✓
✓

✗
✗
✗
✓
✓

✗
✗
✗
✗
✓

0
6.7
4.8
6.5
6.5

13.4 GB
6.7 GB
0.9 GB
0.2 GB
0.2 GB

6.10 GB/s
6.10 GB/s
1.25 GB/s
1.25 GB/s
2.25 GB/s

2130 ms
1090 ms
738 ms
164 ms
87 ms
LLM in a flash