title: LLM in a flash: Efficient Large Language Model Inference with Limited
    Memory
summary: Large language models (LLMs) are central to modern natural language
  processing, delivering exceptional performance in various tasks. However, their
  intensive computational and memory requirements present challenges, especially
  for devices with limited DRAM capacity. This paper tackles the challenge of
  efficiently running LLMs that exceed the available DRAM capacity by storing the
  model parameters on flash memory but bringing them on demand to DRAM. Our
  method involves constructing an inference cost model that harmonizes with the
  flash memory behavior, guiding us to optimize in two critical areas: reducing
  the volume of data transferred from flash and reading data in larger, more
  contiguous chunks. Within this flash memory-informed framework, we introduce
  two principal techniques. First, "windowing'" strategically reduces data
  transfer by reusing previously activated neurons, and second, "row-column
  bundling", tailored to the sequential data access strengths of flash memory,
  increases the size of data chunks read from flash memory. These methods
  collectively enable running models up to twice the size of the available DRAM,
  with a 4-5x and 20-25x increase in inference speed compared to naive loading
  approaches in CPU and GPU, respectively. Our integration of sparsity awareness,
  context-adaptive loading, and a hardware-oriented design paves the way for
  effective inference of LLMs on devices with limited memory.
paper_authors._id: 6582524a5f6d8343825e4af2
paper_authors.name: Keivan Alizadeh
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-04-23T19:22:30.221Z
paper_authors._id: 66280a4c7c988f33b795f90f
paper_authors.avatarUrl: /avatars/7a618b39c4ed20f5490ea64fbb67b5ac.svg
paper_authors.fullname: Iman Mirzadeh
paper_authors.type: user
paper_authors.user: imirzadeh-apple
paper_authors._id: 6582524a5f6d8343825e4af3
paper_authors.name: Iman Mirzadeh
paper_authors._id: 6582524a5f6d8343825e4af4
paper_authors.name: Dmitry Belenko
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-20T08:59:28.800Z
paper_authors._id: 63185e1a70b013d6609416ba
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63185e1a70b013d6609416ba/LOB6bJ71wwgamgN3MCuFb.jpeg
paper_authors.fullname: Dmitry Belenko
paper_authors.type: user
paper_authors.user: depthwise
paper_authors._id: 6582524a5f6d8343825e4af5
paper_authors.name: Karen Khatamifard
paper_authors._id: 6582524a5f6d8343825e4af6
paper_authors.name: Minsik Cho
paper_authors._id: 6582524a5f6d8343825e4af7
paper_authors.name: Carlo C Del Mundo
paper_authors._id: 6582524a5f6d8343825e4af8
paper_authors.name: Mohammad Rastegari
paper_authors._id: 6582524a5f6d8343825e4af9
paper_authors.name: Mehrdad Farajtabar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-20T08:21:56.790Z
paper_authors._id: 62d188a6d50433c35eb14020
paper_authors.avatarUrl: /avatars/fc3d149a27bc8396010a2a02148e8ca0.svg
paper_authors.fullname: Mehrdad Farajtabar
paper_authors.type: user
paper_authors.user: mfarajtabar
paper_title: LLM in a flash: Efficient Large Language Model Inference with Limited
    Memory
paper_summary: Large language models (LLMs) are central to modern natural language
  processing, delivering exceptional performance in various tasks. However, their
  intensive computational and memory requirements present challenges, especially
  for devices with limited DRAM capacity. This paper tackles the challenge of
  efficiently running LLMs that exceed the available DRAM capacity by storing the
  model parameters on flash memory but bringing them on demand to DRAM. Our
  method involves constructing an inference cost model that harmonizes with the
  flash memory behavior, guiding us to optimize in two critical areas: reducing
  the volume of data transferred from flash and reading data in larger, more
  contiguous chunks. Within this flash memory-informed framework, we introduce
  two principal techniques. First, "windowing'" strategically reduces data
  transfer by reusing previously activated neurons, and second, "row-column
  bundling", tailored to the sequential data access strengths of flash memory,
  increases the size of data chunks read from flash memory. These methods
  collectively enable running models up to twice the size of the available DRAM,
  with a 4-5x and 20-25x increase in inference speed compared to naive loading
  approaches in CPU and GPU, respectively. Our integration of sparsity awareness,
  context-adaptive loading, and a hardware-oriented design paves the way for
  effective inference of LLMs on devices with limited memory.
paper_ai_keywords: large language models (LLMs) natural language processing flash memory DRAM capacity inference cost model windowing neuron reuse row-column bundling sequential data access sparsity awareness context-adaptive loading hardware-oriented design