title: Full Parameter Fine-tuning for Large Language Models with Limited
    Resources
summary: Large Language Models (LLMs) have revolutionized Natural Language Processing
  (NLP) but demand massive GPU resources for training. Lowering the threshold for
  LLMs training would encourage greater participation from researchers,
  benefiting both academia and society. While existing approaches have focused on
  parameter-efficient fine-tuning, which tunes or adds a small number of
  parameters, few have addressed the challenge of tuning the full parameters of
  LLMs with limited resources. In this work, we propose a new optimizer,
  LOw-Memory Optimization (LOMO), which fuses the gradient computation and the
  parameter update in one step to reduce memory usage. By integrating LOMO with
  existing memory saving techniques, we reduce memory usage to 10.8% compared to
  the standard approach (DeepSpeed solution). Consequently, our approach enables
  the full parameter fine-tuning of a 65B model on a single machine with 8 RTX
  3090, each with 24GB memory.
paper_authors._id: 648ff410274cc5506c9b1d9c
paper_authors.name: Kai Lv
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-19T11:44:27.450Z
paper_authors._id: 6485d5b300c9cfe5c2470c81
paper_authors.avatarUrl: /avatars/c29aa81d2add795e8448b99274a04b83.svg
paper_authors.fullname: Kai
paper_authors.type: user
paper_authors.user: KaiLv
paper_authors._id: 648ff410274cc5506c9b1d9d
paper_authors.name: Yuqing Yang
paper_authors._id: 648ff410274cc5506c9b1d9e
paper_authors.name: Tengxiao Liu
paper_authors._id: 648ff410274cc5506c9b1d9f
paper_authors.name: Qinghui Gao
paper_authors._id: 648ff410274cc5506c9b1da0
paper_authors.name: Qipeng Guo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-21T09:17:34.949Z
paper_authors._id: 6491cd52b1e5d3444528edb1
paper_authors.avatarUrl: /avatars/a85635d886c7f157b6723dec5c01c030.svg
paper_authors.fullname: Qipeng Guo
paper_authors.type: user
paper_authors.user: QipengGuo
paper_authors._id: 648ff410274cc5506c9b1da1
paper_authors.name: Xipeng Qiu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-19T13:06:31.125Z
paper_authors._id: 61457b8deff2c9fdb4de4988
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1632381702899-61457b8deff2c9fdb4de4988.jpeg
paper_authors.fullname: Xipeng Qiu
paper_authors.type: user
paper_authors.user: xpqiu
paper_title: Full Parameter Fine-tuning for Large Language Models with Limited
    Resources
paper_summary: Large Language Models (LLMs) have revolutionized Natural Language Processing
  (NLP) but demand massive GPU resources for training. Lowering the threshold for
  LLMs training would encourage greater participation from researchers,
  benefiting both academia and society. While existing approaches have focused on
  parameter-efficient fine-tuning, which tunes or adds a small number of
  parameters, few have addressed the challenge of tuning the full parameters of
  LLMs with limited resources. In this work, we propose a new optimizer,
  LOw-Memory Optimization (LOMO), which fuses the gradient computation and the
  parameter update in one step to reduce memory usage. By integrating LOMO with
  existing memory saving techniques, we reduce memory usage to 10.8% compared to
  the standard approach (DeepSpeed solution). Consequently, our approach enables
  the full parameter fine-tuning of a 65B model on a single machine with 8 RTX
  3090, each with 24GB memory.
paper_ai_keywords: Large Language Models (LLMs) Natural Language Processing (NLP) parameter-efficient fine-tuning optimizer LOw-Memory Optimization (LOMO) gradient computation parameter update memory usage memory saving techniques DeepSpeed solution full parameter fine-tuning RTX 3090