title: Optimize Weight Rounding via Signed Gradient Descent for the
    Quantization of LLMs
summary: Large Language Models (LLMs) have proven their exceptional capabilities in
  performing language-related tasks. However, their deployment poses significant
  challenges due to their considerable memory and storage requirements. In
  response to this issue, weight-only quantization, particularly 3 and 4-bit
  weight-only quantization, has emerged as one of the most viable solutions. As
  the number of bits decreases, the quantization grid broadens, thus emphasizing
  the importance of up and down rounding. While previous studies have
  demonstrated that fine-tuning up and down rounding with the addition of
  perturbations can enhance accuracy in some scenarios, our study is driven by
  the precise and limited boundary of these perturbations, where only the
  threshold for altering the rounding value is of significance. Consequently, we
  propose a concise and highly effective approach for optimizing the weight
  rounding task. Our method, named SignRound, involves lightweight block-wise
  tuning using signed gradient descent, enabling us to achieve outstanding
  results within 400 steps. SignRound outperforms the established baseline of
  rounding-to-nearest (RTN) and competes impressively against recent methods,
  without introducing additional inference overhead. The source code will be
  publicly available at https://github.com/intel/neural-compressor soon.
paper_authors._id: 64ffd7a696ec8a52185de903
paper_authors.name: Wenhua Cheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-12T12:17:58.296Z
paper_authors._id: 634772780a9fa1ac49994510
paper_authors.avatarUrl: /avatars/2e88e32ac0d45a1c624026e497eb00b3.svg
paper_authors.fullname: wenhua cheng
paper_authors.type: user
paper_authors.user: wenhuach
paper_authors._id: 64ffd7a696ec8a52185de904
paper_authors.name: Weiwei Zhang
paper_authors._id: 64ffd7a696ec8a52185de905
paper_authors.name: Haihao Shen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-12T12:19:29.197Z
paper_authors._id: 60ac3318e3de7c7440abb850
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60ac3318e3de7c7440abb850/DkMrPBr6Ew_dS_c9kog4e.jpeg
paper_authors.fullname: Haihao Shen
paper_authors.type: user
paper_authors.user: Haihao
paper_authors._id: 64ffd7a696ec8a52185de906
paper_authors.name: Yiyang Cai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-12T12:19:37.996Z
paper_authors._id: 634e0e3b7521b97439520abe
paper_authors.avatarUrl: /avatars/d3aac6582dc4862ed640eff2ddb8b3c3.svg
paper_authors.fullname: Yiyang Cai
paper_authors.type: user
paper_authors.user: frankcai
paper_authors._id: 64ffd7a696ec8a52185de907
paper_authors.name: Xin He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-12T12:19:55.249Z
paper_authors._id: 61e4c97506a27a84a78caaac
paper_authors.avatarUrl: /avatars/5849f8d98755070cbee148a344c899ca.svg
paper_authors.fullname: He, Xin
paper_authors.type: user
paper_authors.user: xinhe
paper_authors._id: 64ffd7a696ec8a52185de908
paper_authors.name: Kaokao Lv
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-12T12:20:08.353Z
paper_authors._id: 630881ff6fb2ea4413f16390
paper_authors.avatarUrl: /avatars/8cdf266604b24bc9ae599aa2def8debd.svg
paper_authors.fullname: lvkaokao
paper_authors.type: user
paper_authors.user: lvkaokao
paper_title: Optimize Weight Rounding via Signed Gradient Descent for the
    Quantization of LLMs
paper_summary: Large Language Models (LLMs) have proven their exceptional capabilities in
  performing language-related tasks. However, their deployment poses significant
  challenges due to their considerable memory and storage requirements. In
  response to this issue, weight-only quantization, particularly 3 and 4-bit
  weight-only quantization, has emerged as one of the most viable solutions. As
  the number of bits decreases, the quantization grid broadens, thus emphasizing
  the importance of up and down rounding. While previous studies have
  demonstrated that fine-tuning up and down rounding with the addition of
  perturbations can enhance accuracy in some scenarios, our study is driven by
  the precise and limited boundary of these perturbations, where only the
  threshold for altering the rounding value is of significance. Consequently, we
  propose a concise and highly effective approach for optimizing the weight
  rounding task. Our method, named SignRound, involves lightweight block-wise
  tuning using signed gradient descent, enabling us to achieve outstanding
  results within 400 steps. SignRound outperforms the established baseline of
  rounding-to-nearest (RTN) and competes impressively against recent methods,
  without introducing additional inference overhead. The source code will be
  publicly available at https://github.com/intel/neural-compressor soon.
paper_ai_summary: SignRound, a lightweight block-wise tuning method using signed gradient descent, optimizes weight rounding in weight-only quantization of Large Language Models, outperforming RTN and other recent methods without inference overhead.
paper_ai_keywords: weight-only quantization 3-bit 4-bit up and down rounding perturbations SignRound signed gradient descent rounding-to-nearest (RTN)