title: Mixture of Scales: Memory-Efficient Token-Adaptive Binarization for
    Large Language Models
summary: Binarization, which converts weight parameters to binary values, has emerged
  as an effective strategy to reduce the size of large language models (LLMs).
  However, typical binarization techniques significantly diminish linguistic
  effectiveness of LLMs. To address this issue, we introduce a novel binarization
  technique called Mixture of Scales (BinaryMoS). Unlike conventional methods,
  BinaryMoS employs multiple scaling experts for binary weights, dynamically
  merging these experts for each token to adaptively generate scaling factors.
  This token-adaptive approach boosts the representational power of binarized
  LLMs by enabling contextual adjustments to the values of binary weights.
  Moreover, because this adaptive process only involves the scaling factors
  rather than the entire weight matrix, BinaryMoS maintains compression
  efficiency similar to traditional static binarization methods. Our experimental
  results reveal that BinaryMoS surpasses conventional binarization techniques in
  various natural language processing tasks and even outperforms 2-bit
  quantization methods, all while maintaining similar model size to static
  binarization techniques.
paper_authors._id: 6672682a6d27e245f657825a
paper_authors.name: Dongwon Jo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-19T07:37:26.386Z
paper_authors._id: 639ffbc6beb95d698de9640d
paper_authors.avatarUrl: /avatars/7ef1aaadd5b378d00e17dc548e42cb7e.svg
paper_authors.fullname: Dongwon Jo
paper_authors.type: user
paper_authors.user: dongwonjo
paper_authors._id: 6672682a6d27e245f657825b
paper_authors.name: Taesu Kim
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-20T10:00:19.449Z
paper_authors._id: 64dc388069d21c567b0794b0
paper_authors.avatarUrl: /avatars/795ab1b7f970df3b589f6c1ca75df9ad.svg
paper_authors.fullname: Taesu Kim
paper_authors.type: user
paper_authors.user: tae-su-kim
paper_authors._id: 6672682a6d27e245f657825c
paper_authors.name: Yulhwa Kim
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-20T10:00:40.017Z
paper_authors._id: 6566ddb96af53c602f80b1e2
paper_authors.avatarUrl: /avatars/403c8e486115920e50867b6462ddfd99.svg
paper_authors.fullname: Yulhwa Kim
paper_authors.type: user
paper_authors.user: YulhwaKim
paper_authors._id: 6672682a6d27e245f657825d
paper_authors.name: Jae-Joon Kim
paper_title: Mixture of Scales: Memory-Efficient Token-Adaptive Binarization for
    Large Language Models
paper_summary: Binarization, which converts weight parameters to binary values, has emerged
  as an effective strategy to reduce the size of large language models (LLMs).
  However, typical binarization techniques significantly diminish linguistic
  effectiveness of LLMs. To address this issue, we introduce a novel binarization
  technique called Mixture of Scales (BinaryMoS). Unlike conventional methods,
  BinaryMoS employs multiple scaling experts for binary weights, dynamically
  merging these experts for each token to adaptively generate scaling factors.
  This token-adaptive approach boosts the representational power of binarized
  LLMs by enabling contextual adjustments to the values of binary weights.
  Moreover, because this adaptive process only involves the scaling factors
  rather than the entire weight matrix, BinaryMoS maintains compression
  efficiency similar to traditional static binarization methods. Our experimental
  results reveal that BinaryMoS surpasses conventional binarization techniques in
  various natural language processing tasks and even outperforms 2-bit
  quantization methods, all while maintaining similar model size to static
  binarization techniques.
paper_ai_keywords: Binarization Large language models (LLMs) Mixture of Scales (BinaryMoS) Scaling experts Token-adaptive approach Representational power Binary weights Contextual adjustments Scaling factors Compression efficiency Static binarization methods Natural language processing tasks Super-bit quantization methods