title: Addition is All You Need for Energy-efficient Language Models
summary: Large neural networks spend most computation on floating point tensor
  multiplications. In this work, we find that a floating point multiplier can be
  approximated by one integer adder with high precision. We propose the
  linear-complexity multiplication L-Mul algorithm that approximates floating
  point number multiplication with integer addition operations. The new algorithm
  costs significantly less computation resource than 8-bit floating point
  multiplication but achieves higher precision. Compared to 8-bit floating point
  multiplications, the proposed method achieves higher precision but consumes
  significantly less bit-level computation. Since multiplying floating point
  numbers requires substantially higher energy compared to integer addition
  operations, applying the L-Mul operation in tensor processing hardware can
  potentially reduce 95% energy cost by element-wise floating point tensor
  multiplications and 80% energy cost of dot products. We calculated the
  theoretical error expectation of L-Mul, and evaluated the algorithm on a wide
  range of textual, visual, and symbolic tasks, including natural language
  understanding, structural reasoning, mathematics, and commonsense question
  answering. Our numerical analysis experiments agree with the theoretical error
  estimation, which indicates that L-Mul with 4-bit mantissa achieves comparable
  precision as float8_e4m3 multiplications, and L-Mul with 3-bit mantissa
  outperforms float8_e5m2. Evaluation results on popular benchmarks show that
  directly applying L-Mul to the attention mechanism is almost lossless. We
  further show that replacing all floating point multiplications with 3-bit
  mantissa L-Mul in a transformer model achieves equivalent precision as using
  float8_e4m3 as accumulation precision in both fine-tuning and inference.
paper_authors._id: 66fc9efa6d3e4a039730c68c
paper_authors.name: Hongyin Luo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-03T09:38:09.909Z
paper_authors._id: 62bc495016897fa76cdcda18
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1656506661693-noauth.jpeg
paper_authors.fullname: Hongyin Luo
paper_authors.type: user
paper_authors.user: luohy
paper_authors._id: 66fc9efa6d3e4a039730c68d
paper_authors.name: Wei Sun
paper_title: Addition is All You Need for Energy-efficient Language Models
paper_summary: Large neural networks spend most computation on floating point tensor
  multiplications. In this work, we find that a floating point multiplier can be
  approximated by one integer adder with high precision. We propose the
  linear-complexity multiplication L-Mul algorithm that approximates floating
  point number multiplication with integer addition operations. The new algorithm
  costs significantly less computation resource than 8-bit floating point
  multiplication but achieves higher precision. Compared to 8-bit floating point
  multiplications, the proposed method achieves higher precision but consumes
  significantly less bit-level computation. Since multiplying floating point
  numbers requires substantially higher energy compared to integer addition
  operations, applying the L-Mul operation in tensor processing hardware can
  potentially reduce 95% energy cost by element-wise floating point tensor
  multiplications and 80% energy cost of dot products. We calculated the
  theoretical error expectation of L-Mul, and evaluated the algorithm on a wide
  range of textual, visual, and symbolic tasks, including natural language
  understanding, structural reasoning, mathematics, and commonsense question
  answering. Our numerical analysis experiments agree with the theoretical error
  estimation, which indicates that L-Mul with 4-bit mantissa achieves comparable
  precision as float8_e4m3 multiplications, and L-Mul with 3-bit mantissa
  outperforms float8_e5m2. Evaluation results on popular benchmarks show that
  directly applying L-Mul to the attention mechanism is almost lossless. We
  further show that replacing all floating point multiplications with 3-bit
  mantissa L-Mul in a transformer model achieves equivalent precision as using
  float8_e4m3 as accumulation precision in both fine-tuning and inference.
paper_ai_keywords: floating point tensor multiplications floating point multiplier integer adder linear-complexity multiplication L-Mul algorithm integer addition operations bit-level computation L-Mul operation tensor processing hardware theoretical error expectation natural language understanding structural reasoning mathematics commonsense question answering attention mechanism transformer model fine-tuning inference float8_e4m3 multiplications float8_e5m2