title: Binary and Ternary Natural Language Generation
summary: Ternary and binary neural networks enable multiplication-free computation and
  promise multiple orders of magnitude efficiency gains over full-precision
  networks if implemented on specialized hardware. However, since both the
  parameter and the output space are highly discretized, such networks have
  proven very difficult to optimize. The difficulties are compounded for the
  class of transformer text generation models due to the sensitivity of the
  attention operation to quantization and the noise-compounding effects of
  autoregressive decoding in the high-cardinality output space. We approach the
  problem with a mix of statistics-based quantization for the weights and elastic
  quantization of the activations and demonstrate the first ternary and binary
  transformer models on the downstream tasks of summarization and machine
  translation. Our ternary BART base achieves an R1 score of 41 on the
  CNN/DailyMail benchmark, which is merely 3.9 points behind the full model while
  being 16x more efficient. Our binary model, while less accurate, achieves a
  highly non-trivial score of 35.6. For machine translation, we achieved BLEU
  scores of 21.7 and 17.6 on the WMT16 En-Ro benchmark, compared with a full
  precision mBART model score of 26.8. We also compare our approach in the 8-bit
  activation setting, where our ternary and even binary weight models can match
  or outperform the best existing 8-bit weight models in the literature. Our code
  and models are available at:
  https://github.com/facebookresearch/Ternary_Binary_Transformer
paper_authors._id: 647ebce9becb41a27297d7f6
paper_authors.name: Zechun Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-06T17:23:48.383Z
paper_authors._id: 620ae55fe190de4b8f27ba9d
paper_authors.avatarUrl: /avatars/aaa2d0eadbcf96c2eb9059e3d73c2760.svg
paper_authors.fullname: Liu
paper_authors.type: user
paper_authors.user: Zechun
paper_authors._id: 647ebce9becb41a27297d7f7
paper_authors.name: Barlas Oguz
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-02-25T00:32:56.632Z
paper_authors._id: 6798100acbb6655a3cbca13b
paper_authors.avatarUrl: /avatars/c2a53c37d39753dc53e12a3349b026a0.svg
paper_authors.fullname: Barlas Oguz
paper_authors.type: user
paper_authors.user: borguz
paper_authors._id: 647ebce9becb41a27297d7f8
paper_authors.name: Aasish Pappu
paper_authors._id: 647ebce9becb41a27297d7f9
paper_authors.name: Yangyang Shi
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-11-01T19:54:26.400Z
paper_authors._id: 638fca982ddd69e70b8bdb8d
paper_authors.avatarUrl: /avatars/58ee470ca8cd3f0ae01bcaad33911755.svg
paper_authors.fullname: Yangyang Shi
paper_authors.type: user
paper_authors.user: yyshi
paper_authors._id: 647ebce9becb41a27297d7fa
paper_authors.name: Raghuraman Krishnamoorthi
paper_title: Binary and Ternary Natural Language Generation
paper_summary: Ternary and binary neural networks enable multiplication-free computation and
  promise multiple orders of magnitude efficiency gains over full-precision
  networks if implemented on specialized hardware. However, since both the
  parameter and the output space are highly discretized, such networks have
  proven very difficult to optimize. The difficulties are compounded for the
  class of transformer text generation models due to the sensitivity of the
  attention operation to quantization and the noise-compounding effects of
  autoregressive decoding in the high-cardinality output space. We approach the
  problem with a mix of statistics-based quantization for the weights and elastic
  quantization of the activations and demonstrate the first ternary and binary
  transformer models on the downstream tasks of summarization and machine
  translation. Our ternary BART base achieves an R1 score of 41 on the
  CNN/DailyMail benchmark, which is merely 3.9 points behind the full model while
  being 16x more efficient. Our binary model, while less accurate, achieves a
  highly non-trivial score of 35.6. For machine translation, we achieved BLEU
  scores of 21.7 and 17.6 on the WMT16 En-Ro benchmark, compared with a full
  precision mBART model score of 26.8. We also compare our approach in the 8-bit
  activation setting, where our ternary and even binary weight models can match
  or outperform the best existing 8-bit weight models in the literature. Our code
  and models are available at:
  https://github.com/facebookresearch/Ternary_Binary_Transformer
paper_ai_keywords: ternary neural networks binary neural networks multiplication-free computation parameter space discretization output space discretization optimization difficulties transformer text generation models attention operation quantization autoregressive decoding statistics-based quantization elastic quantization ternary transformer models binary transformer models R1 score CNN/DailyMail benchmark BLEU scores WMT16 En-Ro benchmark 8-bit activation setting