title: Spectra: A Comprehensive Study of Ternary, Quantized, and FP16 Language
    Models
summary: Post-training quantization is the leading method for addressing
  memory-related bottlenecks in LLM inference, but unfortunately, it suffers from
  significant performance degradation below 4-bit precision. An alternative
  approach involves training compressed models directly at a low bitwidth (e.g.,
  binary or ternary models). However, the performance, training dynamics, and
  scaling trends of such models are not yet well understood. To address this
  issue, we train and openly release the Spectra LLM suite consisting of 54
  language models ranging from 99M to 3.9B parameters, trained on 300B tokens.
  Spectra includes FloatLMs, post-training quantized QuantLMs (3, 4, 6, and 8
  bits), and ternary LLMs (TriLMs) - our improved architecture for ternary
  language modeling, which significantly outperforms previously proposed ternary
  models of a given size (in bits), matching half-precision models at scale. For
  example, TriLM 3.9B is (bit-wise) smaller than the half-precision FloatLM 830M,
  but matches half-precision FloatLM 3.9B in commonsense reasoning and knowledge
  benchmarks. However, TriLM 3.9B is also as toxic and stereotyping as FloatLM
  3.9B, a model six times larger in size. Additionally, TriLM 3.9B lags behind
  FloatLM in perplexity on validation splits and web-based corpora but performs
  better on less noisy datasets like Lambada and PennTreeBank.
    To enhance understanding of low-bitwidth models, we are releasing 500+
  intermediate checkpoints of the Spectra suite at
  https://github.com/NolanoOrg/SpectraSuite{https://github.com/NolanoOrg/SpectraSuite}.
paper_authors._id: 669883b463880317128f22d7
paper_authors.name: Ayush Kaushal
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-02-14T13:18:36.792Z
paper_authors._id: 64d47258c4c5f16a0b6b52ba
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64d47258c4c5f16a0b6b52ba/5F48TqoBy-VSGlPUvMTmi.jpeg
paper_authors.fullname: Ayush Kaushal
paper_authors.type: user
paper_authors.user: Ayushk44
paper_authors._id: 669883b463880317128f22d8
paper_authors.name: Tejas Pandey
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-18T09:05:22.903Z
paper_authors._id: 6515aaf04eb20107d01c74f7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/vMoD0P3uYte3eUAlNhyGT.jpeg
paper_authors.fullname: Tejas Pandey
paper_authors.type: user
paper_authors.user: tejasexpress
paper_authors._id: 669883b463880317128f22d9
paper_authors.name: Tejas Vaidhya
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-19T11:23:44.622Z
paper_authors._id: 635c48e0de16046a3ff7c8e6
paper_authors.avatarUrl: /avatars/662bdb26a9aeabc70124e61dbd4c1c5a.svg
paper_authors.fullname: Tejas Vaidhya
paper_authors.type: user
paper_authors.user: tejasvaidhya
paper_authors._id: 669883b463880317128f22da
paper_authors.name: Aaryan Bhagat
paper_authors._id: 669883b463880317128f22db
paper_authors.name: Irina Rish
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-07-18T02:53:41.248Z
paper_authors._id: 6227f764236b7b2eefa7f173
paper_authors.avatarUrl: /avatars/0204e3da8e9ef12d88c98c6e955f8340.svg
paper_authors.fullname: Irina Rish
paper_authors.type: user
paper_authors.user: irinarish
paper_title: Spectra: A Comprehensive Study of Ternary, Quantized, and FP16 Language
    Models
paper_summary: Post-training quantization is the leading method for addressing
  memory-related bottlenecks in LLM inference, but unfortunately, it suffers from
  significant performance degradation below 4-bit precision. An alternative
  approach involves training compressed models directly at a low bitwidth (e.g.,
  binary or ternary models). However, the performance, training dynamics, and
  scaling trends of such models are not yet well understood. To address this
  issue, we train and openly release the Spectra LLM suite consisting of 54
  language models ranging from 99M to 3.9B parameters, trained on 300B tokens.
  Spectra includes FloatLMs, post-training quantized QuantLMs (3, 4, 6, and 8
  bits), and ternary LLMs (TriLMs) - our improved architecture for ternary
  language modeling, which significantly outperforms previously proposed ternary
  models of a given size (in bits), matching half-precision models at scale. For
  example, TriLM 3.9B is (bit-wise) smaller than the half-precision FloatLM 830M,
  but matches half-precision FloatLM 3.9B in commonsense reasoning and knowledge
  benchmarks. However, TriLM 3.9B is also as toxic and stereotyping as FloatLM
  3.9B, a model six times larger in size. Additionally, TriLM 3.9B lags behind
  FloatLM in perplexity on validation splits and web-based corpora but performs
  better on less noisy datasets like Lambada and PennTreeBank.
    To enhance understanding of low-bitwidth models, we are releasing 500+
  intermediate checkpoints of the Spectra suite at
  https://github.com/NolanoOrg/SpectraSuite{https://github.com/NolanoOrg/SpectraSuite}.
paper_ai_summary: The Spectra LLM suite provides insights into the performance and characteristics of low-bitwidth models through comprehensive training and evaluation of FloatLMs, QuantLMs, and TriLMs across various benchmarks and datasets.
paper_ai_keywords: post-training quantization low bitwidth binary models ternary models TriLMs ternary language modeling commonsense reasoning knowledge benchmarks toxicity stereotyping perplexity Lambada PennTreeBank