Spectra: A Comprehensive Study of Ternary, Quantized, and FP16 Language
  Models
Post-training quantization is the leading method for addressing
memory-related bottlenecks in LLM inference, but unfortunately, it suffers from
significant performance degradation below 4-bit precision. An alternative
approach involves training compressed models directly at a low bitwidth (e.g.,
binary or ternary models). However, the performance, training dynamics, and
scaling trends of such models are not yet well understood. To address this
issue, we train and openly release the Spectra LLM suite consisting of 54
language models ranging from 99M to 3.9B parameters, trained on 300B tokens.
Spectra includes FloatLMs, post-training quantized QuantLMs (3, 4, 6, and 8
bits), and ternary LLMs (TriLMs) - our improved architecture for ternary
language modeling, which significantly outperforms previously proposed ternary
models of a given size (in bits), matching half-precision models at scale. For
example, TriLM 3.9B is (bit-wise) smaller than the half-precision FloatLM 830M,
but matches half-precision FloatLM 3.9B in commonsense reasoning and knowledge
benchmarks. However, TriLM 3.9B is also as toxic and stereotyping as FloatLM
3.9B, a model six times larger in size. Additionally, TriLM 3.9B lags behind
FloatLM in perplexity on validation splits and web-based corpora but performs
better on less noisy datasets like Lambada and PennTreeBank.
  To enhance understanding of low-bitwidth models, we are releasing 500+
intermediate checkpoints of the Spectra suite at
https://github.com/NolanoOrg/SpectraSuite{https://github.com/NolanoOrg/SpectraSuite}.
669883b463880317128f22d7
Ayush Kaushal
extracted_confirmed
2025-02-14T13:18:36.792Z
64d47258c4c5f16a0b6b52ba
https://cdn-avatars.huggingface.co/v1/production/uploads/64d47258c4c5f16a0b6b52ba/5F48TqoBy-VSGlPUvMTmi.jpeg
Ayush Kaushal
user
Ayushk44
669883b463880317128f22d8
Tejas Pandey
claimed_verified
2024-07-18T09:05:22.903Z
6515aaf04eb20107d01c74f7
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/vMoD0P3uYte3eUAlNhyGT.jpeg
Tejas Pandey
user
tejasexpress
669883b463880317128f22d9
Tejas Vaidhya
claimed_verified
2024-07-19T11:23:44.622Z
635c48e0de16046a3ff7c8e6
/avatars/662bdb26a9aeabc70124e61dbd4c1c5a.svg
Tejas Vaidhya
user
tejasvaidhya
669883b463880317128f22da
Aaryan Bhagat
669883b463880317128f22db
Irina Rish
extracted_pending
2024-07-18T02:53:41.248Z
6227f764236b7b2eefa7f173
/avatars/0204e3da8e9ef12d88c98c6e955f8340.svg
Irina Rish
user
irinarish
Spectra: A Comprehensive Study of Ternary, Quantized, and FP16 Language
  Models
Post-training quantization is the leading method for addressing
memory-related bottlenecks in LLM inference, but unfortunately, it suffers from
significant performance degradation below 4-bit precision. An alternative
approach involves training compressed models directly at a low bitwidth (e.g.,
binary or ternary models). However, the performance, training dynamics, and
scaling trends of such models are not yet well understood. To address this
issue, we train and openly release the Spectra LLM suite consisting of 54
language models ranging from 99M to 3.9B parameters, trained on 300B tokens.
Spectra includes FloatLMs, post-training quantized QuantLMs (3, 4, 6, and 8
bits), and ternary LLMs (TriLMs) - our improved architecture for ternary
language modeling, which significantly outperforms previously proposed ternary
models of a given size (in bits), matching half-precision models at scale. For
example, TriLM 3.9B is (bit-wise) smaller than the half-precision FloatLM 830M,
but matches half-precision FloatLM 3.9B in commonsense reasoning and knowledge
benchmarks. However, TriLM 3.9B is also as toxic and stereotyping as FloatLM
3.9B, a model six times larger in size. Additionally, TriLM 3.9B lags behind
FloatLM in perplexity on validation splits and web-based corpora but performs
better on less noisy datasets like Lambada and PennTreeBank.
  To enhance understanding of low-bitwidth models, we are releasing 500+
intermediate checkpoints of the Spectra suite at
https://github.com/NolanoOrg/SpectraSuite{https://github.com/NolanoOrg/SpectraSuite}.
The Spectra LLM suite provides insights into the performance and characteristics of low-bitwidth models through comprehensive training and evaluation of FloatLMs, QuantLMs, and TriLMs across various benchmarks and datasets.
post-training quantization low bitwidth binary models ternary models TriLMs ternary language modeling commonsense reasoning knowledge benchmarks toxicity stereotyping perplexity Lambada PennTreeBank