Investigating the Impact of Quantization Methods on the Safety and
  Reliability of Large Language Models
Large Language Models (LLMs) have emerged as powerful tools for addressing
modern challenges and enabling practical applications. However, their
computational expense remains a significant barrier to widespread adoption.
Quantization has emerged as a promising technique to democratize access and
enable low resource device deployment. Despite these advancements, the safety
and trustworthiness of quantized models remain underexplored, as prior studies
often overlook contemporary architectures and rely on overly simplistic
benchmarks and evaluations. To address this gap, we introduce OpenSafetyMini, a
novel open-ended safety dataset designed to better distinguish between models.
We evaluate 4 state-of-the-art quantization techniques across LLaMA and Mistral
models using 4 benchmarks, including human evaluations. Our findings reveal
that the optimal quantization method varies for 4-bit precision, while vector
quantization techniques deliver the best safety and trustworthiness performance
at 2-bit precision, providing foundation for future research.
67bdf20b7c9bd4f09ebf05ac
Artyom Kharinaev
claimed_verified
2025-02-26T08:29:03.290Z
64b4f8ba2fc8324fcb64c516
/avatars/4cc7ab802a4e0c538c8ae1acb8192528.svg
Artyom Kharinaev
user
kharinaev
67bdf20b7c9bd4f09ebf05ad
Viktor Moskvoretskii
67bdf20b7c9bd4f09ebf05ae
Egor Shvetsov
claimed_verified
2025-02-25T16:41:24.118Z
65afde6ba0b4bf3b0e95b4e8
/avatars/e9b97040b0a619bf6609465d1678705c.svg
Egor Shvetsov
user
dalime
67bdf20b7c9bd4f09ebf05af
Kseniia Studenikina
claimed_verified
2025-02-26T08:29:00.879Z
64c24ba6275309cb8bdab7ba
/avatars/96cca2e5658c36c286d438e5d38f4c2f.svg
Kseniia Studenikina
user
Xeanst
67bdf20b7c9bd4f09ebf05b0
Bykov Mikhail
67bdf20b7c9bd4f09ebf05b1
Evgeny Burnaev
Investigating the Impact of Quantization Methods on the Safety and
  Reliability of Large Language Models
Large Language Models (LLMs) have emerged as powerful tools for addressing
modern challenges and enabling practical applications. However, their
computational expense remains a significant barrier to widespread adoption.
Quantization has emerged as a promising technique to democratize access and
enable low resource device deployment. Despite these advancements, the safety
and trustworthiness of quantized models remain underexplored, as prior studies
often overlook contemporary architectures and rely on overly simplistic
benchmarks and evaluations. To address this gap, we introduce OpenSafetyMini, a
novel open-ended safety dataset designed to better distinguish between models.
We evaluate 4 state-of-the-art quantization techniques across LLaMA and Mistral
models using 4 benchmarks, including human evaluations. Our findings reveal
that the optimal quantization method varies for 4-bit precision, while vector
quantization techniques deliver the best safety and trustworthiness performance
at 2-bit precision, providing foundation for future research.
OpenSafetyMini, a novel safety dataset, evaluates quantization techniques across LLaMA and Mistral models, revealing varying optimal methods by precision and emphasizing vector quantization for safety and trustworthiness.
Large Language Models LLMs quantization OpenSafetyMini LLaMA Mistral vector quantization