title: Investigating the Impact of Quantization Methods on the Safety and
    Reliability of Large Language Models
summary: Large Language Models (LLMs) have emerged as powerful tools for addressing
  modern challenges and enabling practical applications. However, their
  computational expense remains a significant barrier to widespread adoption.
  Quantization has emerged as a promising technique to democratize access and
  enable low resource device deployment. Despite these advancements, the safety
  and trustworthiness of quantized models remain underexplored, as prior studies
  often overlook contemporary architectures and rely on overly simplistic
  benchmarks and evaluations. To address this gap, we introduce OpenSafetyMini, a
  novel open-ended safety dataset designed to better distinguish between models.
  We evaluate 4 state-of-the-art quantization techniques across LLaMA and Mistral
  models using 4 benchmarks, including human evaluations. Our findings reveal
  that the optimal quantization method varies for 4-bit precision, while vector
  quantization techniques deliver the best safety and trustworthiness performance
  at 2-bit precision, providing foundation for future research.
paper_authors._id: 67bdf20b7c9bd4f09ebf05ac
paper_authors.name: Artyom Kharinaev
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-26T08:29:03.290Z
paper_authors._id: 64b4f8ba2fc8324fcb64c516
paper_authors.avatarUrl: /avatars/4cc7ab802a4e0c538c8ae1acb8192528.svg
paper_authors.fullname: Artyom Kharinaev
paper_authors.type: user
paper_authors.user: kharinaev
paper_authors._id: 67bdf20b7c9bd4f09ebf05ad
paper_authors.name: Viktor Moskvoretskii
paper_authors._id: 67bdf20b7c9bd4f09ebf05ae
paper_authors.name: Egor Shvetsov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-25T16:41:24.118Z
paper_authors._id: 65afde6ba0b4bf3b0e95b4e8
paper_authors.avatarUrl: /avatars/e9b97040b0a619bf6609465d1678705c.svg
paper_authors.fullname: Egor Shvetsov
paper_authors.type: user
paper_authors.user: dalime
paper_authors._id: 67bdf20b7c9bd4f09ebf05af
paper_authors.name: Kseniia Studenikina
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-26T08:29:00.879Z
paper_authors._id: 64c24ba6275309cb8bdab7ba
paper_authors.avatarUrl: /avatars/96cca2e5658c36c286d438e5d38f4c2f.svg
paper_authors.fullname: Kseniia Studenikina
paper_authors.type: user
paper_authors.user: Xeanst
paper_authors._id: 67bdf20b7c9bd4f09ebf05b0
paper_authors.name: Bykov Mikhail
paper_authors._id: 67bdf20b7c9bd4f09ebf05b1
paper_authors.name: Evgeny Burnaev
paper_title: Investigating the Impact of Quantization Methods on the Safety and
    Reliability of Large Language Models
paper_summary: Large Language Models (LLMs) have emerged as powerful tools for addressing
  modern challenges and enabling practical applications. However, their
  computational expense remains a significant barrier to widespread adoption.
  Quantization has emerged as a promising technique to democratize access and
  enable low resource device deployment. Despite these advancements, the safety
  and trustworthiness of quantized models remain underexplored, as prior studies
  often overlook contemporary architectures and rely on overly simplistic
  benchmarks and evaluations. To address this gap, we introduce OpenSafetyMini, a
  novel open-ended safety dataset designed to better distinguish between models.
  We evaluate 4 state-of-the-art quantization techniques across LLaMA and Mistral
  models using 4 benchmarks, including human evaluations. Our findings reveal
  that the optimal quantization method varies for 4-bit precision, while vector
  quantization techniques deliver the best safety and trustworthiness performance
  at 2-bit precision, providing foundation for future research.
paper_ai_keywords: quantization computational expense open-ended safety dataset state-of-the-art quantization techniques LLaMA Mistral benchmarks evaluations control vector quantization