title: Quantizing Large Language Models for Code Generation: A Differentiated
    Replication
summary: Large Language Models (LLMs) have shown an impressive capability in code
  generation and, specifically, to automatically implement requirements described
  in natural language. The LLM effectiveness generally increases with its size:
  The higher the number of LLM's trainable parameters the better its ability to
  implement code. However, when it comes to deploying LLM-based code generators,
  larger LLMs pose significant challenges related to their memory (and,
  consequently, carbon) footprint. A previous work by Wei et al. proposed to
  leverage quantization techniques to reduce the memory footprint of LLM-based
  code generators without substantially degrading their effectiveness. In short,
  they studied LLMs featuring up to 16B parameters, quantizing their precision
  from floating point 32 bits down to int 8 bits and showing their limited impact
  on code generation performance. Given the fast pace at which LLM capabilities
  and quantization techniques are evolving, in this work we present a
  differentiated replication of the work by Wei et al. in which we consider (i)
  on the one side, more recent and larger code-related LLMs, of up to 34B
  parameters; (ii) the latest advancements in model quantization techniques,
  which allow pushing the compression to the extreme quantization level of 2 bits
  per model parameter and; (iii) different types of calibration datasets to guide
  the quantization process, including code-specific ones. Our empirical
  evaluation reveals that the new frontier for LLM quantization is 4-bit
  precision, resulting in an average memory footprint reduction of 70% compared
  to the original model without observing any significant decrease in
  performance. Additionally, when the quantization becomes even more extreme (3
  and 2 bits), a code-specific calibration dataset helps to limit the loss of
  performance.
paper_authors._id: 67d27d4c6fbfb8ee21886b60
paper_authors.name: Alessandro Giagnorio
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-13T08:23:50.840Z
paper_authors._id: 663486a1f64712540644cb68
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/663486a1f64712540644cb68/YZFR41ERY6UrC6rCC6Nan.jpeg
paper_authors.fullname: Alessandro
paper_authors.type: user
paper_authors.user: Devy1
paper_authors._id: 67d27d4c6fbfb8ee21886b61
paper_authors.name: Antonio Mastropaolo
paper_authors._id: 67d27d4c6fbfb8ee21886b62
paper_authors.name: Saima Afrin
paper_authors._id: 67d27d4c6fbfb8ee21886b63
paper_authors.name: Massimiliano Di Penta
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-03-13T06:38:05.744Z
paper_authors._id: 63f378004745321de351a554
paper_authors.avatarUrl: /avatars/3be79f0f6eb0bcfdc9f31b46d2bafc14.svg
paper_authors.fullname: Max Di Penta
paper_authors.type: user
paper_authors.user: mdiipenta
paper_authors._id: 67d27d4c6fbfb8ee21886b64
paper_authors.name: Gabriele Bavota
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-06T16:53:43.585Z
paper_authors._id: 6638bea59e57161faac814e7
paper_authors.avatarUrl: /avatars/91375b88945af50e51b7229a789a31b8.svg
paper_authors.fullname: Gabriele Bavota
paper_authors.type: user
paper_authors.user: gbavota
paper_title: Quantizing Large Language Models for Code Generation: A Differentiated
    Replication
paper_summary: Large Language Models (LLMs) have shown an impressive capability in code
  generation and, specifically, to automatically implement requirements described
  in natural language. The LLM effectiveness generally increases with its size:
  The higher the number of LLM's trainable parameters the better its ability to
  implement code. However, when it comes to deploying LLM-based code generators,
  larger LLMs pose significant challenges related to their memory (and,
  consequently, carbon) footprint. A previous work by Wei et al. proposed to
  leverage quantization techniques to reduce the memory footprint of LLM-based
  code generators without substantially degrading their effectiveness. In short,
  they studied LLMs featuring up to 16B parameters, quantizing their precision
  from floating point 32 bits down to int 8 bits and showing their limited impact
  on code generation performance. Given the fast pace at which LLM capabilities
  and quantization techniques are evolving, in this work we present a
  differentiated replication of the work by Wei et al. in which we consider (i)
  on the one side, more recent and larger code-related LLMs, of up to 34B
  parameters; (ii) the latest advancements in model quantization techniques,
  which allow pushing the compression to the extreme quantization level of 2 bits
  per model parameter and; (iii) different types of calibration datasets to guide
  the quantization process, including code-specific ones. Our empirical
  evaluation reveals that the new frontier for LLM quantization is 4-bit
  precision, resulting in an average memory footprint reduction of 70% compared
  to the original model without observing any significant decrease in
  performance. Additionally, when the quantization becomes even more extreme (3
  and 2 bits), a code-specific calibration dataset helps to limit the loss of
  performance.
paper_ai_summary: Replication of LLM quantization strategies reveals that 4-bit precision can reduce memory footprint by 70% for large code-related LLMs with up to 34B parameters without significant performance degradation.
paper_ai_keywords: Large Language Models (LLMs) code generation trainable parameters memory footprint carbon footprint quantization techniques precision calibration datasets