title: Enhancing Code Generation for Low-Resource Languages: No Silver Bullet
summary: The advent of Large Language Models (LLMs) has significantly advanced the
  field of automated code generation. LLMs rely on large and diverse datasets to
  learn syntax, semantics, and usage patterns of programming languages. For
  low-resource languages (i.e., niche programming languages characterized by the
  scarcity of training data), the limited availability of such data hampers the
  models' ability to generalize effectively, resulting in poorer code generation
  performance as compared to high-resource languages. For this reason, there is a
  quest for techniques able to close this performance gap. We present an
  empirical study investigating the effectiveness of several approaches for
  boosting LLMs' performance on low-resource languages, namely: (i) a classic
  fine-tuning, which is however capped in size by the scarcity of training data;
  (ii) three variants of in-context learning, with prompts crafted to provide the
  LLM with additional information about the low-resource language (e.g., few-shot
  examples showcasing features of the targeted language); and (iii) a
  pre-training objective teaching the model how to translate between high- and
  low-resource languages. The context of our study are two low-resource languages
  (R and Racket) and six LLMs having different architectures and sizes. Our
  findings reveal that a fine-tuning is usually the best choice for smaller LLMs,
  possibly due to the fact that even a small dataset is sufficient to train their
  limited number of parameters. With the increase in size of the models,
  in-context learning becomes more and more effective, representing a safe and
  cheap bet (i.e., it always helps, but with different magnitudes). Differently,
  very large LLMs may deteriorate their performance on low-resource languages
  when fine-tuning is performed, possibly due to the lack of enough data needed
  to effectively update their weights.
paper_authors._id: 67a5b65fe7798ca5b7473a45
paper_authors.name: Alessandro Giagnorio
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-07T09:57:46.514Z
paper_authors._id: 663486a1f64712540644cb68
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/663486a1f64712540644cb68/YZFR41ERY6UrC6rCC6Nan.jpeg
paper_authors.fullname: Alessandro
paper_authors.type: user
paper_authors.user: Devy1
paper_authors._id: 67a5b65fe7798ca5b7473a46
paper_authors.name: Alberto Martin-Lopez
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-07T10:46:32.306Z
paper_authors._id: 65a7cb0fc5ffe1d019a21cb3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/GShcO1DwVNzlIUr3n1ifi.jpeg
paper_authors.fullname: Alberto Martín López 
paper_authors.type: user
paper_authors.user: AML14
paper_authors._id: 67a5b65fe7798ca5b7473a47
paper_authors.name: Gabriele Bavota
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-07T10:46:38.124Z
paper_authors._id: 6638bea59e57161faac814e7
paper_authors.avatarUrl: /avatars/91375b88945af50e51b7229a789a31b8.svg
paper_authors.fullname: Gabriele Bavota
paper_authors.type: user
paper_authors.user: gbavota
paper_title: Enhancing Code Generation for Low-Resource Languages: No Silver Bullet
paper_summary: The advent of Large Language Models (LLMs) has significantly advanced the
  field of automated code generation. LLMs rely on large and diverse datasets to
  learn syntax, semantics, and usage patterns of programming languages. For
  low-resource languages (i.e., niche programming languages characterized by the
  scarcity of training data), the limited availability of such data hampers the
  models' ability to generalize effectively, resulting in poorer code generation
  performance as compared to high-resource languages. For this reason, there is a
  quest for techniques able to close this performance gap. We present an
  empirical study investigating the effectiveness of several approaches for
  boosting LLMs' performance on low-resource languages, namely: (i) a classic
  fine-tuning, which is however capped in size by the scarcity of training data;
  (ii) three variants of in-context learning, with prompts crafted to provide the
  LLM with additional information about the low-resource language (e.g., few-shot
  examples showcasing features of the targeted language); and (iii) a
  pre-training objective teaching the model how to translate between high- and
  low-resource languages. The context of our study are two low-resource languages
  (R and Racket) and six LLMs having different architectures and sizes. Our
  findings reveal that a fine-tuning is usually the best choice for smaller LLMs,
  possibly due to the fact that even a small dataset is sufficient to train their
  limited number of parameters. With the increase in size of the models,
  in-context learning becomes more and more effective, representing a safe and
  cheap bet (i.e., it always helps, but with different magnitudes). Differently,
  very large LLMs may deteriorate their performance on low-resource languages
  when fine-tuning is performed, possibly due to the lack of enough data needed
  to effectively update their weights.
paper_ai_summary: The study evaluates various techniques to enhance the performance of large language models on low-resource programming languages, finding that fine-tuning works best for smaller models, while in-context learning is more effective for larger models.
paper_ai_keywords: Large Language Models LLMs low-resource languages fine-tuning in-context learning few-shot examples pre-training objective R Racket parameter-efficient fine-tuning model weights