title: MPIrigen: MPI Code Generation through Domain-Specific Language Models
summary: The imperative need to scale computation across numerous nodes highlights the
  significance of efficient parallel computing, particularly in the realm of
  Message Passing Interface (MPI) integration. The challenging parallel
  programming task of generating MPI-based parallel programs has remained
  unexplored. This study first investigates the performance of state-of-the-art
  language models in generating MPI-based parallel programs. Findings reveal that
  widely used models such as GPT-3.5 and PolyCoder (specialized multi-lingual
  code models) exhibit notable performance degradation, when generating MPI-based
  programs compared to general-purpose programs. In contrast, domain-specific
  models such as MonoCoder, which are pretrained on MPI-related programming
  languages of C and C++, outperform larger models. Subsequently, we introduce a
  dedicated downstream task of MPI-based program generation by fine-tuning
  MonoCoder on HPCorpusMPI. We call the resulting model as MPIrigen. We propose
  an innovative preprocessing for completion only after observing the whole code,
  thus enabling better completion with a wider context. Comparative analysis
  against GPT-3.5 zero-shot performance, using a novel HPC-oriented evaluation
  method, demonstrates that MPIrigen excels in generating accurate MPI functions
  up to 0.8 accuracy in location and function predictions, and with more than 0.9
  accuracy for argument predictions. The success of this tailored solution
  underscores the importance of domain-specific fine-tuning in optimizing
  language models for parallel computing code generation, paving the way for a
  new generation of automatic parallelization tools. The sources of this work are
  available at our GitHub MPIrigen repository:
  https://github.com/Scientific-Computing-Lab-NRCN/MPI-rigen
paper_authors._id: 65cd99f61b2738e61ebbce18
paper_authors.name: Nadav Schneider
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-15T17:28:28.001Z
paper_authors._id: 65ce0b4a03a8179f5da5d4ef
paper_authors.avatarUrl: /avatars/c0b68efb885486c5180d1d69c4e317ac.svg
paper_authors.fullname: Nadav Schneider
paper_authors.type: user
paper_authors.user: nadavsc
paper_authors._id: 65cd99f61b2738e61ebbce19
paper_authors.name: Niranjan Hasabnis
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-02-15T04:58:30.887Z
paper_authors._id: 650dd3a6c945dfc9385e52d7
paper_authors.avatarUrl: /avatars/c9a41b4cb4733f136d4a65e3d33693c4.svg
paper_authors.fullname: Niranjan Hasabnis
paper_authors.type: user
paper_authors.user: nhasabni
paper_authors._id: 65cd99f61b2738e61ebbce1a
paper_authors.name: Vy A. Vo
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-02-15T04:58:30.887Z
paper_authors._id: 62fabd8920be129c47683fe4
paper_authors.avatarUrl: /avatars/c5ce60cb85d1626f61679042d51aa427.svg
paper_authors.fullname: Vy Vo
paper_authors.type: user
paper_authors.user: vyaivo
paper_authors._id: 65cd99f61b2738e61ebbce1b
paper_authors.name: Tal Kadosh
paper_authors._id: 65cd99f61b2738e61ebbce1c
paper_authors.name: Neva Krien
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-02-15T04:58:30.887Z
paper_authors._id: 63a325778c0c89dcae33484b
paper_authors.avatarUrl: /avatars/3eca79d6a0a00652c906e30a83d4f8af.svg
paper_authors.fullname: נבו קרין
paper_authors.type: user
paper_authors.user: nevakrien
paper_authors._id: 65cd99f61b2738e61ebbce1d
paper_authors.name: Mihai Capotă
paper_authors._id: 65cd99f61b2738e61ebbce1e
paper_authors.name: Abdul Wasay
paper_authors._id: 65cd99f61b2738e61ebbce1f
paper_authors.name: Guy Tamir
paper_authors._id: 65cd99f61b2738e61ebbce20
paper_authors.name: Ted Willke
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-02-16T01:44:55.610Z
paper_authors._id: 65cebdbce0723b01a3774e8c
paper_authors.avatarUrl: /avatars/5ac01078ab89b3525fbb5d0069e16607.svg
paper_authors.fullname: Ted Willke
paper_authors.type: user
paper_authors.user: tlwillke
paper_authors._id: 65cd99f61b2738e61ebbce21
paper_authors.name: Nesreen Ahmed
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-01-07T07:13:11.063Z
paper_authors._id: 663e588ccca86c8371a913d9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/663e588ccca86c8371a913d9/wgVGw5U77cqwtjovhsXyR.jpeg
paper_authors.fullname: Nesreen Ahmed
paper_authors.type: user
paper_authors.user: nkahmed
paper_authors._id: 65cd99f61b2738e61ebbce22
paper_authors.name: Yuval Pinter
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-15T17:28:23.651Z
paper_authors._id: 65562ab891ec2c213fba90d9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/pywmO-m8WhvkQFTqZmCum.jpeg
paper_authors.fullname: Yuval Pinter
paper_authors.type: user
paper_authors.user: yuvalpinter
paper_authors._id: 65cd99f61b2738e61ebbce23
paper_authors.name: Timothy Mattson
paper_authors._id: 65cd99f61b2738e61ebbce24
paper_authors.name: Gal Oren
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-15T17:28:25.926Z
paper_authors._id: 65ce14f4167be8a5f3e7646e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/nzeXCf5QQKLlFaich-d1u.jpeg
paper_authors.fullname: Gal Oren
paper_authors.type: user
paper_authors.user: GalOren8
paper_title: MPIrigen: MPI Code Generation through Domain-Specific Language Models
paper_summary: The imperative need to scale computation across numerous nodes highlights the
  significance of efficient parallel computing, particularly in the realm of
  Message Passing Interface (MPI) integration. The challenging parallel
  programming task of generating MPI-based parallel programs has remained
  unexplored. This study first investigates the performance of state-of-the-art
  language models in generating MPI-based parallel programs. Findings reveal that
  widely used models such as GPT-3.5 and PolyCoder (specialized multi-lingual
  code models) exhibit notable performance degradation, when generating MPI-based
  programs compared to general-purpose programs. In contrast, domain-specific
  models such as MonoCoder, which are pretrained on MPI-related programming
  languages of C and C++, outperform larger models. Subsequently, we introduce a
  dedicated downstream task of MPI-based program generation by fine-tuning
  MonoCoder on HPCorpusMPI. We call the resulting model as MPIrigen. We propose
  an innovative preprocessing for completion only after observing the whole code,
  thus enabling better completion with a wider context. Comparative analysis
  against GPT-3.5 zero-shot performance, using a novel HPC-oriented evaluation
  method, demonstrates that MPIrigen excels in generating accurate MPI functions
  up to 0.8 accuracy in location and function predictions, and with more than 0.9
  accuracy for argument predictions. The success of this tailored solution
  underscores the importance of domain-specific fine-tuning in optimizing
  language models for parallel computing code generation, paving the way for a
  new generation of automatic parallelization tools. The sources of this work are
  available at our GitHub MPIrigen repository:
  https://github.com/Scientific-Computing-Lab-NRCN/MPI-rigen
paper_ai_summary: Domain-specific fine-tuning of MonoCoder on MPI-related code significantly improves the generation of MPI-based parallel programs compared to general-purpose models.
paper_ai_keywords: GPT-3.5 PolyCoder MonoCoder HPCorpusMPI MPIrigen MPI-based programs parallel computing code generation domain-specific fine-tuning HPC-oriented evaluation MPI functions argument predictions