title: BioCoder: A Benchmark for Bioinformatics Code Generation with Contextual
    Pragmatic Knowledge
summary: Pre-trained language models like ChatGPT have significantly improved code
  generation. As these models scale up, there is an increasing need for the
  output to handle more intricate tasks. Moreover, in bioinformatics, generating
  functional programs poses additional notable challenges due to the amount of
  domain knowledge, the need for complicated data operations, and intricate
  functional dependencies between the operations. Here, we present BioCoder, a
  benchmark developed to evaluate existing pre-trained models in generating
  bioinformatics code. In relation to function-code generation, BioCoder covers
  potential package dependencies, class declarations, and global variables. It
  incorporates 1026 functions and 1243 methods in Python and Java from GitHub and
  253 examples from the Rosalind Project. BioCoder incorporates a fuzz-testing
  framework for evaluation, and we have applied it to evaluate many models
  including InCoder, CodeGen, CodeGen2, SantaCoder, StarCoder, StarCoder+,
  InstructCodeT5+, and ChatGPT. Our detailed analysis of these models emphasizes
  the importance of domain knowledge, pragmatic code generation, and contextual
  understanding. Our dataset, benchmark, Docker images, and scripts required for
  testing are all available at https://github.com/gersteinlab/biocoder.
paper_authors._id: 64f1581a3bdfd057d4567a17
paper_authors.name: Xiangru Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-01T12:35:58.595Z
paper_authors._id: 64709e80d9360cd9d8e7073d
paper_authors.avatarUrl: /avatars/ec37977b9a10e823e0d5bb73d152f253.svg
paper_authors.fullname: tang xiangrun
paper_authors.type: user
paper_authors.user: stxrun
paper_authors._id: 64f1581a3bdfd057d4567a18
paper_authors.name: Bill Qian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-01T12:36:33.046Z
paper_authors._id: 641fa2e0863b87326f4a7fd5
paper_authors.avatarUrl: /avatars/7b29c20f942d83c08f63f04a85c15c13.svg
paper_authors.fullname: Bill Qian
paper_authors.type: user
paper_authors.user: lilbillbiscuit
paper_authors._id: 64f1581a3bdfd057d4567a19
paper_authors.name: Rick Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-01T12:36:41.741Z
paper_authors._id: 649e6ead35032bf979eded1c
paper_authors.avatarUrl: /avatars/c2b5bc24abd5d28bcf29b01f185fbe67.svg
paper_authors.fullname: Rick Gao
paper_authors.type: user
paper_authors.user: rikkii
paper_authors._id: 64f1581a3bdfd057d4567a1a
paper_authors.name: Jiakang Chen
paper_authors._id: 64f1581a3bdfd057d4567a1b
paper_authors.name: Xinyun Chen
paper_authors._id: 64f1581a3bdfd057d4567a1c
paper_authors.name: Mark Gerstein
paper_title: BioCoder: A Benchmark for Bioinformatics Code Generation with Contextual
    Pragmatic Knowledge
paper_summary: Pre-trained language models like ChatGPT have significantly improved code
  generation. As these models scale up, there is an increasing need for the
  output to handle more intricate tasks. Moreover, in bioinformatics, generating
  functional programs poses additional notable challenges due to the amount of
  domain knowledge, the need for complicated data operations, and intricate
  functional dependencies between the operations. Here, we present BioCoder, a
  benchmark developed to evaluate existing pre-trained models in generating
  bioinformatics code. In relation to function-code generation, BioCoder covers
  potential package dependencies, class declarations, and global variables. It
  incorporates 1026 functions and 1243 methods in Python and Java from GitHub and
  253 examples from the Rosalind Project. BioCoder incorporates a fuzz-testing
  framework for evaluation, and we have applied it to evaluate many models
  including InCoder, CodeGen, CodeGen2, SantaCoder, StarCoder, StarCoder+,
  InstructCodeT5+, and ChatGPT. Our detailed analysis of these models emphasizes
  the importance of domain knowledge, pragmatic code generation, and contextual
  understanding. Our dataset, benchmark, Docker images, and scripts required for
  testing are all available at https://github.com/gersteinlab/biocoder.
paper_ai_keywords: pre-trained language models code generation bioinformatics functional programs domain knowledge data operations functional dependencies BioCoder function-code generation package dependencies class declarations global variables Python Java GitHub Rosalind Project fuzz-testing framework InCoder CodeGen CodeGen2 SantaCoder StarCoder StarCoder+ InstructCodeT5+ ChatGPT pragmatic code generation contextual understanding