BioCoder: A Benchmark for Bioinformatics Code Generation with Large Language Models

📄 Paper 🌐 Website 📜 Dataset

BioCoder is a challenging bioinformatics code generation benchmark for examining the capabilities of state-of-the-art large language models (LLMs).

Project Structure

The repository comprises 4 main parts: evaluation, inference, parsing, and data. We use the data folder to store results between each step in the operations. Ensure that each step is run from the root directory, i.e. python3 parsing/parse.py and not cd parsing && python3 parse.py in order to avoid issues with relative paths.

Each section is described below.

Parsing

The parsing section contains the code used to analyze, parse, and extract functions from the repositories.

Begin the process by going through a list of GitHub repositories. Download the list of repositories (named pone.0205898.s005.xlsx) from this paper first, then place it in the data directory.

A large-scale analysis of bioinformatics code on GitHub

Then, open the notebook in GithubAnalysis to start the repository metadata-gathering process. Note that this may fully clone all available GitHub repositories in the list. This may take a long time. Then, use the scripts in FunctionSelection to filter the GitHub functions accordingly.

Next, we need to do the function extraction step. We have created parsers for Java and Python so far. The corresponding documentation can be found in the respective folders:

• Java Parsing
• Python Parsing

After parsing, the output should be a JSON format with consistent properties across all languages. The following columns are guaranteed, however, they may be extra columns for specific languages:

• function_id: A unique identifier for the function. This is the hashcode of the function's signature.
• numParams: The number of parameters the function has.
• numLines: The number of lines the function spans.
• numChars: The number of characters the function spans.
• lineStart: The line number the function starts on.
• lineEnd: The line number the function ends on.
• returnType: The return type of the function.
• params: A list of the function's parameters.
• parentClass: The class the function is in.
• filePath: The path to the file the function is in.
• signature: The function's signature.
• content: The function's content.
• comment: The function's comment.
• numCommentLines: The number of lines the function's comment spans.
• numCommentChars: The number of characters the function's comment spans.
• packageName: The package the function is in.
• repoName: The repository the function is in.
• imports: A list of the function's imports.
• additionalImports: A list of the function's additional imports.
• intraClassFieldsUsed: A list of the function's intra-class fields used.
• intraClassFieldsUsed: A list of the function's intra-class fields used.

Then, use the scripts in FunctionProcessing to process the functions. This includes filtering out functions that are too short, too long, or have too many parameters. It also includes filtering out functions that are too similar to each other. It also includes scripts to generate summaries, prompts, and other necessary prompt data with ChatGPT

Note that there are also some scripts that assist with the manual annotation of context generation. These are not necessary for the benchmark, but are included for completeness.

Custom Python Parsing

In order to perform custom parsing on a Python repository, one can use the parse.py file under `parsing/FunctionExt