Multi-Programming Language Evaluation of Large Language Models of Code (MultiPL-E)

New: For a more challenging multi-language benchmark, check out Ag-LiveCodeBench-X and its accompanying paper, Agnostics.

Introduction

MultiPL-E is a system for translating unit test-driven neural code generation benchmarks to new languages. We have used MultiPL-E to translate two popular Python benchmarks (HumanEval and MBPP) to 18 other programming languages.

For more information:

• MultiPL-E is part of the [BigCode Code Generation LM Harness]. This is the easiest way to use MultiPL-E.
• The [Multilingual Code Models Evaluation] by BigCode evaluates Code LLMs using several benchmarks, including MultiPL-E.
• Read our paper [MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation].
• The [MultiPL-E dataset] of translated prompts is available on the Hugging Face Hub.

Tutorial

These are instructions on how to use MultiPl-E directly, without the BigCode evaluation harness.

In this tutorial, we will run a small experiment to evaluate the performance of [SantaCoder] on Rust with a small subset of the MBPP benchmarks. We will only fetch 20 completions per problem, so that you can run it quickly on a single machine. You can also run on the full suite of benchmarks or substitute your own benchmark programs. Later, we'll show you how to add support for other languages and evaluate other models.

Prerequisites

1. You will need Python 3.8 or higher.

2. You will need to install some Python packages:

   pip3 install aiohttp numpy tqdm pytest datasets torch transformers
   

3. You need to install one of [Podman] or [Docker].

4. Check out the repository:

   git clone https://github.com/nuprl/MultiPL-E
   

5. Enter the repository directory:

   cd MultiPL-E
   

Background

Out of the box, MultiPL-E supports several models, programming languages, and datasets. Using MultiPL-E is a two step process:

1. We generate completions, which requires a GPU.

2. We execute the generated completions, which requires a machine that supports Docker or Podman.

Generation

**The following directions are for evaluating base models. If you want to evaluate a chat model, see chat_completions.py.

The following command will generate completions for the HumanEval benchmark, which is originally in Python, but translated to Rust with MultiPL-E:

mkdir tutorial
python3 automodel.py \
    --name bigcode/gpt_bigcode-santacoder \
    --root-dataset humaneval \
    --lang rs \
    --temperature 0.2 \
    --batch-size 20 \
    --completion-limit 20 \
    --output-dir-prefix tutorial

The model name above refers to the SantaCoder model on the Hugging Face Hub. You can use any other text generation model instead.

Notes:

1. This command requires about 13 GB VRAM and takes 30 minutes with a Quadro RTX 6000.

2. If you have less VRAM, you can set --batch-size to a smaller value. E.g., with --batch-size 10 it should work on consumer graphics cards, such as the RTX series cards.

3. If you're feeling impatient, you can kill the command early (use Ctrl+C) before all generations are complete. Your results won't be accurate, but you can move on to the evaluation step to get a partial result. Before killing generation, ensure that a few files have been generated:

   ls tutorial/*/*.json.gz
   

Execution

You can run MultiPL-E's execution with or without a container, but we strongly recommend using the container that we have provided. The container includes the toolchains for all languages that we support. Without it, you will need to painstakingly install them again. There is also a risk that the generated code may do something that breaks your system. The container