{"data":{"slug":"nuprl-multipl-e","name":"MultiPL-E","tagline":"A multi-programming language benchmark for LLMs","github_url":"https://github.com/nuprl/MultiPL-E","owner":"nuprl","repo":"MultiPL-E","owner_avatar_url":"https://avatars.githubusercontent.com/u/4161665?v=4","primary_language":"Python","stars":311,"forks":57,"topics":[],"archived":false,"github_pushed_at":"2026-04-12T16:59:02+00:00","maintenance_label":"Slowing","url":"https://www.graphcanon.com/tools/nuprl-multipl-e","markdown_url":"https://www.graphcanon.com/tools/nuprl-multipl-e.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/nuprl-multipl-e","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=nuprl-multipl-e","description":"A multi-programming language benchmark for LLMs","homepage_url":null,"license":"Other","open_issues":16,"watchers":13,"ai_summary":null,"readme_excerpt":"# Multi-Programming Language Evaluation of Large Language Models of Code (MultiPL-E)\n\n**New**: For a more challenging multi-language benchmark, check out [Ag-LiveCodeBench-X](https://github.com/nuprl/Ag-LiveCodeBench-X)\nand its accompanying paper, [Agnostics](https://arxiv.org/abs/2508.04865).\n\n## Introduction\n\nMultiPL-E is a system for translating unit test-driven neural code generation\nbenchmarks to new languages. We have used MultiPL-E to translate two popular\nPython benchmarks (HumanEval and MBPP) to 18 other programming languages.\n\nFor more information:\n\n- MultiPL-E is part of the [BigCode Code Generation LM Harness]. This\n  is the easiest way to use MultiPL-E.\n- The [Multilingual Code Models Evaluation] by BigCode evaluates Code LLMs\n  using several benchmarks, including MultiPL-E.\n- Read our paper [MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation].\n- The [MultiPL-E dataset] of translated prompts is available on the Hugging Face\n  Hub.\n\n## Tutorial\n\nThese are instructions on how to use MultiPl-E directly, without the\nBigCode evaluation harness.\n\nIn this tutorial, we will run a small experiment to evaluate the performance of\n[SantaCoder] on Rust with a small subset of the MBPP benchmarks.\nWe will only fetch 20 completions per problem, so that you\ncan run it quickly on a single machine.\nYou can also run on the full suite of benchmarks or substitute your own\nbenchmark programs. Later, we'll show you how to add support for other languages\nand evaluate other models.\n\n### Prerequisites\n\n1. You will need Python 3.8 or higher.\n\n2. You will need to install some Python packages:\n\n    ```bash\n    pip3 install aiohttp numpy tqdm pytest datasets torch transformers\n    ```\n\n3. You need to install one of [Podman] or [Docker].\n\n3. Check out the repository:\n\n   ```bash\n   git clone https://github.com/nuprl/MultiPL-E\n   ```\n\n4. Enter the repository directory:\n\n   ```bash\n   cd MultiPL-E\n   ```\n\n### Background\n\nOut of the box, MultiPL-E supports several models, programming languages,\nand datasets.  Using MultiPL-E is a two step process:\n\n1. We *generate* completions, which requires a GPU.\n\n2. We *execute* the generated completions, which requires a machine that\n   supports Docker or Podman.\n\n### Generation\n\n**The following directions are for evaluating base models. If you want to\nevaluate a chat model, see [chat_completions.py](chat_completions.py).\n\nThe following command will generate completions for the HumanEval benchmark,\nwhich is originally in Python, but translated to Rust with MultiPL-E:\n\n```\nmkdir tutorial\npython3 automodel.py \\\n    --name bigcode/gpt_bigcode-santacoder \\\n    --root-dataset humaneval \\\n    --lang rs \\\n    --temperature 0.2 \\\n    --batch-size 20 \\\n    --completion-limit 20 \\\n    --output-dir-prefix tutorial\n```\n\nThe model name above refers to the\n[SantaCoder](https://huggingface.co/bigcode/gpt_bigcode-santacoder) model on the\nHugging Face Hub. You can use any other text generation model instead.\n\nNotes:\n\n1. This command requires about 13 GB VRAM and takes 30 minutes with a Quadro RTX\n   6000.\n2. If you have less VRAM, you can set `--batch-size` to a smaller value.\n   E.g., with `--batch-size 10` it should work on consumer graphics cards,\n   such as the RTX series cards.\n3. If you're feeling impatient, you can kill the command early (use `Ctrl+C`)\n   before all generations are complete. Your results won't be accurate,\n   but you can move on to the evaluation step to get a partial result. Before\n   killing generation, ensure that a few files have been generated:\n\n   ```bash\n   ls tutorial/*/*.json.gz\n   ```\n\n### Execution\n\nYou can run MultiPL-E's execution with or without a container, but we strongly\nrecommend using the container that we have provided. The container includes the\ntoolchains for all languages that we support. Without it, you will need to\npainstakingly install them again. There is also a risk that the generated code\nmay do something that breaks your system. The container","github_created_at":"2022-07-25T14:07:20+00:00","created_at":"2026-07-11T23:46:32.885796+00:00","updated_at":"2026-07-11T23:46:38.174061+00:00","categories":[{"slug":"evaluation-observability","name":"Evaluation & Observability","url":"https://www.graphcanon.com/categories/evaluation-observability","markdown_url":"https://www.graphcanon.com/categories/evaluation-observability.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/evaluation-observability"},{"slug":"llm-frameworks","name":"LLM Frameworks","url":"https://www.graphcanon.com/categories/llm-frameworks","markdown_url":"https://www.graphcanon.com/categories/llm-frameworks.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/llm-frameworks"},{"slug":"model-training","name":"Model Training","url":"https://www.graphcanon.com/categories/model-training","markdown_url":"https://www.graphcanon.com/categories/model-training.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/model-training"}],"tags":[{"slug":"python","name":"python"}],"trust":{"provenance":{"is_fork":false,"github_id":517691031,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-07-11T23:46:33.778Z","maintenance":{"label":"Slowing","score":36,"methodology":"github_public_v1","releases_90d":0,"days_since_push":90,"last_release_at":null},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T23:46:34.301Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-07-11T23:46:33.542Z"},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-07-11T23:46:33.542Z"},"license_spdx":{"value":"Other","source":"github.license","observed_at":"2026-07-11T23:46:33.542Z"}}}}