{"data":{"slug":"bytedance-fullstackbench","name":"FullStackBench","tagline":"Official repository for our paper \"FullStack Bench: Evaluating LLMs as Full Stack Coders\"","github_url":"https://github.com/bytedance/FullStackBench","owner":"bytedance","repo":"FullStackBench","owner_avatar_url":"https://avatars.githubusercontent.com/u/4158466?v=4","primary_language":"Python","stars":121,"forks":9,"topics":["research"],"archived":false,"github_pushed_at":"2025-05-07T09:30:55+00:00","maintenance_label":"Dormant","url":"https://www.graphcanon.com/tools/bytedance-fullstackbench","markdown_url":"https://www.graphcanon.com/tools/bytedance-fullstackbench.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/bytedance-fullstackbench","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=bytedance-fullstackbench","description":"Official repository for our paper \"FullStack Bench: Evaluating LLMs as Full Stack Coders\"","homepage_url":null,"license":"Apache-2.0","open_issues":1,"watchers":3,"ai_summary":null,"readme_excerpt":"

FullStack Bench: Evaluating LLMs as Full Stack Coders

\n\n
\n \n \"Build\"\n \n \n \"Code\n \n \n \"Data\n \n
\n\n
\nOfficial repository for our paper \"FullStack Bench: Evaluating LLMs as Full Stack Coders\"\n
\n\n

\n 🏠 FullStack Bench Code •\n 📊 Benchmark Data •\n 📚 SandboxFusion \n

\n\n## Table of contents\n- [FullStack Bench: Evaluating LLMs as Full Stack Coders](#Introduction)\n - [📌 Introduction](#introduction)\n - [📚 SandboxFusion](#leaderboard)\n - [📊 Data](#data)\n - [💻 Usage](#usage)\n - [📖 Citation](#citation)\n\n## 📌Introduction\n**FullStack Bench** is a multilingual benchmark for full-stack programming, covering a wide range of application domains and **16** programming languages with **3K** test samples, which substantially pushes the limits of code LLMs in code-related abilities of the real-world code development scenarios.\n

\n\"FullStack\n

\n\n### Task Examples\n**FullStack Bench** covers more mainstream application domains when compared to existing code\nevaluation benchmarks. Here is a visualization example from FullStack Bench, where the model is tasked with solving problems in the domain of desktop and web development using HTML.\n

\n\"FullStack\n

\n\nRefer to our paper or dataset for more details. \n\n### Results\n

\n\"results\"\n

\nRefer to our paper for more results.\n\n## 📚SandboxFusion\n**SandboxFusion** is an an effective code sandbox execution tool to evaluate different programming tasks from different languages. It incorporates over 10 coding-related evaluation datasets, featuring a standardized data format and accessible via a uniform HTTP API.\n

\n\"FullStack\n

\nRefer to our paper and 📚 Tutorial for more Details.\n\n## 📊Data\n
\n\n| **Dataset** | **Download** |\n| :------------: | :------------: |\n| FullStack Bench Dataset | [🤗 HuggingFace](https://huggingface.co/datasets/ByteDance/FullStackBench) |\n\n
\n\n## 💻Usage\nStart the [sandbox server](https://bytedance.github.io/SandboxFusion/):\n\n```bash\ndocker run -d --rm -p 8080:8080 volcengine/sandbox-fusion:server-20241204\n```\n\nFor users in mainland China, the following mirror is provided:\n\n```bash\ndocker run -d --rm -p 8080:8080 vemlp-cn-beijing.cr.volces.com/preset-images/code-sandbox:server-20241204\n```\n\nThen, run the benchmark:\n\n```bash\ngit clone https://github.com/bytedance/FullStackBench.git\ncd FullStackBench\npip install -r requirements.txt\n# modify the model configs in src/main.py\npython src/main.py\n```\n## 📖Citation\nIf you find our work helpful, please use the following citations.\n```\n@misc{liu2024fullstackbenchevaluatingllms,\n title={FullStack Bench: Evaluating LLMs as Full Stack Coders}, \n author={Siyao Liu and He Zhu and Jerry Liu and Shulin Xin and Aoyan Li and Rui Long and Li Chen and Jack Yang","github_created_at":"2024-11-28T06:25:15+00:00","created_at":"2026-07-11T23:45:24.46024+00:00","updated_at":"2026-07-11T23:45:31.052871+00:00","categories":[{"slug":"llm-frameworks","name":"LLM Frameworks","url":"https://www.graphcanon.com/categories/llm-frameworks","markdown_url":"https://www.graphcanon.com/categories/llm-frameworks.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/llm-frameworks"},{"slug":"computer-vision","name":"Computer Vision","url":"https://www.graphcanon.com/categories/computer-vision","markdown_url":"https://www.graphcanon.com/categories/computer-vision.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/computer-vision"},{"slug":"evaluation-observability","name":"Evaluation & Observability","url":"https://www.graphcanon.com/categories/evaluation-observability","markdown_url":"https://www.graphcanon.com/categories/evaluation-observability.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/evaluation-observability"}],"tags":[{"slug":"research","name":"research"},{"slug":"python","name":"python"}],"trust":{"provenance":{"is_fork":false,"github_id":895402209,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-07-11T23:45:26.397Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":430,"last_release_at":null},"security_summary":{"status":"ok","scanner":"osv@v1","low_count":0,"high_count":0,"last_scan_at":"2026-07-11T23:45:26.977Z","medium_count":0,"scan_profile":"deps","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-07-11T23:45:26.146Z"},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-07-11T23:45:26.146Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-07-11T23:45:26.146Z"}}}}