SWE-bench
Benchmark for assessing language models' capability to resolve real-world Github issues
GraphCanon updated 3w · GitHub synced 3w
Decision brief
SWE-bench serves as a benchmark for assessing how well language models can tackle real-world software engineering issues from GitHub.
Good fit when
- When you need to evaluate the effectiveness of your language model in resolving practical software engineering challenges found in open-source repositories like GitHub.
- If your research or testing involves understanding how well AI systems can generalize to visual aspects of software domains, as seen in the SWE-bench Multimodal version.
Avoid when
- Do not use SWE-bench if your language model's primary application is outside the context of real-world GitHub issue resolution.
- Avoid using this tool if you are not interested in testing AI systems' capabilities across visual software domains; it's more specialized for that specific area, unlike general-purpose benchmarks.
Observed Jul 17, 2026 · Source: enrich:decision_facts
Verify the decision
Maintenance and security
Full trust report- Maintenance
- Active (9d since push)
- As of 3w
- Provenance
- Not a fork · Organization account
- As of 3w
- Security (OSV)
- No lockfile
- As of 1mo
Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.
Install
pip install SWE-benchSimilar tools
Same-category neighbours. No typed graph edges are catalogued for this tool yet.
Evidence and technical details
Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.
Overview
A benchmark designed to evaluate how effectively language models can address practical software engineering challenges found in GitHub issue reports
Capability facts
- Languages
- python
Source: github.language+pyproject.toml · Aug 5, 2026
Categories
Tags
README
✍️ Citation & license
MIT license. Check LICENSE.md.
If you find our work helpful, please use the following citations.
For SWE-bench (Verified):
@inproceedings{
jimenez2024swebench,
title={{SWE}-bench: Can Language Models Resolve Real-world Github Issues?},
author={Carlos E Jimenez and John Yang and Alexander Wettig and Shunyu Yao and Kexin Pei and Ofir Press and Karthik R Narasimhan},
booktitle={The Twelfth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=VTF8yNQM66}
}
For SWE-bench Multimodal
@inproceedings{
yang2024swebenchmultimodal,
title={{SWE}-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?},
author={John Yang and Carlos E. Jimenez and Alex L. Zhang and Kilian Lieret and Joyce Yang and Xindi Wu and Ori Press and Niklas Muennighoff and Gabriel Synnaeve and Karthik R. Narasimhan and Diyi Yang and Sida I. Wang and Ofir Press},
booktitle={The Thirteenth International Conference on Learning Representations},
year={2025},
url={https://openreview.net/forum?id=riTiq3i21b}
}
For SWE-bench Multilingual
@misc{yang2025swesmith,
title={SWE-smith: Scaling Data for Software Engineering Agents},
author={John Yang and Kilian Lieret and Carlos E. Jimenez and Alexander Wettig and Kabir Khandpur and Yanzhe Zhang and Binyuan Hui and Ofir Press and Ludwig Schmidt and Diyi Yang},
year={2025},
eprint={2504.21798},
archivePrefix={arXiv},
primaryClass={cs.SE},
url={https://arxiv.org/abs/2504.21798},
}
For agents
This page has a .md twin and JSON over the API.