Home/Evaluation & Observability/SWE-bench

SWE-bench

SWE-bench/SWE-bench

Benchmark for assessing language models' capability to resolve real-world Github issues

GitHub Website

GraphCanon updated Oct 8, 2026 · GitHub synced Oct 8, 2026

39views this month

6.0k stars1.0k forksLast push Sep 18, 2026 Python MIT

Decision brief

SWE-bench serves as a benchmark for assessing how well language models can tackle real-world software engineering issues from GitHub.

Good fit when

When you need to evaluate the effectiveness of your language model in resolving practical software engineering challenges found in open-source repositories like GitHub.
If your research or testing involves understanding how well AI systems can generalize to visual aspects of software domains, as seen in the SWE-bench Multimodal version.

Avoid when

Do not use SWE-bench if your language model's primary application is outside the context of real-world GitHub issue resolution.
Avoid using this tool if you are not interested in testing AI systems' capabilities across visual software domains; it's more specialized for that specific area, unlike general-purpose benchmarks.

Observed Jul 17, 2026 · Source: enrich:decision_facts

SWE-bench alternatives Trust report

Verify the decision

Maintenance and security

Full trust report

Maintenance: Active (20d since push); As of Oct 8, 2026
Provenance: Not a fork · Organization account; As of Oct 8, 2026
Security (OSV): No lockfile; As of Jul 11, 2026

Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.

Install

pip install SWE-bench

PyPI

Similar tools

Same-category neighbours. No typed graph edges are catalogued for this tool yet.

awesome-copilot

Community-contributed extensions for GitHub Copilot

39k★

open-code-review

Hybrid architecture code review tool with LLM Agent for precise line-level comments and built-in security rule checks.

21k★

pua

A skill package for enhancing the functionality of AI agents within development environments

19k★

Evidence and technical details

Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.

Overview

A benchmark designed to evaluate how effectively language models can address practical software engineering challenges found in GitHub issue reports

Capability facts

CLI: CLI entrypoint
Source: pyproject.toml:[project.scripts] · Oct 8, 2026
Languages: python
Source: github.language+pyproject.toml · Oct 8, 2026

README

✍️ Citation & license MIT license. Check . If you find our work helpful, please use the following citations. For SWE bench (Verified): For SWE bench Multimodal For SWE bench Multilingual

For agents

This page has a .md twin and JSON over the API.

/tools/swe-bench-swe-bench.md GET /api/graphcanon/tools/swe-bench-swe-bench GET /api/graphcanon/graph?tool=swe-bench-swe-bench