GraphCanon updated Oct 18, 2026 · GitHub synced Oct 18, 2026
Decision brief
Unverified catalog guidance: qa_metrics is a Python library for evaluating LLMs using standardized QA and semantic metrics, including support for Black-box and open-source models along with APIs from OpenAI and Anthropic.
Good fit when
- Unverified catalog guidance: When you need to evaluate the performance of large language models with built-in standardized metrics like exact match and F1 Score.
- Unverified catalog guidance: If your project requires both black-box evaluation capabilities and access to APIs from major providers for model assessment.
Avoid when
- Unverified catalog guidance: Avoid if you seek advanced customization or fine-tuning options not present in qa_metrics for metric calculation methods beyond its provided set.
- Unverified catalog guidance: Not ideal when needing specific evaluation tools that are not Black-box or open-source models, as the package focuses on these types of evaluations primarily.
Observed Jul 16, 2026 · Source: enrich:decision_facts
Verify the decision
Maintenance and security
Full trust report- Maintenance
- Dormant (456d since push)
- As of Oct 18, 2026
- Provenance
- Not a fork · Personal account
- As of Oct 18, 2026
- Security (OSV)
- No lockfile
- As of Jul 15, 2026
Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.
Install
pip install qa-metricsREADME installation instruction · observed Oct 18, 2026
PyPISimilar tools
Same-category neighbours. No typed graph edges are catalogued for this tool yet.
Evidence and technical details
Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.
Overview
Provides standardized metrics and tools for evaluating LLMs
Capability facts
- MCP server
- Unknown; MCP server not established by this scan
Source: repo_scan · Oct 18, 2026
- Languages
- python
Source: github.language · Oct 18, 2026
Categories
Compatibility
Sourced claims from the README excerpt - not unsourced marketing copy.
Source: README excerpt (readme_declaration_v2, Oct 18, 2026)
pip install qa-metricsSource link
Tags
README
Installation 📝 License This project is licensed under the MIT License.
For agents
This page has a .md twin and JSON over the API.