GraphCanon updated Sep 20, 2026 · GitHub synced Sep 20, 2026
30views this month
Decision brief
qa_metrics is a Python library for evaluating LLMs using standardized QA and semantic metrics, including support for Black-box and open-source models along with APIs from OpenAI and Anthropic.
Good fit when
- When you need to evaluate the performance of large language models with built-in standardized metrics like exact match and F1 Score.
- If your project requires both black-box evaluation capabilities and access to APIs from major providers for model assessment.
Avoid when
- Avoid if you seek advanced customization or fine-tuning options not present in qa_metrics for metric calculation methods beyond its provided set.
- Not ideal when needing specific evaluation tools that are not Black-box or open-source models, as the package focuses on these types of evaluations primarily.
Observed Jul 16, 2026 · Source: enrich:decision_facts
Verify the decision
Maintenance and security
Full trust report- Maintenance
- Dormant (417d since push)
- As of Sep 9, 2026
- Provenance
- Not a fork · Personal account
- As of Sep 9, 2026
- Security (OSV)
- No lockfile
- As of Jul 15, 2026
Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.
Install
pip install qa_metricsSimilar tools
Same-category neighbours. No typed graph edges are catalogued for this tool yet.
Evidence and technical details
Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.
Overview
Provides standardized metrics and tools for evaluating LLMs
Capability facts
- Languages
- python
Source: github.language · Sep 9, 2026
Categories
Compatibility
Sourced claims from the README excerpt - not unsourced marketing copy.
Tags
README
Installation 📝 License This project is licensed under the MIT License.
For agents
This page has a .md twin and JSON over the API.