qa_metrics logo

qa_metrics

zli12321/qa_metrics

A Python package for basic QA evaluations of large language models.

GraphCanon updated Oct 18, 2026 · GitHub synced Oct 18, 2026

64 stars6 forksLast push Jul 18, 2025 Python MIT

Decision brief

Unverified catalog guidance: qa_metrics is a Python library for evaluating LLMs using standardized QA and semantic metrics, including support for Black-box and open-source models along with APIs from OpenAI and Anthropic.

Good fit when

  • Unverified catalog guidance: When you need to evaluate the performance of large language models with built-in standardized metrics like exact match and F1 Score.
  • Unverified catalog guidance: If your project requires both black-box evaluation capabilities and access to APIs from major providers for model assessment.

Avoid when

  • Unverified catalog guidance: Avoid if you seek advanced customization or fine-tuning options not present in qa_metrics for metric calculation methods beyond its provided set.
  • Unverified catalog guidance: Not ideal when needing specific evaluation tools that are not Black-box or open-source models, as the package focuses on these types of evaluations primarily.

Observed Jul 16, 2026 · Source: enrich:decision_facts

Verify the decision

Maintenance and security

Full trust report
Maintenance
Dormant (456d since push)
As of Oct 18, 2026
Provenance
Not a fork · Personal account
As of Oct 18, 2026
Security (OSV)
No lockfile
As of Jul 15, 2026

Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.

Install

pip install qa-metrics

README installation instruction · observed Oct 18, 2026

PyPI

Similar tools

Same-category neighbours. No typed graph edges are catalogued for this tool yet.

Evidence and technical details

Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.

Overview

Provides standardized metrics and tools for evaluating LLMs

Capability facts

MCP server
Unknown; MCP server not established by this scan

Source: repo_scan · Oct 18, 2026

Languages
python

Source: github.language · Oct 18, 2026

Categories

Compatibility

Sourced claims from the README excerpt - not unsourced marketing copy.

Documented Python runtime (not runtime-tested)Python

Source: README excerpt (readme_declaration_v2, Oct 18, 2026)

pip install qa-metrics
Source link

Tags

README

Installation 📝 License This project is licensed under the MIT License.

For agents

This page has a .md twin and JSON over the API.

Was this helpful?

Anonymous feedback helps us improve pages and translations.