“benchmark”

vector-db-benchmark

Framework for benchmarking vector search engines

ClawBench

Open-source benchmark for browser AI agents on daily tasks

HPOBench

A collection of hyperparameter optimization benchmark problems

MixEval

Evaluation suite and dynamic data release for MixEval

whichllm

Command-line tool to find and benchmark local LLM performance

myclaw-bench

Benchmark for AI agents on OpenClaw

BizFinBench

A Business-Driven Real-World Financial Benchmark for Evaluating LLMs

chinese-llm-benchmark

ReLE评测：中文AI大模型能力评测

VectorDBBench

Benchmark for vector databases

SWE-bench

Benchmark for assessing language models' capability to resolve real-world Github issues

circle-guard-bench

AI benchmark for evaluating LLM guard systems

llm-leaderboard

Comprehensive LLM benchmark scores and provider prices

ALERT

A Comprehensive Benchmark for Assessing Large Language Models' Safety Through Red Teaming

DS-1000

Benchmark and code for evaluating large language models on data science tasks

anubis-oss

Local LLM Testing & Benchmarking for Apple Silicon

jailbreakbench

An Open Robustness Benchmark for Jailbreaking Language Models

WeaveBench

A Long-Horizon Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces

BIG-bench

Collaborative benchmark for language model capabilities

EnterpriseRAG-Bench

Dataset and benchmark for RAG on company internal documents

Awesome-Datasets-Hub

Curated collection of datasets for Large Language Models (LLMs)

mempalace

The best-benchmarked open-source AI memory system.

MultiPL-E

A multi-programming language benchmark for LLMs

evals

Framework for evaluating LLMs and LLM systems with an open-source registry of benchmarks.

awesome-evals

A curated library of resources for building and evaluating AI agents

IndustryBench

A multi-lingual benchmark for evaluating industrial domain knowledge of LLMs

ITBench

An open source benchmarking framework for IT automation

code-eval

Run evaluation on LLMs using human-eval benchmark.

lmms-eval

One-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks

FlashRAG

A Python toolkit for efficient RAG research

chain-of-thought-hub

Benchmarking large language models' complex reasoning ability with chain-of-thought prompting

stepshield

Temporal evaluation benchmark for AI agent guardrails

strix-halo-guide

AMD Ryzen AI Halo setup guide for LLM frameworks like Ollama and llama.cpp Vulkan on Radeon hardware

repobench

Benchmarking Repository-Level Code Auto-Completion Systems

evalplus

Rigorous evaluation of LLM-synthesized code

LLMEvaluation

A comprehensive guide to LLM evaluation methods

cceval

CrossCodeEval Benchmark for Cross-File Code Completion

BIPIA

Benchmark for evaluating LLM robustness to indirect prompt injection attacks.

GTA

GTA: General Tool Agent Benchmark and Evaluation Framework

CoDA-Bench

Benchmark for code agents on data-intensive tasks

LeanEuclid

Benchmark for autoformalization in Euclidean geometry targeting Lean proof assistant.