{"data":{"node":{"slug":"open-compass-vlmevalkit","name":"VLMEvalKit","tagline":"An open-source evaluation toolkit for large vision-language models","github_url":"https://github.com/open-compass/VLMEvalKit","owner":"open-compass","repo":"VLMEvalKit","owner_avatar_url":"https://avatars.githubusercontent.com/u/143521324?v=4","primary_language":"Python","stars":4345,"forks":745,"topics":["chatgpt","claude","clip","computer-vision","evaluation","gemini","gpt","gpt-4v","gpt4","large-language-models","llava","llm","multi-modal","openai","openai-api","pytorch","qwen","vit","vqa"],"archived":false,"github_pushed_at":"2026-08-17T17:16:26+00:00","maintenance_label":"Very active","stars_delta_30d":60,"url":"https://www.graphcanon.com/tools/open-compass-vlmevalkit","markdown_url":"https://www.graphcanon.com/tools/open-compass-vlmevalkit.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/open-compass-vlmevalkit","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=open-compass-vlmevalkit"},"categories":[{"slug":"evaluation-observability","name":"Evaluation & Observability","url":"https://www.graphcanon.com/categories/evaluation-observability","markdown_url":"https://www.graphcanon.com/categories/evaluation-observability.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/evaluation-observability"}],"tags":[{"slug":"computer-vision","name":"computer-vision"},{"slug":"evaluation","name":"evaluation"},{"slug":"large-language-models","name":"large language models"},{"slug":"llm","name":"llm"},{"slug":"multi-modal","name":"multi-modal"}],"edges":[{"type":"related","direction":"out","explanation":"VLMEvalKit focuses on evaluating multimodal large language models, and Awesome-Multimodal-Large-Language-Models provides a curated list of such models and resources. They are related but serve different purposes.","successor_context":null,"tool":{"slug":"bradyfu-awesome-multimodal-large-language-models","name":"Awesome-Multimodal-Large-Language-Models","tagline":"Latest Advances on Multimodal Large Language Models","github_url":"https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models","owner":"BradyFU","repo":"Awesome-Multimodal-Large-Language-Models","owner_avatar_url":"https://avatars.githubusercontent.com/u/54254631?v=4","primary_language":null,"stars":17978,"forks":1133,"topics":["chain-of-thought","in-context-learning","instruction-following","instruction-tuning","large-language-models","large-vision-language-model","large-vision-language-models","multi-modality","multimodal-chain-of-thought","multimodal-in-context-learning","multimodal-instruction-tuning","multimodal-large-language-models","visual-instruction-tuning"],"archived":false,"github_pushed_at":"2026-08-14T17:17:50+00:00","maintenance_label":"Very active","stars_delta_30d":29,"url":"https://www.graphcanon.com/tools/bradyfu-awesome-multimodal-large-language-models","markdown_url":"https://www.graphcanon.com/tools/bradyfu-awesome-multimodal-large-language-models.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/bradyfu-awesome-multimodal-large-language-models","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=bradyfu-awesome-multimodal-large-language-models"}},{"type":"integrates_with","direction":"out","explanation":"RagaAI-Catalyst provides observability and monitoring for LLM applications, which can integrate with VLMEvalKit to provide feedback loops for model improvements.","successor_context":null,"tool":{"slug":"raga-ai-hub-ragaai-catalyst","name":"RagaAI-Catalyst","tagline":"Python SDK for AI agent observability and evaluation","github_url":"https://github.com/raga-ai-hub/RagaAI-Catalyst","owner":"raga-ai-hub","repo":"RagaAI-Catalyst","owner_avatar_url":"https://avatars.githubusercontent.com/u/161833182?v=4","primary_language":"Python","stars":16148,"forks":3565,"topics":["agentic-ai","agentic-ai-development","agentneo","agents","ai-agent-monitoring","ai-application-debugging","ai-evaluation-tools","ai-performance-optimization","ai-tool-interaction-monitoring","llm-testing","llm-tracing","llmops"],"archived":false,"github_pushed_at":"2026-02-11T14:43:33+00:00","maintenance_label":"Slowing","stars_delta_30d":5,"url":"https://www.graphcanon.com/tools/raga-ai-hub-ragaai-catalyst","markdown_url":"https://www.graphcanon.com/tools/raga-ai-hub-ragaai-catalyst.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/raga-ai-hub-ragaai-catalyst","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=raga-ai-hub-ragaai-catalyst"}},{"type":"alternative","direction":"out","explanation":"VLMEvalKit and lmms-eval both serve as evaluation toolkits for multimodal large language models, addressing the same problem with potentially different approaches or features.","successor_context":null,"tool":{"slug":"evolvinglmms-lab-lmms-eval","name":"lmms-eval","tagline":"One-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks","github_url":"https://github.com/EvolvingLMMs-Lab/lmms-eval","owner":"EvolvingLMMs-Lab","repo":"lmms-eval","owner_avatar_url":"https://avatars.githubusercontent.com/u/154951679?v=4","primary_language":"Python","stars":4368,"forks":639,"topics":["agi","audio-evaluation","benchmark","evaluation","large-language-models","llm-evaluation","multimodal","multimodal-evaluation","video-understanding","vision-language-model","vlm"],"archived":false,"github_pushed_at":"2026-08-06T02:22:23+00:00","maintenance_label":"Active","stars_delta_30d":52,"url":"https://www.graphcanon.com/tools/evolvinglmms-lab-lmms-eval","markdown_url":"https://www.graphcanon.com/tools/evolvinglmms-lab-lmms-eval.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/evolvinglmms-lab-lmms-eval","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=evolvinglmms-lab-lmms-eval"}},{"type":"successor","direction":"out","explanation":"VLMEvalKit has a 'successor' relationship to 'lmms-eval' because while lmms-eval provides a unified evaluation framework for multimodal large language models across various media types, VLMEvalKit specifically focuses on and simplifies the evaluation of vision-language models through dedicated generation-based methods and exact matching techniques, offering more specialized capabilities within a细分","successor_context":null,"tool":{"slug":"evolvinglmms-lab-lmms-eval","name":"lmms-eval","tagline":"One-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks","github_url":"https://github.com/EvolvingLMMs-Lab/lmms-eval","owner":"EvolvingLMMs-Lab","repo":"lmms-eval","owner_avatar_url":"https://avatars.githubusercontent.com/u/154951679?v=4","primary_language":"Python","stars":4368,"forks":639,"topics":["agi","audio-evaluation","benchmark","evaluation","large-language-models","llm-evaluation","multimodal","multimodal-evaluation","video-understanding","vision-language-model","vlm"],"archived":false,"github_pushed_at":"2026-08-06T02:22:23+00:00","maintenance_label":"Active","stars_delta_30d":52,"url":"https://www.graphcanon.com/tools/evolvinglmms-lab-lmms-eval","markdown_url":"https://www.graphcanon.com/tools/evolvinglmms-lab-lmms-eval.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/evolvinglmms-lab-lmms-eval","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=evolvinglmms-lab-lmms-eval"}},{"type":"integrates_with","direction":"out","explanation":"Both VLMEvalKit and Langfuse aim at evaluating and maintaining large models, making them complementary tools within the evaluation ecosystem.","successor_context":null,"tool":{"slug":"langfuse-langfuse","name":"langfuse","tagline":"Open source AI engineering platform: LLM evals, observability, metrics, prompt management, playground, datasets","github_url":"https://github.com/langfuse/langfuse","owner":"langfuse","repo":"langfuse","owner_avatar_url":"https://avatars.githubusercontent.com/u/134601687?v=4","primary_language":"TypeScript","stars":32271,"forks":3466,"topics":["analytics","autogen","evaluation","langchain","large-language-models","llama-index","llm","llm-evaluation","llm-observability","llmops","monitoring","observability","open-source","openai","playground","prompt-engineering","prompt-management","self-hosted","ycombinator"],"archived":false,"github_pushed_at":"2026-07-31T22:58:07+00:00","maintenance_label":"Very active","url":"https://www.graphcanon.com/tools/langfuse-langfuse","markdown_url":"https://www.graphcanon.com/tools/langfuse-langfuse.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/langfuse-langfuse","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=langfuse-langfuse"}},{"type":"related","direction":"out","explanation":"Both focus on enhancing retrieval-augmented generation techniques and could be used in conjunction with RAG applications that require evaluation.","successor_context":null,"tool":{"slug":"nirdiamant-rag-techniques","name":"RAG_Techniques","tagline":"Showcases advanced techniques for Retrieval-Augmented Generation (RAG) systems with detailed notebook tutorials.","github_url":"https://github.com/NirDiamant/RAG_Techniques","owner":"NirDiamant","repo":"RAG_Techniques","owner_avatar_url":"https://avatars.githubusercontent.com/u/28316913?v=4","primary_language":"Jupyter Notebook","stars":29076,"forks":3540,"topics":["agentic-rag","ai","embeddings","generative-ai","gpt","langchain","llama-index","llm","llms","machine-learning","nlp","openai","python","rag","retrieval-augmented-generation","semantic-search","tutorials","vector-database"],"archived":false,"github_pushed_at":"2026-08-15T00:52:05+00:00","maintenance_label":"Very active","stars_delta_30d":455,"url":"https://www.graphcanon.com/tools/nirdiamant-rag-techniques","markdown_url":"https://www.graphcanon.com/tools/nirdiamant-rag-techniques.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/nirdiamant-rag-techniques","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=nirdiamant-rag-techniques"}},{"type":"integrates_with","direction":"out","explanation":"Evidently, an ML and LLM observability framework, can work with VLMEvalKit by providing insights into the performance of models evaluated using VLMEvalKit.","successor_context":null,"tool":{"slug":"evidentlyai-evidently","name":"evidently","tagline":"An open-source ML and LLM observability framework.","github_url":"https://github.com/evidentlyai/evidently","owner":"evidentlyai","repo":"evidently","owner_avatar_url":"https://avatars.githubusercontent.com/u/75031056?v=4","primary_language":"Jupyter Notebook","stars":7790,"forks":895,"topics":["data-drift","data-quality","data-science","data-validation","generative-ai","hacktoberfest","html-report","jupyter-notebook","llm","llmops","machine-learning","mlops","model-monitoring","pandas-dataframe"],"archived":false,"github_pushed_at":"2026-08-05T16:29:57+00:00","maintenance_label":"Very active","stars_delta_30d":117,"url":"https://www.graphcanon.com/tools/evidentlyai-evidently","markdown_url":"https://www.graphcanon.com/tools/evidentlyai-evidently.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/evidentlyai-evidently","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=evidentlyai-evidently"}},{"type":"related","direction":"out","explanation":null,"successor_context":null,"tool":{"slug":"comet-ml-opik","name":"opik","tagline":"Debug, evaluate, and monitor your LLM applications with comprehensive tracing and production-ready dashboards","github_url":"https://github.com/comet-ml/opik","owner":"comet-ml","repo":"opik","owner_avatar_url":"https://avatars.githubusercontent.com/u/31487821?v=4","primary_language":"Python","stars":21177,"forks":1681,"topics":["evaluation","hacktoberfest","hacktoberfest2025","langchain","llama-index","llm","llm-evaluation","llm-observability","llmops","open-source","openai","playground","prompt-engineering"],"archived":false,"github_pushed_at":"2026-08-07T11:46:34+00:00","maintenance_label":"Very active","stars_delta_30d":767,"url":"https://www.graphcanon.com/tools/comet-ml-opik","markdown_url":"https://www.graphcanon.com/tools/comet-ml-opik.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/comet-ml-opik","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=comet-ml-opik"}},{"type":"related","direction":"out","explanation":null,"successor_context":null,"tool":{"slug":"agenta-ai-agenta","name":"agenta","tagline":"The open-source LLMOps platform for prompt management, evaluation, and observability.","github_url":"https://github.com/Agenta-AI/agenta","owner":"Agenta-AI","repo":"agenta","owner_avatar_url":"https://avatars.githubusercontent.com/u/127993667?v=4","primary_language":"TypeScript","stars":4445,"forks":609,"topics":["agent-builder","agent-observability","agent-orchestration","agent-workspace","agentic-ai","ai-agent","ai-agents","ai-automation","ai-skills-manager","ai-workflow-builder","harness","mcp","open-source","self-hosted","workflow-automation"],"archived":false,"github_pushed_at":"2026-08-07T10:41:36+00:00","maintenance_label":"Very active","stars_delta_30d":170,"url":"https://www.graphcanon.com/tools/agenta-ai-agenta","markdown_url":"https://www.graphcanon.com/tools/agenta-ai-agenta.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/agenta-ai-agenta","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=agenta-ai-agenta"}},{"type":"integrates_with","direction":"out","explanation":"OpenLLMetry is designed for observability of LLM applications and can work with VLMEvalKit to track the performance of evaluation processes.","successor_context":null,"tool":{"slug":"traceloop-openllmetry","name":"openllmetry","tagline":"Open-source observability for GenAI and LLM applications based on OpenTelemetry.","github_url":"https://github.com/traceloop/openllmetry","owner":"traceloop","repo":"openllmetry","owner_avatar_url":"https://avatars.githubusercontent.com/u/125419530?v=4","primary_language":"Python","stars":7377,"forks":1047,"topics":["artifical-intelligence","datascience","generative-ai","good-first-issue","good-first-issues","help-wanted","llm","llmops","metrics","ml","model-monitoring","monitoring","observability","open-source","open-telemetry","opentelemetry","opentelemetry-python","python"],"archived":false,"github_pushed_at":"2026-08-10T08:49:01+00:00","maintenance_label":"Very active","stars_delta_30d":75,"url":"https://www.graphcanon.com/tools/traceloop-openllmetry","markdown_url":"https://www.graphcanon.com/tools/traceloop-openllmetry.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/traceloop-openllmetry","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=traceloop-openllmetry"}}],"neighbours":[{"slug":"vllm-project-vllm","name":"vllm","tagline":"A high-throughput and memory-efficient inference and serving engine for LLMs","github_url":"https://github.com/vllm-project/vllm","owner":"vllm-project","repo":"vllm","owner_avatar_url":"https://avatars.githubusercontent.com/u/136984999?v=4","primary_language":"Python","stars":87847,"forks":20135,"topics":["amd","blackwell","cuda","deepseek","deepseek-v3","gpt","gpt-oss","inference","kimi","llama","llm","llm-serving","model-serving","moe","openai","pytorch","qwen","qwen3","tpu","transformer"],"archived":false,"github_pushed_at":"2026-08-01T11:55:36+00:00","maintenance_label":"Active","url":"https://www.graphcanon.com/tools/vllm-project-vllm","markdown_url":"https://www.graphcanon.com/tools/vllm-project-vllm.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/vllm-project-vllm","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=vllm-project-vllm","shared_categories":[]},{"slug":"openai-evals","name":"evals","tagline":"Framework for evaluating LLMs and LLM systems with an open-source registry of benchmarks.","github_url":"https://github.com/openai/evals","owner":"openai","repo":"evals","owner_avatar_url":"https://avatars.githubusercontent.com/u/14957082?v=4","primary_language":"Python","stars":19127,"forks":3050,"topics":[],"archived":false,"github_pushed_at":"2026-04-14T15:29:57+00:00","maintenance_label":"Slowing","url":"https://www.graphcanon.com/tools/openai-evals","markdown_url":"https://www.graphcanon.com/tools/openai-evals.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/openai-evals","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=openai-evals","shared_categories":["evaluation-observability"]},{"slug":"bradyfu-awesome-multimodal-large-language-models","name":"Awesome-Multimodal-Large-Language-Models","tagline":"Latest Advances on Multimodal Large Language Models","github_url":"https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models","owner":"BradyFU","repo":"Awesome-Multimodal-Large-Language-Models","owner_avatar_url":"https://avatars.githubusercontent.com/u/54254631?v=4","primary_language":null,"stars":17978,"forks":1133,"topics":["chain-of-thought","in-context-learning","instruction-following","instruction-tuning","large-language-models","large-vision-language-model","large-vision-language-models","multi-modality","multimodal-chain-of-thought","multimodal-in-context-learning","multimodal-instruction-tuning","multimodal-large-language-models","visual-instruction-tuning"],"archived":false,"github_pushed_at":"2026-08-14T17:17:50+00:00","maintenance_label":"Active","url":"https://www.graphcanon.com/tools/bradyfu-awesome-multimodal-large-language-models","markdown_url":"https://www.graphcanon.com/tools/bradyfu-awesome-multimodal-large-language-models.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/bradyfu-awesome-multimodal-large-language-models","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=bradyfu-awesome-multimodal-large-language-models","shared_categories":["evaluation-observability"]},{"slug":"confident-ai-deepeval","name":"deepeval","tagline":"LLM Evaluation Framework.","github_url":"https://github.com/confident-ai/deepeval","owner":"confident-ai","repo":"deepeval","owner_avatar_url":"https://avatars.githubusercontent.com/u/130858411?v=4","primary_language":"Python","stars":17226,"forks":1736,"topics":["evaluation-framework","evaluation-metrics","llm-evaluation","llm-evaluation-framework","llm-evaluation-metrics","python"],"archived":false,"github_pushed_at":"2026-07-27T11:33:31+00:00","maintenance_label":"Active","url":"https://www.graphcanon.com/tools/confident-ai-deepeval","markdown_url":"https://www.graphcanon.com/tools/confident-ai-deepeval.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/confident-ai-deepeval","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=confident-ai-deepeval","shared_categories":["evaluation-observability"]},{"slug":"eleutherai-lm-evaluation-harness","name":"lm-evaluation-harness","tagline":"A framework for few-shot evaluation of language models.","github_url":"https://github.com/EleutherAI/lm-evaluation-harness","owner":"EleutherAI","repo":"lm-evaluation-harness","owner_avatar_url":"https://avatars.githubusercontent.com/u/68924597?v=4","primary_language":"Python","stars":13560,"forks":3467,"topics":["evaluation-framework","language-model","transformer"],"archived":false,"github_pushed_at":"2026-07-13T20:18:15+00:00","maintenance_label":"Steady","url":"https://www.graphcanon.com/tools/eleutherai-lm-evaluation-harness","markdown_url":"https://www.graphcanon.com/tools/eleutherai-lm-evaluation-harness.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/eleutherai-lm-evaluation-harness","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=eleutherai-lm-evaluation-harness","shared_categories":["evaluation-observability"]},{"slug":"shishirpatil-gorilla","name":"gorilla","tagline":"Training and Evaluating LLMs for Function Calls (Tool Calls)","github_url":"https://github.com/ShishirPatil/gorilla","owner":"ShishirPatil","repo":"gorilla","owner_avatar_url":"https://avatars.githubusercontent.com/u/30296397?v=4","primary_language":"Python","stars":12988,"forks":1397,"topics":["api","api-documentation","chatgpt","claude-api","gpt-4-api","llm","openai-api","openai-functions"],"archived":false,"github_pushed_at":"2026-04-13T03:19:45+00:00","maintenance_label":"Slowing","url":"https://www.graphcanon.com/tools/shishirpatil-gorilla","markdown_url":"https://www.graphcanon.com/tools/shishirpatil-gorilla.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/shishirpatil-gorilla","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=shishirpatil-gorilla","shared_categories":["evaluation-observability"]},{"slug":"kalyanks-nlp-llm-engineer-toolkit","name":"llm-engineer-toolkit","tagline":"A curated list of over 120 LLM libraries categorized.","github_url":"https://github.com/KalyanKS-NLP/llm-engineer-toolkit","owner":"KalyanKS-NLP","repo":"llm-engineer-toolkit","owner_avatar_url":"https://avatars.githubusercontent.com/u/202506543?v=4","primary_language":null,"stars":10767,"forks":1682,"topics":["ai-engineer","generative-ai","large-language-models","llm-engineer","llms"],"archived":false,"github_pushed_at":"2026-08-16T13:05:43+00:00","maintenance_label":"Very active","url":"https://www.graphcanon.com/tools/kalyanks-nlp-llm-engineer-toolkit","markdown_url":"https://www.graphcanon.com/tools/kalyanks-nlp-llm-engineer-toolkit.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/kalyanks-nlp-llm-engineer-toolkit","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=kalyanks-nlp-llm-engineer-toolkit","shared_categories":["evaluation-observability"]},{"slug":"optimalscale-lmflow","name":"LMFlow","tagline":"An Extensible Toolkit for Finetuning and Inference of Large Foundation Models","github_url":"https://github.com/OptimalScale/LMFlow","owner":"OptimalScale","repo":"LMFlow","owner_avatar_url":"https://avatars.githubusercontent.com/u/128913633?v=4","primary_language":"Python","stars":8486,"forks":825,"topics":["chatgpt","deep-learning","instruction-following","language-model","pretrained-models","pytorch","transformer"],"archived":false,"github_pushed_at":"2026-05-22T02:57:26+00:00","maintenance_label":"Slowing","url":"https://www.graphcanon.com/tools/optimalscale-lmflow","markdown_url":"https://www.graphcanon.com/tools/optimalscale-lmflow.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/optimalscale-lmflow","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=optimalscale-lmflow","shared_categories":[]},{"slug":"evolvinglmms-lab-lmms-eval","name":"lmms-eval","tagline":"One-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks","github_url":"https://github.com/EvolvingLMMs-Lab/lmms-eval","owner":"EvolvingLMMs-Lab","repo":"lmms-eval","owner_avatar_url":"https://avatars.githubusercontent.com/u/154951679?v=4","primary_language":"Python","stars":4368,"forks":639,"topics":["agi","audio-evaluation","benchmark","evaluation","large-language-models","llm-evaluation","multimodal","multimodal-evaluation","video-understanding","vision-language-model","vlm"],"archived":false,"github_pushed_at":"2026-08-06T02:22:23+00:00","maintenance_label":"Active","url":"https://www.graphcanon.com/tools/evolvinglmms-lab-lmms-eval","markdown_url":"https://www.graphcanon.com/tools/evolvinglmms-lab-lmms-eval.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/evolvinglmms-lab-lmms-eval","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=evolvinglmms-lab-lmms-eval","shared_categories":["evaluation-observability"]},{"slug":"stanford-crfm-helm","name":"helm","tagline":"Holistic, reproducible and transparent evaluation of foundation models","github_url":"https://github.com/stanford-crfm/helm","owner":"stanford-crfm","repo":"helm","owner_avatar_url":"https://avatars.githubusercontent.com/u/75054807?v=4","primary_language":"Python","stars":2873,"forks":406,"topics":[],"archived":false,"github_pushed_at":"2026-08-01T01:23:17+00:00","maintenance_label":"Active","url":"https://www.graphcanon.com/tools/stanford-crfm-helm","markdown_url":"https://www.graphcanon.com/tools/stanford-crfm-helm.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/stanford-crfm-helm","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=stanford-crfm-helm","shared_categories":["evaluation-observability"]},{"slug":"huggingface-lighteval","name":"lighteval","tagline":"All-in-one toolkit for evaluating LLMs across multiple backends","github_url":"https://github.com/huggingface/lighteval","owner":"huggingface","repo":"lighteval","owner_avatar_url":"https://avatars.githubusercontent.com/u/25720743?v=4","primary_language":"Python","stars":2508,"forks":523,"topics":["evaluation","evaluation-framework","evaluation-metrics","huggingface"],"archived":false,"github_pushed_at":"2026-06-29T13:03:33+00:00","maintenance_label":"Steady","url":"https://www.graphcanon.com/tools/huggingface-lighteval","markdown_url":"https://www.graphcanon.com/tools/huggingface-lighteval.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/huggingface-lighteval","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=huggingface-lighteval","shared_categories":["evaluation-observability"]},{"slug":"evalplus-evalplus","name":"evalplus","tagline":"Rigorous evaluation of LLM-synthesized code","github_url":"https://github.com/evalplus/evalplus","owner":"evalplus","repo":"evalplus","owner_avatar_url":"https://avatars.githubusercontent.com/u/132106461?v=4","primary_language":"Python","stars":1794,"forks":205,"topics":["benchmark","chatgpt","efficiency","gpt-4","large-language-models","program-synthesis","testing"],"archived":false,"github_pushed_at":"2025-10-02T22:56:38+00:00","maintenance_label":"Slowing","url":"https://www.graphcanon.com/tools/evalplus-evalplus","markdown_url":"https://www.graphcanon.com/tools/evalplus-evalplus.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/evalplus-evalplus","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=evalplus-evalplus","shared_categories":["evaluation-observability"]}]}}