{"data":{"node":{"slug":"evolvinglmms-lab-lmms-eval","name":"lmms-eval","tagline":"One-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks","github_url":"https://github.com/EvolvingLMMs-Lab/lmms-eval","owner":"EvolvingLMMs-Lab","repo":"lmms-eval","owner_avatar_url":"https://avatars.githubusercontent.com/u/154951679?v=4","primary_language":"Python","stars":4368,"forks":639,"topics":["agi","audio-evaluation","benchmark","evaluation","large-language-models","llm-evaluation","multimodal","multimodal-evaluation","video-understanding","vision-language-model","vlm"],"archived":false,"github_pushed_at":"2026-08-06T02:22:23+00:00","maintenance_label":"Active","stars_delta_30d":52,"url":"https://www.graphcanon.com/tools/evolvinglmms-lab-lmms-eval","markdown_url":"https://www.graphcanon.com/tools/evolvinglmms-lab-lmms-eval.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/evolvinglmms-lab-lmms-eval","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=evolvinglmms-lab-lmms-eval"},"categories":[{"slug":"evaluation-observability","name":"Evaluation & Observability","url":"https://www.graphcanon.com/categories/evaluation-observability","markdown_url":"https://www.graphcanon.com/categories/evaluation-observability.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/evaluation-observability"}],"tags":[{"slug":"agi","name":"agi"},{"slug":"audio-evaluation","name":"audio-evaluation"},{"slug":"benchmark","name":"benchmark"},{"slug":"evaluation","name":"evaluation"},{"slug":"large-language-models","name":"large-language-models"},{"slug":"llm-evaluation","name":"llm-evaluation"},{"slug":"multimodal","name":"multimodal"},{"slug":"multimodal-evaluation","name":"multimodal-evaluation"}],"edges":[{"type":"integrates_with","direction":"out","explanation":"Both repositories provide tools for evaluating and testing generative AI systems, with lmms-eval focusing on multimodal evaluations while Giskard OSS focuses broadly on evals and red teaming.","successor_context":null,"tool":{"slug":"giskard-ai-giskard-oss","name":"giskard-oss","tagline":"Open-Source Evaluation & Testing library for LLM Agents","github_url":"https://github.com/Giskard-AI/giskard-oss","owner":"Giskard-AI","repo":"giskard-oss","owner_avatar_url":"https://avatars.githubusercontent.com/u/71782571?v=4","primary_language":"Python","stars":5824,"forks":537,"topics":["agent-evaluation","ai-red-team","ai-security","ai-testing","fairness-ai","llm","llm-eval","llm-evaluation","llm-security","llmops","ml-testing","ml-validation","mlops","rag-evaluation","red-team-tools","responsible-ai","trustworthy-ai"],"archived":false,"github_pushed_at":"2026-09-18T07:09:06+00:00","maintenance_label":"Very active","stars_delta_30d":97,"url":"https://www.graphcanon.com/tools/giskard-ai-giskard-oss","markdown_url":"https://www.graphcanon.com/tools/giskard-ai-giskard-oss.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/giskard-ai-giskard-oss","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=giskard-ai-giskard-oss"}},{"type":"alternative","direction":"out","explanation":"Both repositories deal with multimodal large language models, but they approach the evaluation and listing of these models differently.","successor_context":null,"tool":{"slug":"bradyfu-awesome-multimodal-large-language-models","name":"Awesome-Multimodal-Large-Language-Models","tagline":"Latest Advances on Multimodal Large Language Models","github_url":"https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models","owner":"BradyFU","repo":"Awesome-Multimodal-Large-Language-Models","owner_avatar_url":"https://avatars.githubusercontent.com/u/54254631?v=4","primary_language":null,"stars":18026,"forks":1136,"topics":["chain-of-thought","in-context-learning","instruction-following","instruction-tuning","large-language-models","large-vision-language-model","large-vision-language-models","multi-modality","multimodal-chain-of-thought","multimodal-in-context-learning","multimodal-instruction-tuning","multimodal-large-language-models","visual-instruction-tuning"],"archived":false,"github_pushed_at":"2026-09-18T03:09:18+00:00","maintenance_label":"Very active","stars_delta_30d":48,"url":"https://www.graphcanon.com/tools/bradyfu-awesome-multimodal-large-language-models","markdown_url":"https://www.graphcanon.com/tools/bradyfu-awesome-multimodal-large-language-models.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/bradyfu-awesome-multimodal-large-language-models","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=bradyfu-awesome-multimodal-large-language-models"}},{"type":"related","direction":"out","explanation":"The repository provides a list of resources regarding multmodal LLMs, which is the domain that lmms-eval aims to evaluate and improve upon, though it does not directly integrate or depend on this resource.","successor_context":null,"tool":{"slug":"bradyfu-awesome-multimodal-large-language-models","name":"Awesome-Multimodal-Large-Language-Models","tagline":"Latest Advances on Multimodal Large Language Models","github_url":"https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models","owner":"BradyFU","repo":"Awesome-Multimodal-Large-Language-Models","owner_avatar_url":"https://avatars.githubusercontent.com/u/54254631?v=4","primary_language":null,"stars":18026,"forks":1136,"topics":["chain-of-thought","in-context-learning","instruction-following","instruction-tuning","large-language-models","large-vision-language-model","large-vision-language-models","multi-modality","multimodal-chain-of-thought","multimodal-in-context-learning","multimodal-instruction-tuning","multimodal-large-language-models","visual-instruction-tuning"],"archived":false,"github_pushed_at":"2026-09-18T03:09:18+00:00","maintenance_label":"Very active","stars_delta_30d":48,"url":"https://www.graphcanon.com/tools/bradyfu-awesome-multimodal-large-language-models","markdown_url":"https://www.graphcanon.com/tools/bradyfu-awesome-multimodal-large-language-models.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/bradyfu-awesome-multimodal-large-language-models","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=bradyfu-awesome-multimodal-large-language-models"}},{"type":"alternative","direction":"out","explanation":"Both tools are designed to evaluate LLMs by providing ways to test and red-team LLM applications, but they use different methodologies for evaluation.","successor_context":null,"tool":{"slug":"promptfoo-promptfoo","name":"promptfoo","tagline":"Test prompts, agents, and RAGs. Compare performance of various LLMs.","github_url":"https://github.com/promptfoo/promptfoo","owner":"promptfoo","repo":"promptfoo","owner_avatar_url":"https://avatars.githubusercontent.com/u/137907881?v=4","primary_language":"TypeScript","stars":25251,"forks":2334,"topics":["ci","ci-cd","cicd","evaluation","evaluation-framework","llm","llm-eval","llm-evaluation","llm-evaluation-framework","llmops","pentesting","prompt-engineering","prompt-testing","prompts","rag","red-teaming","testing","vulnerability-scanners"],"archived":false,"github_pushed_at":"2026-09-18T07:59:45+00:00","maintenance_label":"Very active","stars_delta_30d":1413,"url":"https://www.graphcanon.com/tools/promptfoo-promptfoo","markdown_url":"https://www.graphcanon.com/tools/promptfoo-promptfoo.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/promptfoo-promptfoo","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=promptfoo-promptfoo"}},{"type":"integrates_with","direction":"out","explanation":"Both tools focus on evaluating LLMs, lmms-eval for multimodal evaluations and promptfoo for evaluating and red-teaming LLM apps.","successor_context":null,"tool":{"slug":"promptfoo-promptfoo","name":"promptfoo","tagline":"Test prompts, agents, and RAGs. Compare performance of various LLMs.","github_url":"https://github.com/promptfoo/promptfoo","owner":"promptfoo","repo":"promptfoo","owner_avatar_url":"https://avatars.githubusercontent.com/u/137907881?v=4","primary_language":"TypeScript","stars":25251,"forks":2334,"topics":["ci","ci-cd","cicd","evaluation","evaluation-framework","llm","llm-eval","llm-evaluation","llm-evaluation-framework","llmops","pentesting","prompt-engineering","prompt-testing","prompts","rag","red-teaming","testing","vulnerability-scanners"],"archived":false,"github_pushed_at":"2026-09-18T07:59:45+00:00","maintenance_label":"Very active","stars_delta_30d":1413,"url":"https://www.graphcanon.com/tools/promptfoo-promptfoo","markdown_url":"https://www.graphcanon.com/tools/promptfoo-promptfoo.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/promptfoo-promptfoo","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=promptfoo-promptfoo"}},{"type":"integrates_with","direction":"out","explanation":"Both tools aim at observability of AI systems, with lmms-eval focusing on multimodal evaluations while LMNR provides a broader platform for AI agent observability.","successor_context":null,"tool":{"slug":"lmnr-ai-lmnr","name":"lmnr","tagline":"Laminar - open-source observability platform for AI agents","github_url":"https://github.com/lmnr-ai/lmnr","owner":"lmnr-ai","repo":"lmnr","owner_avatar_url":"https://avatars.githubusercontent.com/u/161496104?v=4","primary_language":"TypeScript","stars":3265,"forks":239,"topics":["agent-observability","agents","ai","ai-observability","aiops","analytics","developer-tools","evals","evaluation","llm-evaluation","llm-observability","llmops","monitoring","observability","open-source","rust","rust-lang","self-hosted","ts","typescript"],"archived":false,"github_pushed_at":"2026-09-17T23:58:29+00:00","maintenance_label":"Very active","stars_delta_30d":162,"url":"https://www.graphcanon.com/tools/lmnr-ai-lmnr","markdown_url":"https://www.graphcanon.com/tools/lmnr-ai-lmnr.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/lmnr-ai-lmnr","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=lmnr-ai-lmnr"}},{"type":"alternative","direction":"out","explanation":"Both tools focus on evaluating LLMs, but they offer different functionalities and approaches. Langfuse offers a broader platform for AI engineering with observability features, whereas lmms-eval is specialized in multimodal evaluations.","successor_context":null,"tool":{"slug":"langfuse-langfuse","name":"langfuse","tagline":"Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.","github_url":"https://github.com/langfuse/langfuse","owner":"langfuse","repo":"langfuse","owner_avatar_url":"https://avatars.githubusercontent.com/u/134601687?v=4","primary_language":"TypeScript","stars":34763,"forks":3793,"topics":["analytics","autogen","evaluation","langchain","large-language-models","llama-index","llm","llm-evaluation","llm-observability","llmops","monitoring","observability","open-source","openai","playground","prompt-engineering","prompt-management","self-hosted","ycombinator"],"archived":false,"github_pushed_at":"2026-09-18T08:08:19+00:00","maintenance_label":"Very active","stars_delta_30d":2492,"url":"https://www.graphcanon.com/tools/langfuse-langfuse","markdown_url":"https://www.graphcanon.com/tools/langfuse-langfuse.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/langfuse-langfuse","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=langfuse-langfuse"}},{"type":"alternative","direction":"out","explanation":"Both tools offer observability solutions for ML and LLM models, but Evidently is an open-source framework tailored toward broader ML applications while lmms-eval focuses specifically on multimodal evaluation across various data types.","successor_context":null,"tool":{"slug":"evidentlyai-evidently","name":"evidently","tagline":"An open-source ML and LLM observability framework for evaluating, testing, and monitoring AI systems and data pipelines.","github_url":"https://github.com/evidentlyai/evidently","owner":"evidentlyai","repo":"evidently","owner_avatar_url":"https://avatars.githubusercontent.com/u/75031056?v=4","primary_language":"Jupyter Notebook","stars":7924,"forks":919,"topics":["data-drift","data-quality","data-science","data-validation","generative-ai","hacktoberfest","html-report","jupyter-notebook","llm","llmops","machine-learning","mlops","model-monitoring","pandas-dataframe"],"archived":false,"github_pushed_at":"2026-09-11T09:03:28+00:00","maintenance_label":"Very active","stars_delta_30d":134,"url":"https://www.graphcanon.com/tools/evidentlyai-evidently","markdown_url":"https://www.graphcanon.com/tools/evidentlyai-evidently.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/evidentlyai-evidently","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=evidentlyai-evidently"}},{"type":"related","direction":"in","explanation":null,"successor_context":null,"tool":{"slug":"prometheus-eval-prometheus-eval","name":"prometheus-eval","tagline":"Evaluate your LLM's response with Prometheus and GPT4","github_url":"https://github.com/prometheus-eval/prometheus-eval","owner":"prometheus-eval","repo":"prometheus-eval","owner_avatar_url":"https://avatars.githubusercontent.com/u/167460660?v=4","primary_language":"Python","stars":1116,"forks":69,"topics":["evaluation","gpt4","litellm","llm","llm-as-a-judge","llm-as-evaluator","llmops","python","vllm"],"archived":false,"github_pushed_at":"2025-04-25T03:58:37+00:00","maintenance_label":"Dormant","stars_delta_30d":9,"url":"https://www.graphcanon.com/tools/prometheus-eval-prometheus-eval","markdown_url":"https://www.graphcanon.com/tools/prometheus-eval-prometheus-eval.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/prometheus-eval-prometheus-eval","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=prometheus-eval-prometheus-eval"}},{"type":"successor","direction":"in","explanation":"VLMEvalKit has a 'successor' relationship to 'lmms-eval' because while lmms-eval provides a unified evaluation framework for multimodal large language models across various media types, VLMEvalKit specifically focuses on and simplifies the evaluation of vision-language models through dedicated generation-based methods and exact matching techniques, offering more specialized capabilities within a细分","successor_context":null,"tool":{"slug":"open-compass-vlmevalkit","name":"VLMEvalKit","tagline":"An open-source evaluation toolkit for large vision-language models","github_url":"https://github.com/open-compass/VLMEvalKit","owner":"open-compass","repo":"VLMEvalKit","owner_avatar_url":"https://avatars.githubusercontent.com/u/143521324?v=4","primary_language":"Python","stars":4345,"forks":745,"topics":["chatgpt","claude","clip","computer-vision","evaluation","gemini","gpt","gpt-4v","gpt4","large-language-models","llava","llm","multi-modal","openai","openai-api","pytorch","qwen","vit","vqa"],"archived":false,"github_pushed_at":"2026-08-17T17:16:26+00:00","maintenance_label":"Very active","stars_delta_30d":60,"url":"https://www.graphcanon.com/tools/open-compass-vlmevalkit","markdown_url":"https://www.graphcanon.com/tools/open-compass-vlmevalkit.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/open-compass-vlmevalkit","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=open-compass-vlmevalkit"}},{"type":"integrates_with","direction":"in","explanation":"The LMMS-evolution toolkit could be used for evaluating models highlighted in this repository.","successor_context":null,"tool":{"slug":"wangrongsheng-awesome-llm-resources","name":"awesome-LLM-resources","tagline":"Summary of the world's best LLM resources.","github_url":"https://github.com/WangRongsheng/awesome-LLM-resources","owner":"WangRongsheng","repo":"awesome-LLM-resources","owner_avatar_url":"https://avatars.githubusercontent.com/u/55651568?v=4","primary_language":null,"stars":8968,"forks":993,"topics":["awesome-list","book","course","large-language-models","llama","llm","mistral","openai","qwen","rag","retrieval-augmented-generation","webui"],"archived":false,"github_pushed_at":"2026-09-14T09:00:10+00:00","maintenance_label":"Very active","stars_delta_30d":123,"url":"https://www.graphcanon.com/tools/wangrongsheng-awesome-llm-resources","markdown_url":"https://www.graphcanon.com/tools/wangrongsheng-awesome-llm-resources.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/wangrongsheng-awesome-llm-resources","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=wangrongsheng-awesome-llm-resources"}},{"type":"alternative","direction":"in","explanation":"VLMEvalKit and lmms-eval both serve as evaluation toolkits for multimodal large language models, addressing the same problem with potentially different approaches or features.","successor_context":null,"tool":{"slug":"open-compass-vlmevalkit","name":"VLMEvalKit","tagline":"An open-source evaluation toolkit for large vision-language models","github_url":"https://github.com/open-compass/VLMEvalKit","owner":"open-compass","repo":"VLMEvalKit","owner_avatar_url":"https://avatars.githubusercontent.com/u/143521324?v=4","primary_language":"Python","stars":4345,"forks":745,"topics":["chatgpt","claude","clip","computer-vision","evaluation","gemini","gpt","gpt-4v","gpt4","large-language-models","llava","llm","multi-modal","openai","openai-api","pytorch","qwen","vit","vqa"],"archived":false,"github_pushed_at":"2026-08-17T17:16:26+00:00","maintenance_label":"Very active","stars_delta_30d":60,"url":"https://www.graphcanon.com/tools/open-compass-vlmevalkit","markdown_url":"https://www.graphcanon.com/tools/open-compass-vlmevalkit.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/open-compass-vlmevalkit","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=open-compass-vlmevalkit"}},{"type":"integrates_with","direction":"in","explanation":"NExT-GPT can be evaluated using lmms-eval, which is a unified evaluation toolkit for multimodal large language models.","successor_context":null,"tool":{"slug":"next-gpt-next-gpt","name":"NExT-GPT","tagline":"Code and models for ICML 2024 paper on multimodal large language model","github_url":"https://github.com/NExT-GPT/NExT-GPT","owner":"NExT-GPT","repo":"NExT-GPT","owner_avatar_url":"https://avatars.githubusercontent.com/u/143576855?v=4","primary_language":"Python","stars":3637,"forks":359,"topics":["chatgpt","foundation-models","gpt-4","instruction-tuning","large-language-models","llm","mllm","multi-modal-chatgpt","multimodal","visual-language-learning"],"archived":false,"github_pushed_at":"2025-05-13T09:57:47+00:00","maintenance_label":"Dormant","stars_delta_30d":-1,"url":"https://www.graphcanon.com/tools/next-gpt-next-gpt","markdown_url":"https://www.graphcanon.com/tools/next-gpt-next-gpt.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/next-gpt-next-gpt","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=next-gpt-next-gpt"}},{"type":"related","direction":"in","explanation":"Both deal with multimodal LLMs; MGM can be evaluated using lmms-eval toolkit, though they serve different functions.","successor_context":null,"tool":{"slug":"jia-lab-research-mgm","name":"MGM","tagline":"Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models","github_url":"https://github.com/JIA-Lab-research/MGM","owner":"JIA-Lab-research","repo":"MGM","owner_avatar_url":"https://avatars.githubusercontent.com/u/64006090?v=4","primary_language":"Python","stars":3331,"forks":276,"topics":["generation","large-language-models","vision-language-model"],"archived":false,"github_pushed_at":"2024-05-04T14:36:51+00:00","maintenance_label":"Dormant","stars_delta_30d":1,"url":"https://www.graphcanon.com/tools/jia-lab-research-mgm","markdown_url":"https://www.graphcanon.com/tools/jia-lab-research-mgm.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/jia-lab-research-mgm","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=jia-lab-research-mgm"}}],"neighbours":[{"slug":"confident-ai-deepeval","name":"deepeval","tagline":"LLM Evaluation Framework.","github_url":"https://github.com/confident-ai/deepeval","owner":"confident-ai","repo":"deepeval","owner_avatar_url":"https://avatars.githubusercontent.com/u/130858411?v=4","primary_language":"Python","stars":18342,"forks":1953,"topics":["evaluation-framework","evaluation-metrics","llm-evaluation","llm-evaluation-framework","llm-evaluation-metrics","python"],"archived":false,"github_pushed_at":"2026-09-18T17:06:58+00:00","maintenance_label":"Active","url":"https://www.graphcanon.com/tools/confident-ai-deepeval","markdown_url":"https://www.graphcanon.com/tools/confident-ai-deepeval.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/confident-ai-deepeval","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=confident-ai-deepeval","shared_categories":["evaluation-observability"]},{"slug":"eleutherai-lm-evaluation-harness","name":"lm-evaluation-harness","tagline":"A framework for few-shot evaluation of language models.","github_url":"https://github.com/EleutherAI/lm-evaluation-harness","owner":"EleutherAI","repo":"lm-evaluation-harness","owner_avatar_url":"https://avatars.githubusercontent.com/u/68924597?v=4","primary_language":"Python","stars":14192,"forks":3664,"topics":["evaluation-framework","language-model","transformer"],"archived":false,"github_pushed_at":"2026-09-14T10:51:06+00:00","maintenance_label":"Active","url":"https://www.graphcanon.com/tools/eleutherai-lm-evaluation-harness","markdown_url":"https://www.graphcanon.com/tools/eleutherai-lm-evaluation-harness.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/eleutherai-lm-evaluation-harness","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=eleutherai-lm-evaluation-harness","shared_categories":["evaluation-observability"]},{"slug":"oumi-ai-oumi","name":"oumi","tagline":"Easily fine-tune, evaluate and deploy open source LLMs/VLMs","github_url":"https://github.com/oumi-ai/oumi","owner":"oumi-ai","repo":"oumi","owner_avatar_url":"https://avatars.githubusercontent.com/u/167452922?v=4","primary_language":"Python","stars":9387,"forks":790,"topics":["dpo","evaluation","fine-tuning","gpt-oss","gpt-oss-120b","gpt-oss-20b","inference","llama","llms","open-weight","open-weight-models","open-weights","sft","slms","vlms"],"archived":false,"github_pushed_at":"2026-09-20T05:45:37+00:00","maintenance_label":"Active","url":"https://www.graphcanon.com/tools/oumi-ai-oumi","markdown_url":"https://www.graphcanon.com/tools/oumi-ai-oumi.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/oumi-ai-oumi","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=oumi-ai-oumi","shared_categories":["evaluation-observability"]},{"slug":"andyyyy64-whichllm","name":"whichllm","tagline":"Command-line tool to find and benchmark local LLM performance","github_url":"https://github.com/Andyyyy64/whichllm","owner":"Andyyyy64","repo":"whichllm","owner_avatar_url":"https://avatars.githubusercontent.com/u/105579829?v=4","primary_language":"Python","stars":6666,"forks":368,"topics":["localllm"],"archived":false,"github_pushed_at":"2026-09-19T16:22:49+00:00","maintenance_label":"Active","url":"https://www.graphcanon.com/tools/andyyyy64-whichllm","markdown_url":"https://www.graphcanon.com/tools/andyyyy64-whichllm.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/andyyyy64-whichllm","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=andyyyy64-whichllm","shared_categories":["evaluation-observability"]},{"slug":"luban-agi-awesome-aigc-tutorials","name":"Awesome-AIGC-Tutorials","tagline":"Curated tutorials and resources for Large Language Models, AI Painting, and more","github_url":"https://github.com/luban-agi/Awesome-AIGC-Tutorials","owner":"luban-agi","repo":"Awesome-AIGC-Tutorials","owner_avatar_url":"https://avatars.githubusercontent.com/u/142385464?v=4","primary_language":null,"stars":4547,"forks":298,"topics":["ai","aigc","awesome","chatgpt","courses-resource","deep-learning","llm","midjourney","multimodal","nlp","prompt-engineering","stable-diffusion","tutorials"],"archived":false,"github_pushed_at":"2024-03-31T09:18:04+00:00","maintenance_label":"Dormant","url":"https://www.graphcanon.com/tools/luban-agi-awesome-aigc-tutorials","markdown_url":"https://www.graphcanon.com/tools/luban-agi-awesome-aigc-tutorials.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/luban-agi-awesome-aigc-tutorials","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=luban-agi-awesome-aigc-tutorials","shared_categories":[]},{"slug":"nyldn-claude-octopus","name":"claude-octopus","tagline":"Surface AI blindspots before you ship","github_url":"https://github.com/nyldn/claude-octopus","owner":"nyldn","repo":"claude-octopus","owner_avatar_url":"https://avatars.githubusercontent.com/u/4805949?v=4","primary_language":"Shell","stars":4090,"forks":378,"topics":["ai-agents","ai-orchestration","claude-code","claude-code-plugin","codex","copilot","developer-tools","double-diamond","gemini","multi-ai","multi-llm","ollama"],"archived":false,"github_pushed_at":"2026-09-20T05:07:44+00:00","maintenance_label":"Active","url":"https://www.graphcanon.com/tools/nyldn-claude-octopus","markdown_url":"https://www.graphcanon.com/tools/nyldn-claude-octopus.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/nyldn-claude-octopus","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=nyldn-claude-octopus","shared_categories":[]},{"slug":"stochasticai-xturing","name":"xTuring","tagline":"Build, personalize and control your own LLMs","github_url":"https://github.com/stochasticai/xTuring","owner":"stochasticai","repo":"xTuring","owner_avatar_url":"https://avatars.githubusercontent.com/u/66399337?v=4","primary_language":"Python","stars":2674,"forks":211,"topics":["adapter","deep-learning","fine-tuning","finetuning","gen-ai","generative-ai","gpt-2","gpt-j","language-model","llama","llm","lora","mistral","mixed-precision","peft","quantization"],"archived":false,"github_pushed_at":"2026-09-12T23:50:56+00:00","maintenance_label":"Steady","url":"https://www.graphcanon.com/tools/stochasticai-xturing","markdown_url":"https://www.graphcanon.com/tools/stochasticai-xturing.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/stochasticai-xturing","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=stochasticai-xturing","shared_categories":[]},{"slug":"benchflow-ai-awesome-evals","name":"awesome-evals","tagline":"A curated library of resources for building and evaluating AI agents","github_url":"https://github.com/benchflow-ai/awesome-evals","owner":"benchflow-ai","repo":"awesome-evals","owner_avatar_url":"https://avatars.githubusercontent.com/u/190338344?v=4","primary_language":null,"stars":901,"forks":104,"topics":["agent-evaluation","ai-agents","awesome","awesome-list","benchmarks","evals","llm","llm-evaluation","rl-environments"],"archived":false,"github_pushed_at":"2026-09-15T17:49:19+00:00","maintenance_label":"Active","url":"https://www.graphcanon.com/tools/benchflow-ai-awesome-evals","markdown_url":"https://www.graphcanon.com/tools/benchflow-ai-awesome-evals.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/benchflow-ai-awesome-evals","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=benchflow-ai-awesome-evals","shared_categories":["evaluation-observability"]},{"slug":"georgian-io-llm-finetuning-toolkit","name":"LLM-Finetuning-Toolkit","tagline":"Toolkit for fine-tuning and testing open-source large language models","github_url":"https://github.com/georgian-io/LLM-Finetuning-Toolkit","owner":"georgian-io","repo":"LLM-Finetuning-Toolkit","owner_avatar_url":"https://avatars.githubusercontent.com/u/10764713?v=4","primary_language":"Python","stars":870,"forks":107,"topics":["ablation-study","classification","falcon","fine-tuning","finetuning","flan-t5","large-language-models","llama2","llm-test","lora","mistral-7b","nlp","nlp-machine-learning","qlora","redpajama","summarization","unit-testing","zephyr"],"archived":false,"github_pushed_at":"2026-05-04T16:33:40+00:00","maintenance_label":"Slowing","url":"https://www.graphcanon.com/tools/georgian-io-llm-finetuning-toolkit","markdown_url":"https://www.graphcanon.com/tools/georgian-io-llm-finetuning-toolkit.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/georgian-io-llm-finetuning-toolkit","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=georgian-io-llm-finetuning-toolkit","shared_categories":[]},{"slug":"openinfer-project-openinfer","name":"openinfer","tagline":"Pure Rust CUDA LLM inference engine serving multiple models including Qwen3 and Kimi-K2","github_url":"https://github.com/openinfer-project/openinfer","owner":"openinfer-project","repo":"openinfer","owner_avatar_url":"https://avatars.githubusercontent.com/u/292134277?v=4","primary_language":"Rust","stars":705,"forks":107,"topics":["cuda","cuda-kernels","deepseek","gpu","inference","inference-engine","kimi","kimi-k2","kv-cache","llm","llm-inference","llm-serving","model-serving","moe","openai-api","paged-attention","qwen","qwen3","rust","vllm"],"archived":false,"github_pushed_at":"2026-09-18T17:57:56+00:00","maintenance_label":"Active","url":"https://www.graphcanon.com/tools/openinfer-project-openinfer","markdown_url":"https://www.graphcanon.com/tools/openinfer-project-openinfer.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/openinfer-project-openinfer","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=openinfer-project-openinfer","shared_categories":[]},{"slug":"declare-lab-instruct-eval","name":"instruct-eval","tagline":"Quantitative evaluation for instruction-tuned language models","github_url":"https://github.com/declare-lab/instruct-eval","owner":"declare-lab","repo":"instruct-eval","owner_avatar_url":"https://avatars.githubusercontent.com/u/59164695?v=4","primary_language":"Python","stars":553,"forks":45,"topics":["instruct-tuning","llm"],"archived":false,"github_pushed_at":"2024-03-10T05:00:00+00:00","maintenance_label":"Dormant","url":"https://www.graphcanon.com/tools/declare-lab-instruct-eval","markdown_url":"https://www.graphcanon.com/tools/declare-lab-instruct-eval.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/declare-lab-instruct-eval","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=declare-lab-instruct-eval","shared_categories":["evaluation-observability"]},{"slug":"curated-awesome-lists-awesome-llms-fine-tuning","name":"awesome-llms-fine-tuning","tagline":"A comprehensive collection of resources for fine-tuning Large Language Models.","github_url":"https://github.com/Curated-Awesome-Lists/awesome-llms-fine-tuning","owner":"Curated-Awesome-Lists","repo":"awesome-llms-fine-tuning","owner_avatar_url":"https://avatars.githubusercontent.com/u/142611331?v=4","primary_language":null,"stars":527,"forks":80,"topics":["ai","awesome-list","deep-learning","fine-tuning","gpt","large-language-models","llms","machine-learning","nlp","transformers"],"archived":false,"github_pushed_at":"2026-09-04T19:22:27+00:00","maintenance_label":"Steady","url":"https://www.graphcanon.com/tools/curated-awesome-lists-awesome-llms-fine-tuning","markdown_url":"https://www.graphcanon.com/tools/curated-awesome-lists-awesome-llms-fine-tuning.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/curated-awesome-lists-awesome-llms-fine-tuning","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=curated-awesome-lists-awesome-llms-fine-tuning","shared_categories":[]}]}}