{"data":{"slug":"onejune2018-awesome-llm-eval","name":"Awesome-LLM-Eval","tagline":"Curated list for evaluation of large language models","github_url":"https://github.com/onejune2018/Awesome-LLM-Eval","owner":"onejune2018","repo":"Awesome-LLM-Eval","owner_avatar_url":"https://avatars.githubusercontent.com/u/35061855?v=4","primary_language":null,"stars":656,"forks":90,"topics":["awsome-list","awsome-lists","benchmark","bert","chatglm","chatgpt","dataset","evaluation","gpt3","large-language-model","leaderboard","llama","llm","llm-evaluation","machine-learning","nlp","openai","qwen","rag"],"archived":false,"github_pushed_at":"2025-11-24T01:59:12+00:00","maintenance_label":"Slowing","stars_delta_30d":2,"url":"https://www.graphcanon.com/tools/onejune2018-awesome-llm-eval","markdown_url":"https://www.graphcanon.com/tools/onejune2018-awesome-llm-eval.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/onejune2018-awesome-llm-eval","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=onejune2018-awesome-llm-eval","description":"Awesome-LLM-Eval: a curated list of tools, datasets/benchmark, demos, leaderboard, papers, docs and models, mainly for Evaluation on LLMs.  一个由工具、基准/数据、演示、排行榜和大模型等组成的精选列表，主要面向基础大模型评测，旨在探求生成式AI的技术边界.","homepage_url":"https://arxiv.org/abs/2508.18646","license":"MIT","open_issues":55,"watchers":8,"ai_summary":"Awesome-LLM-Eval: a curated list featuring tools, benchmarks, datasets, leaderboard, and documents for evaluating large language models.","readme_excerpt":"### Leaderboards for popular Provider (performance and cost, 2024-05-14)\n\n| Provider (link to pricing)                                   | [OpenAI](https://openai.com/pricing) | [OpenAI](https://openai.com/pricing) | [Anthropic](https://www.anthropic.com/api) | [Google](https://cloud.google.com/vertex-ai/generative-ai/pricing) | [Replicate](https://replicate.com/pricing)                   | [DeepSeek](https://www.deepseek.com/)                        | [Mistral](https://docs.mistral.ai/platform/pricing/)         | [Anthropic](https://www.anthropic.com/api) | [Google](https://cloud.google.com/vertex-ai/generative-ai/pricing) | [Mistral](https://docs.mistral.ai/platform/pricing/) | [Cohere](https://cohere.com/command)                         | [Anthropic](https://www.anthropic.com/api) | [Mistral](https://docs.mistral.ai/platform/pricing/) | [Replicate](https://replicate.com/pricing)                   | [Mistral](https://docs.mistral.ai/platform/pricing/)         | [OpenAI](https://openai.com/pricing) |      | [Groq](https://wow.groq.com/) | [OpenAI](https://openai.com/pricing) | [Mistral](https://docs.mistral.ai/platform/pricing/) | [Anthropic](https://www.anthropic.com/api) | [Groq](https://wow.groq.com/) | [Anthropic](https://www.anthropic.com/api) | [Anthropic](https://www.anthropic.com/api) | Microsoft                                                    | Microsoft                                                    | [Mistral](https://docs.mistral.ai/platform/pricing/) |\n| ------------------------------------------------------------ | ------------------------------------ | ------------------------------------ | ------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------ | ------------------------------------------------------------ | ---------------------------------------------------- | ------------------------------------------------------------ | ------------------------------------------ | ---------------------------------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------ | ---- | ----------------------------- | ------------------------------------ | ---------------------------------------------------- | ------------------------------------------ | ----------------------------- | ------------------------------------------ | ------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | ---------------------------------------------------- |\n| Model name                                                   | GPT-4o                               | GPT-4 Turbo                          | Claude 3 Opus                              | Gemini 1.5 Pro                                               | [Llama 3 70B](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct) | [DeepSeek-V2](https://huggingface.co/deepseek-ai/DeepSeek-V2-Chat) | [Mixtral 8x22B](https://huggingface.co/mistralai/Mixtral-8x22B-Instruct-v0.1) | Claude 3 Sonnet                            | Gemini 1.5 Flash                                             | Mistral Large                                        | [Command R+](https://huggingface.co/CohereForAI/c4ai-command-r-plus) | Claude 3 Haiku                             | Mistral Small                                        | [Llama 3 8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) | [Mixtral 8x7B](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1) | GPT-3.5 Turbo                        |      | Llama 3 70B (Groq)            | GPT-4                                | Mistral M","github_created_at":"2023-04-26T10:10:06+00:00","created_at":"2026-07-11T11:59:50.251234+00:00","updated_at":"2026-09-20T06:57:41.795835+00:00","categories":[{"slug":"evaluation-observability","name":"Evaluation & Observability","url":"https://www.graphcanon.com/categories/evaluation-observability","markdown_url":"https://www.graphcanon.com/categories/evaluation-observability.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/evaluation-observability"}],"tags":[{"slug":"awesome-list","name":"awesome-list"},{"slug":"benchmark","name":"benchmark"},{"slug":"datasets","name":"datasets"},{"slug":"evaluation","name":"evaluation"},{"slug":"large-language-models","name":"large-language-models"},{"slug":"leaderboard","name":"leaderboard"},{"slug":"llm-evaluation","name":"llm-evaluation"},{"slug":"nlp","name":"nlp"}],"trust":{"provenance":{"is_fork":false,"github_id":632878141,"owner_type":"User","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-09-20T06:57:39.986Z","maintenance":{"label":"Slowing","score":36,"methodology":"github_public_v1","releases_90d":0,"days_since_push":300,"last_release_at":null,"stars_delta_30d":2,"open_issues_delta_30d":11},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T11:59:51.521Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-09-20T06:57:41.247Z"},"license_spdx":{"value":"MIT","source":"github.license","observed_at":"2026-09-20T06:57:41.247Z"}},"decision_facts":{"hosting":null,"pricing":{"model":"freemium","summary":"The core resources listed in Awesome-LLM-Eval are freely accessible under MIT license, however, certain datasets or tools might have individual licensing terms."},"requirements":{"notes":["The resources listed may vary in their own requirements, including software dependencies and hardware specifications."],"min_ram_gb":null,"requires_docker":false},"constraints":{"min_ram_gb":null,"pricing_model":"freemium","requires_docker":false},"when_to_use":["When you specifically need access to an extensive compilation of evaluation-related resources tailored towards large language model assessment.","If your focus is on the latest advancements in LLM evaluations as documented by a curated list that covers wide-ranging aspects from benchmarks to demonstrations."],"when_not_to_use":["You require real-time testing capabilities or interactive features; Awesome-LLM-Eval is a static resource list and not an interactive platform.","If integration with specific third-party platforms or direct API access is necessary, since the repository predominantly serves as a reference point rather than an operational tool."],"source":"enrich:decision_facts","observed_at":"2026-07-17T00:03:22.922Z"},"constraint_facets":{"min_ram_gb":null,"pricing_model":"freemium","requires_docker":false},"decision_summary":[{"label":"Pricing","value":"freemium - The core resources listed in Awesome-LLM-Eval are freely accessible under MIT license, however, certain datasets or tools might have individual licensing terms."},{"label":"Requirements","value":"The resources listed may vary in their own requirements, including software dependencies and hardware specifications."},{"label":"Adopt for","value":"Awesome-LLM-Eval provides a comprehensive curated list of resources for evaluating large language models including tools, datasets, and benchmarks."}]}}