{"data":{"slug":"stanford-futuredata-ares","name":"ARES","tagline":"Automated Evaluation of RAG Systems","github_url":"https://github.com/stanford-futuredata/ARES","owner":"stanford-futuredata","repo":"ARES","owner_avatar_url":"https://avatars.githubusercontent.com/u/16332752?v=4","primary_language":"Python","stars":733,"forks":67,"topics":[],"archived":false,"github_pushed_at":"2025-03-28T11:25:50+00:00","maintenance_label":"Dormant","stars_delta_30d":2,"url":"https://www.graphcanon.com/tools/stanford-futuredata-ares","markdown_url":"https://www.graphcanon.com/tools/stanford-futuredata-ares.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/stanford-futuredata-ares","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=stanford-futuredata-ares","description":"Automated Evaluation of RAG Systems","homepage_url":"https://ares-ai.vercel.app/","license":"Apache-2.0","open_issues":22,"watchers":10,"ai_summary":"ARES is a Python-based tool for automated evaluation of RAG (Retrieval-Augmented Generation) systems, providing scoring and comparison functionalities.","readme_excerpt":"### ⚙️ Installation\n<a id=\"section1\"></a>\n<hr>\n​\nTo install ARES, run the following commands:\n​\n\n```python\n\npip install ares-ai\n\n```\n​\n*Optional: Initalize OpenAI or TogetherAI API key with the following command:*\n\n\n```python\n\nexport OPENAI_API_KEY=<your key here>\nexport TOGETHER_API_KEY=<your key here>\n\n```\n\n---\n\n### 📝 Requirements\n<a id=\"section2\"></a>\n<hr>\n\nTo implement ARES for scoring your RAG system and comparing to other RAG configurations, you need three components:​\n\n* A human preference validation set of annotated query, document, and answer triples for the evaluation criteria (e.g. context relevance, answer faithfulness, and/or answer relevance). There should be at least 50 examples but several hundred examples is ideal.\n* A set of few-shot examples for scoring context relevance, answer faithfulness, and/or answer relevance in your system\n* A much larger set of unlabeled query-document-answer triples outputted by your RAG system for scoring\n\n<a id=\"section3\"></a>\n<hr>\n\nTo get started with ARES, you'll need to set up your configuration. Below is an example of a configuration for ARES!\n\nCopy-paste each step to see ARES in action!\n\n<hr>\n\n---\n\n# For purposes of our quick start guide, we rename nq_ratio_0.5 to nq_unlabeled_output and nq_labeled_output.\n```\n<hr>\n\n---\n\n### 🚀 Quick Start - #1\n\n<hr>\n\nTo get started with ARES's PPI, you'll need to set up your configuration. Below is an example of a configuration for ARES!\n\nJust copy-paste as you go to see ARES in action!\n\n#### Step 1) Run the following to retrieve the UES/IDP scores with GPT3.5!\n\n```python\nfrom ares import ARES\n\nues_idp_config = {\n    \"in_domain_prompts_dataset\": \"nq_few_shot_prompt_for_judge_scoring.tsv\",\n    \"unlabeled_evaluation_set\": \"nq_unlabeled_output.tsv\", \n    \"model_choice\" : \"gpt-3.5-turbo-0125\"\n} \n\nares = ARES(ues_idp=ues_idp_config)\nresults = ares.ues_idp()\nprint(results)\n\n---\n\n### 🚀 Quick Start - #2\n\n<hr>\n\n#### Step 1) Run the following to see GPT 3.5's accuracy on the NQ unlabeled dataset!\n\n```python\nfrom ares import ARES\n\nues_idp_config = {\n    \"in_domain_prompts_dataset\": \"nq_few_shot_prompt_for_judge_scoring.tsv\",\n    \"unlabeled_evaluation_set\": \"nq_unlabeled_output.tsv\", \n    \"model_choice\" : \"gpt-3.5-turbo-0125\"\n} \n\nares = ARES(ues_idp=ues_idp_config)\nresults = ares.ues_idp()\nprint(results)\n\n---\n\n### Machine requirements and setup when not using OpenAI API\n**Machine requirements**\n\n- Over ~100 GB of available disk space\n- GPU\n    - Should work: A100 (e.g. `Standard_NC24ads_A100_v4` on Azure)\n    - Does not work:\n        - Tested on 2023-12-17 with both `Standard_NC6s_v3` and `Standard_NC12s_v3`, and ran into this error: `torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 160.00 MiB (GPU 0; 15.77 GiB total capacity; 15.12 GiB already allocated; 95.44 MiB free; 15.12 GiB reserved in total by PyTorch)`\n\n\n**Machine setup**\n\nFor example, on an Azure VM running Linux (ubuntu 20.04), you will need to do the following:\n- Install conda\n    - First set of commands (can copy-paste multiple lines)\n        - `wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh`\n        - `chmod +x Miniconda3-latest-Linux-x86_64.sh`\n        - `./Miniconda3-latest-Linux-x86_64.sh -b`\n    - Second set of commands (can copy-paste multiple lines)\n        - `export PATH=\"~/miniconda3/bin:$PATH\"`\n        - `conda init`\n- Install gcc\n    - `sudo apt-get -y update`\n    - `sudo apt-get -y upgrade`\n    - `sudo apt-get -y install build-essential`\n    - `sudo apt-get -y install libpcre3-dev`\n- Install NVIDIA drivers\n    - `sudo apt install ubuntu-drivers-common -y`\n    - `sudo ubuntu-drivers autoinstall`\n    - `sudo reboot`\n    - SSH in again and confirm the installation was successful by running `nvidia-smi`\n- `cd` to ARES folder and follow the rest of the README","github_created_at":"2023-09-27T03:56:19+00:00","created_at":"2026-07-11T23:07:15.932604+00:00","updated_at":"2026-09-20T04:29:07.097063+00:00","categories":[{"slug":"evaluation-observability","name":"Evaluation & Observability","url":"https://www.graphcanon.com/categories/evaluation-observability","markdown_url":"https://www.graphcanon.com/categories/evaluation-observability.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/evaluation-observability"}],"tags":[{"slug":"api-integration","name":"api integration"},{"slug":"automated-evaluation","name":"automated evaluation"},{"slug":"comparison","name":"comparison"},{"slug":"evaluation-criteria","name":"evaluation criteria"},{"slug":"gpt-3-5","name":"gpt-3.5"},{"slug":"python","name":"python"},{"slug":"rag-systems","name":"rag systems"},{"slug":"scoring","name":"scoring"}],"trust":{"provenance":{"is_fork":false,"github_id":697100198,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-09-18T08:13:01.923Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":538,"last_release_at":null,"stars_delta_30d":2,"open_issues_delta_30d":1},"security_summary":{"status":"findings","scanner":"osv@v1","low_count":344,"high_count":0,"last_scan_at":"2026-09-18T08:13:04.376Z","medium_count":0,"scan_profile":"deps","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-09-18T08:13:00.740Z"},"has_cli":{"value":true,"source":"pyproject.toml:[project.scripts]","observed_at":"2026-09-18T08:13:00.740Z"},"languages":{"value":["python"],"source":"github.language+pyproject.toml","observed_at":"2026-09-18T08:13:00.740Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-09-18T08:13:00.740Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":{"notes":["Requires over 100 GB of available disk space and a compatible GPU like the A100.","Installation of conda and gcc is necessary for setup."],"min_ram_gb":null,"requires_docker":false},"constraints":{"min_ram_gb":null,"requires_docker":false},"when_to_use":["Use ARES when you need to evaluate RAG systems with specific criteria such as context relevance, answer faithfulness, and answer relevance.","Consider ARES if you have a human preference validation set of at least 50 annotated query-document-answer triples, and a larger set of unlabeled triples for scoring.","Use ARES when you can provide a set of few-shot examples for scoring context relevance, answer faithfulness, and answer relevance."],"when_not_to_use":["Do not use ARES if you lack the necessary human-annotated datasets for evaluation criteria or if you do not have a large set of unlabeled triples for scoring.","Avoid ARES if your machine does not meet the requirements, such as having over 100 GB of available disk space and a compatible GPU like the A100.","Do not use ARES if you cannot set up the required environment, including conda and gcc, on your machine."],"source":"enrich:decision_facts","observed_at":"2026-09-18T09:05:22.746Z"},"constraint_facets":{"min_ram_gb":null,"requires_docker":false},"decision_summary":[{"label":"Requirements","value":"Requires over 100 GB of available disk space and a compatible GPU like the A100.; Installation of conda and gcc is necessary for setup."},{"label":"Adopt for","value":"ARES is a Python-based tool for automated evaluation of RAG systems, offering scoring and comparison functionalities."}]}}