GraphCanon updated 2w · GitHub synced 2w
Decision brief
Automated evaluation for RAG systems with API integrations like OpenAI.
Good fit when
- Evaluating Retrieval-Augmented Generation (RAG) systems that require automatic scoring using human-annotated data and few-shot examples.
Avoid when
- Avoid if limited to non-GPU machines with less than ~100GB available disk space, as it encounters CUDA out-of-memory errors without compatible GPU setups.
Observed Jul 12, 2026 · Source: enrich:decision_facts
Verify the decision
Maintenance and security
Full trust report- Maintenance
- Dormant (491d since push)
- As of 2w
- Provenance
- Not a fork · Organization account
- As of 2w
- Security (OSV)
- 154 low (154 low)
- As of 1mo
Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.
Install
pip install ARESSimilar tools
Same-category neighbours. No typed graph edges are catalogued for this tool yet.
Evidence and technical details
Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.
Overview
ARES is a tool for automating the evaluation and scoring of Retrieval-Augmented Generation (RAG) systems using human-annotated datasets, few-shot examples, and a large set of unlabeled triples generated by RAG systems. It supports multiple API integrations such as OpenAI and Together AI.
Capability facts
- CLI
- CLI entrypoint
Source: pyproject.toml:[project.scripts] · Aug 1, 2026
- Languages
- python
Source: github.language+pyproject.toml · Aug 1, 2026
Categories
Compatibility
Sourced claims from the README excerpt - not unsourced marketing copy.
Source: README excerpt (regex_v1, Aug 1, 2026)
### Machine requirements and setup when not using OpenAI APISource link
Source: README excerpt (regex_v1, Aug 1, 2026)
```python pip install ares-aiSource link
Tags
README
⚙️ Installation
To install ARES, run the following commands:
pip install ares-ai
Optional: Initalize OpenAI or TogetherAI API key with the following command:
export OPENAI_API_KEY=<your key here>
export TOGETHER_API_KEY=<your key here>
📝 Requirements
To implement ARES for scoring your RAG system and comparing to other RAG configurations, you need three components:
- A human preference validation set of annotated query, document, and answer triples for the evaluation criteria (e.g. context relevance, answer faithfulness, and/or answer relevance). There should be at least 50 examples but several hundred examples is ideal.
- A set of few-shot examples for scoring context relevance, answer faithfulness, and/or answer relevance in your system
- A much larger set of unlabeled query-document-answer triples outputted by your RAG system for scoring
To get started with ARES, you'll need to set up your configuration. Below is an example of a configuration for ARES!
Copy-paste each step to see ARES in action!
For purposes of our quick start guide, we rename nq_ratio_0.5 to nq_unlabeled_output and nq_labeled_output.
<hr>
---
### 🚀 Quick Start - #1
<hr>
To get started with ARES's PPI, you'll need to set up your configuration. Below is an example of a configuration for ARES!
Just copy-paste as you go to see ARES in action!
#### Step 1) Run the following to retrieve the UES/IDP scores with GPT3.5!
```python
from ares import ARES
ues_idp_config = {
"in_domain_prompts_dataset": "nq_few_shot_prompt_for_judge_scoring.tsv",
"unlabeled_evaluation_set": "nq_unlabeled_output.tsv",
"model_choice" : "gpt-3.5-turbo-0125"
}
ares = ARES(ues_idp=ues_idp_config)
results = ares.ues_idp()
print(results)
---
### 🚀 Quick Start - #2
<hr>
#### Step 1) Run the following to see GPT 3.5's accuracy on the NQ unlabeled dataset!
```python
from ares import ARES
ues_idp_config = {
"in_domain_prompts_dataset": "nq_few_shot_prompt_for_judge_scoring.tsv",
"unlabeled_evaluation_set": "nq_unlabeled_output.tsv",
"model_choice" : "gpt-3.5-turbo-0125"
}
ares = ARES(ues_idp=ues_idp_config)
results = ares.ues_idp()
print(results)
---
### Machine requirements and setup when not using OpenAI API
**Machine requirements**
- Over ~100 GB of available disk space
- GPU
- Should work: A100 (e.g. `Standard_NC24ads_A100_v4` on Azure)
- Does not work:
- Tested on 2023-12-17 with both `Standard_NC6s_v3` and `Standard_NC12s_v3`, and ran into this error: `torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 160.00 MiB (GPU 0; 15.77 GiB total capacity; 15.12 GiB already allocated; 95.44 MiB free; 15.12 GiB reserved in total by PyTorch)`
**Machine setup**
For example, on an Azure VM running Linux (ubuntu 20.04), you will need to do the following:
- Install conda
- First set of commands (can copy-paste multiple lines)
- `wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh`
- `chmod +x Miniconda3-latest-Linux-x86_64.sh`
- `./Miniconda3-latest-Linux-x86_64.sh -b`
- Second set of commands (can copy-paste multiple lines)
- `export PATH="~/miniconda3/bin:$PATH"`
- `conda init`
- Install gcc
- `sudo apt-get -y update`
- `sudo apt-get -y upgrade`
- `sudo apt-get -y install build-essential`
- `sudo apt-get -y install libpcre3-dev`
- Install NVIDIA drivers
- `sudo apt install ubuntu-drivers-common -y`
- `sudo ubuntu-drivers autoinstall`
- `sudo reboot`
- SSH in again and confirm the installation was successful by running `nvidia-smi`
- `cd` to ARES folder and follow the rest of the README
For agents
This page has a .md twin and JSON over the API.