⚙️ Installation

---

​ To install ARES, run the following commands: ​

pip install ares-ai

​ Optional: Initalize OpenAI or TogetherAI API key with the following command:

export OPENAI_API_KEY=<your key here>
export TOGETHER_API_KEY=<your key here>

---

📝 Requirements

---

To implement ARES for scoring your RAG system and comparing to other RAG configurations, you need three components:​

• A human preference validation set of annotated query, document, and answer triples for the evaluation criteria (e.g. context relevance, answer faithfulness, and/or answer relevance). There should be at least 50 examples but several hundred examples is ideal.
• A set of few-shot examples for scoring context relevance, answer faithfulness, and/or answer relevance in your system
• A much larger set of unlabeled query-document-answer triples outputted by your RAG system for scoring

---

To get started with ARES, you'll need to set up your configuration. Below is an example of a configuration for ARES!

Copy-paste each step to see ARES in action!

---

---

For purposes of our quick start guide, we rename nq_ratio_0.5 to nq_unlabeled_output and nq_labeled_output.

<hr>

---

### 🚀 Quick Start - #1

<hr>

To get started with ARES's PPI, you'll need to set up your configuration. Below is an example of a configuration for ARES!

Just copy-paste as you go to see ARES in action!

#### Step 1) Run the following to retrieve the UES/IDP scores with GPT3.5!

```python
from ares import ARES

ues_idp_config = {
    "in_domain_prompts_dataset": "nq_few_shot_prompt_for_judge_scoring.tsv",
    "unlabeled_evaluation_set": "nq_unlabeled_output.tsv", 
    "model_choice" : "gpt-3.5-turbo-0125"
} 

ares = ARES(ues_idp=ues_idp_config)
results = ares.ues_idp()
print(results)

---

### 🚀 Quick Start - #2

<hr>

#### Step 1) Run the following to see GPT 3.5's accuracy on the NQ unlabeled dataset!

```python
from ares import ARES

ues_idp_config = {
    "in_domain_prompts_dataset": "nq_few_shot_prompt_for_judge_scoring.tsv",
    "unlabeled_evaluation_set": "nq_unlabeled_output.tsv", 
    "model_choice" : "gpt-3.5-turbo-0125"
} 

ares = ARES(ues_idp=ues_idp_config)
results = ares.ues_idp()
print(results)

---

### Machine requirements and setup when not using OpenAI API
**Machine requirements**

- Over ~100 GB of available disk space
- GPU
    - Should work: A100 (e.g. `Standard_NC24ads_A100_v4` on Azure)
    - Does not work:
        - Tested on 2023-12-17 with both `Standard_NC6s_v3` and `Standard_NC12s_v3`, and ran into this error: `torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 160.00 MiB (GPU 0; 15.77 GiB total capacity; 15.12 GiB already allocated; 95.44 MiB free; 15.12 GiB reserved in total by PyTorch)`


**Machine setup**

For example, on an Azure VM running Linux (ubuntu 20.04), you will need to do the following:
- Install conda
    - First set of commands (can copy-paste multiple lines)
        - `wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh`
        - `chmod +x Miniconda3-latest-Linux-x86_64.sh`
        - `./Miniconda3-latest-Linux-x86_64.sh -b`
    - Second set of commands (can copy-paste multiple lines)
        - `export PATH="~/miniconda3/bin:$PATH"`
        - `conda init`
- Install gcc
    - `sudo apt-get -y update`
    - `sudo apt-get -y upgrade`
    - `sudo apt-get -y install build-essential`
    - `sudo apt-get -y install libpcre3-dev`
- Install NVIDIA drivers
    - `sudo apt install ubuntu-drivers-common -y`
    - `sudo ubuntu-drivers autoinstall`
    - `sudo reboot`
    - SSH in again and confirm the installation was successful by running `nvidia-smi`
- `cd` to ARES folder and follow the rest of the README