intellagent

Simulate interactions, analyze performance, and gain actionable insights for conversational agents. Test, evaluate, and optimize your agent to ensure reliable real-world deployment.

IntellAgent is an advanced multi-agent framework that transforms the evaluation and optimization of conversational agents. By simulating thousands of realistic, challenging interactions, IntellAgent stress-tests agents to uncover hidden failure points. These insights enhance agent performance, reliability, and user experience.

Key Features

• 🔬 Generate Thousands of Edge-Case Scenarios:
  Automatically generate highly realistic edge-case scenarios tailored specifically to your agent.

• 🤖 Simulate Diverse User Interactions:
  Evaluate your agent across a wide spectrum of scenarios with varying complexity levels.

• 📊 Comprehensive Performance Evaluations:
  Access detailed analysis to identify performance gaps, prioritize improvements, and compare outcomes across experiments.

• 💪 Simple integration:
  Simple integration to your conversational agent.

How it works

IntellAgent framework consists of three steps:

• Given the user prompt (and optional additional information such as tools and database schema)
  • The system decomposes the prompt into a policy graph.
  • It samples a subset of policies based on their concurrence in real conversation distributions.
  • It generates a scenario of user-chatbot interaction (including system databases) to address the selected subset of policies.
• Simulating the user-chatbot interaction using a user agent.
• Critiquing the conversation and providing feedback on the tested policies.

To better understand the key concepts and how the IntellAgent system operates, refer to the system overview guide

🔍 Demo

:fire: Quickstart

For a more detailed and comprehensive guide, see the Start Guide.

IntellAgent requires python >= 3.9

Step 1 - Download and install

git clone git@github.com:plurai-ai/intellagent.git
cd intellagent

You can use Conda or pip to install the dependencies.

Using pip:

pip install -r requirements.txt

Step 2 - Set your LLM API Key

Edit the config/llm_env.yml file to set up your LLM configuration (OpenAI/Azure/Vertex/Anthropic):

openai:
  OPENAI_API_KEY: "your-api-key-here"

To change the default LLM provider or model for either the IntellAgent system or the chatbot, you can easily update the configuration file. For instance, modify the config/config_education.yml file:

llm_intellagent:
    type: 'azure'

llm_chat:
    type: 'azure'

To change the number of samples in the database you should modify the num_samples in the config file:

dataset:
    num_samples: 30

Tokens Usage

We invest lots of effort in minimizing the total cost of running the simulator

• Using the default parameters, the expected cost per sample is approximately $0.10
• You can control expenses by modifying the cost_limit limit parameter in the config file
• We are working on leveraging user data which will significantly reduce the cost per sample

Step 3 - Run the Simulator

If you're utilizing Azure OpenAI services for the llm_intellagent, ensure you disable the default jailbreak filter before running the simulator.

For fast simple environment without a database, run the following command:

python